Grant: Discovery Grants Program - Individual

Sign language consists of symbolic gestures to establish communication by combining the shape, orientation and movement of hands and arms, as well as facial expressions to fluidly express a speaker's thoughts. American Sign Language is the primary means of communication for more than half million deaf people in the United States and Canada. My research is focused on communication with the hearing-impaired by developing a software tool that can interpret sign language. There are two methods of collecting information about hand gestures, data gloves and video cameras. Data glove is a glove equipped with sensors that sense the movements of the hand and interfaces those movements with a computer. The data obtained from data gloves are concise and requires little post-processing. However, the data gloves are limited by the number of sensors and they do not capture facial expressions which some signers use to communicate additional information. Some recent research has been shown that video cameras can capture hands positions as well as facial expressions. However, such cameras are prone to typical computer vision problems such as occlusion and sensitivity to lighting conditions. These vision-based methods typically use image processing techniques to segment the hand from the image background, and extract features such as the fingertips for use in gesture recognition. The proposed system depends on the inputs from the gloves and a single camera. It consists of four modules. First, the real time hand tracking and extraction algorithm is applied to trace the moving hand and extract the hand region. Secondly, Fourier or wavelet descriptors will be adopted to characterize the spatial features. A motion analysis will be conducted to characterize the temporal features. The new feature vector will be created with both spatial and temporal features. In the third module, the description of hand shape, orientation, and movements are used as inputs to the training module for the segmentation and recognition. The collected data will be transferred to the fourth module for classification and the word corresponding to the sign will be determined. This research will allow a hearing-impaired person to communicate with other people.