Real-Time American Sign Language Interpretation Using Deep Convolutional Neural Networks

In spite of being the 4th most commonly used language in the United States, sign language is actively used by only 10–14% of the members of the speech and hearing impairment community and continues to be one of the most understudied areas. In compliance with recent advances in deep learning, this paper explores the possibility of using deep convolutional neural network to interpret American Sign Language in real-time. The paper aims to develop a CNN from scratch and train it using a dataset patterned to closely match the format of the classic MNIST dataset (28 × 28 pixel images with pixel values ranging from 0 to 255). The evaluation of the network shows that it outperforms all previous implementations surrounding this task with 89.6% accuracy during real-time user testing (99.7% accuracy on validation set). Such high accuracy measure and fast converging time can be attributed to modelling the training data as a dataframe of pixel values rather than using traditional images and including batch normalization, concepts which have not been employed in earlier implementations to the best of our knowledge. To make the solution as accessible as possible, we refrained from using sophisticated hardware like motion-tracking gloves and depth-sensing cameras and deployed the trained model as a multi-platform mobile application.