Utilizing Contactless Sensing Technology for the Identification of Hand and Head Movements in Conjunction With Facial Expressions

The sign language serves as a primary mode for communication for hearing-impaired individuals, where one utilizes movements of hand, body gestures, and facial expressions. The complexity of sign language encompasses various hand and finger articulations, often coordinated with head, face, and body movements. Despite nearly three decades of research, automatic sign language recognition remains an evolving field, presenting considerable challenges. Existing wearable, audio-based, and vision-based systems have some loopholes including privacy, lighting issues, noisy environment, and maintenance. To solve these issues, we proposed a contactless radar-based system for recognizing expressions through the analysis of head and hand movements, and facial expressions and translating them. Based on our information, this is the first contactless recognition system that recognizes head and hand movements, and facial expressions simultaneously using radar and deep learning (DL) models. This study addresses the challenge of expression recognition by leveraging micro-Doppler signatures acquired through radar sensor technology. Our suggested approach extracts 2-D spatiotemporal features from radar data and employs state-of-the-art DL architectures for the classification of 16 expressions such as Ashamed, Cheerful, Enormal, Furious, GoodIdea, Guilty, Lonely, Normal, Ok, Playful, Proud, Sad, Shocked, Surprised, Thinking, and Worried. We collected a diverse dataset of 1440 samples from human subjects aged 20–40. Four pretrained DL models, GoogleNet, SqueezeNet, VGG16, and VGG19 were applied to this dataset after preprocessing to classify the expressions. Notably, VGG16 outperformed other models with 94.2% accurate results.