Sign_language

Description

This is a project which produces the alphabets from the hand gestures based on American Sign Language from live video input.There are two models implemented here one trained on the MNIST dataset and the other trained on ASL dataset . Both the models are CNN architecture . All contributions are welcome .

Setting up the environment

Datasets

• MNIST Dataset
• ALS Dataset

Cloning the Repo

git clone https://github.com/19-ade/Sign_language.git

Once the repo has been cloned ,the folder with the checkpoints for the ASL Model needs to be downloaded and pasted in the same folder as the project. Due to github size limitation for uploading files I had to take this path . Don't change the name of the folder or the files within. Here's the link.

Run the Requirements.py script to install all the required libraries.

python requirements.py

Run capture.py once everything has been configured and achieved

python capture.py

Model

This is the input that the Model recieves

MNIST CNN

The CNN model was trained for 15 epochs. The following plots show the variation of accuracy and loss of the validation and training split wrt epochs

ASL CNN

The CNN model was trained for 10 epochs . It is a much more computation-intensive model, so it is advised to use GPU for training the model. The following plots show the variation of accuracy and loss of the validation and training split wrt epochs

Scope

• The ASL CNN can be modified to learn from RGB data (in our program it is (64 X 64 X 1) dimension, grayscale data). Might imporve the accuracy even more.
• As of now no proper measures have been taken to isolate the hand area from ROI in the opencv Script . Proper algorithms can be added to isolate said hand , remove noise from the data .
• The red rectangle is a fixed ROI . Perhaps an algorithm can be implemented that can recognise the hand in the video , thus allowing flexibility.
• The dataset can be expanded to include numbers, or modified to read sentences

Output (Some Examples)

Few things:

• make sure the background is relatively noise free
• make sure to keep your hand at approximately 30 cm distance so that the entire palm fits into the red rectangle .