Multi-functional extensible project based on gesture recognition

This is a sample code program that builds functionalities (virtual keyboard, touchpad, and drawboard) upon hand gesture recognition project (this one) which utilizes a simple MLP using key points detected with Mediapipe.

This repository contains the following contents.

• Sample program
• Hand sign recognition model(TFLite)
• Hand controlled virtual touchpad
• Hand controlled virtual keyboard
• Hand controlled virtual drawboard

Requirements

* mediapipe 0.8.1
* OpenCV 3.4.2 or Later
* Tensorflow 2.3.0 or Later<br>tf-nightly 2.5.0.dev or later (Only when creating a TFLite for an LSTM model)

• scikit-learn 0.23.2 or Later (Only if you want to display the confusion matrix)
• matplotlib 3.3.2 or Later (Only if you want to display the confusion matrix)
• pyautogui 0.9.54 or Later

Demo

Here's how to run the demo using your webcam.

python app.py

The following options can be specified when running the demo.

• --device
  Specifying the camera device number (Default：0)
• --width
  Width at the time of camera capture (Default：960)
• --height
  Height at the time of camera capture (Default：540)
• --use_static_image_mode
  Whether to use static_image_mode option for MediaPipe inference (Default：Unspecified)
• --min_detection_confidence
  Detection confidence threshold (Default：0.5)
• --min_tracking_confidence
  Tracking confidence threshold (Default：0.5)
• --function to be enabled
• --max_num of hands detected

Directory

│ app.py
│ keypoint_classification.ipynb
│ point_history_classification.ipynb
│ ├─model
│ ├─keypoint_classifier
│ │ │ keypoint.csv
│ │ │ keypoint_classifier.hdf5
│ │ │ keypoint_classifier.py
│ │ │ keypoint_classifier.tflite
│ │ └─ keypoint_classifier_label.csv
│ │ │ └─point_history_classifier
│ │ point_history.csv
│ │ point_history_classifier.hdf5
│ │ point_history_classifier.py
│ │ point_history_classifier.tflite
│ └─ point_history_classifier_label.csv
├─plugin
│ ├─cnn_model
│ │ │ dataCNN
│ │ │ cnn_labels.json
│ │ │ cnn.py
│ │ │ getdata.py
│ │ │ load_model.py
│ │ └─ make_labels.py
│ │ │ │ blackboard.py
│ │ keyboard.py
│ │ mouse.py
│ │ stablediffusion.py
│ │ UI.py │ └─point_history_classifier
│ │ └─utils
└─cvfpscalc.py

app.py

This is a sample program for inference.
In addition, learning data (key points) for hand sign recognition,
You can also collect training data (index finger coordinate history) for finger gesture recognition.

keypoint_classification.ipynb

This is a model training script for hand sign recognition.

point_history_classification.ipynb

This is a model training script for finger gesture recognition.

model/keypoint_classifier

This directory stores files related to hand sign recognition.
The following files are stored.

• Training data(keypoint.csv)
• Trained model(keypoint_classifier.tflite)
• Label data(keypoint_classifier_label.csv)
• Inference module(keypoint_classifier.py)

model/point_history_classifier

This directory stores files related to finger gesture recognition.
The following files are stored.

• Training data(point_history.csv)
• Trained model(point_history_classifier.tflite)
• Label data(point_history_classifier_label.csv)
• Inference module(point_history_classifier.py)

model/point_history_classifier

This is directory stores the different functions as plugins that will be inserted in app.py.
The following folders/files are stored.

• cnn_model
• blackboard.py
• keyboard.py
• mouse.py
• stablediffusion.py
• UI.py

utils/cvfpscalc.py

This is a module for FPS measurement.