An AI-powered mobile accessibility system designed for users with motor and visual impairments, enabling touch-free interaction through gesture recognition, voice commands, and eye-tracking.
GestureSmart bridges the accessibility gap in mobile interfaces by offering an inclusive and adaptive system that enables users to control mobile devices through gestures, voice, or eye movements. This combination of AI technologies ensures usability across diverse environments and offers users the freedom to choose their preferred interaction mode.
Explore the GestureSmart interface and features through our comprehensive screenshot gallery:
- Gesture Control Hand-based navigation using MediaPipe Gestures and HandPose
- Voice Commands Real-time speech input via AssemblyAI API
- Eye Tracking MediaPipe FaceMesh-based gaze and blink detection for UI control
- Seamless Mode Switching Intelligent fallback mechanisms between input modes
- No personal data collection - only anonymized device tokens and session logs
- On-device processing for core gesture features
- Encrypted communication using HTTPS
- WCAG 2.1 compliant accessibility standards
- Core gesture features work offline using on-device TensorFlow Lite models
- Ideal for rural or low-connectivity areas
- Ensures privacy and usability without internet dependency
- Designed following WCAG 2.1 principles for inclusive design
- Voice alerts and audio cues for visual feedback
- Minimal visual clutter to support users with cognitive or visual impairments
- Expo SDK 53 - React Native development platform
- React Native 0.79.5 - Cross-platform mobile framework
- React 19.0.0 - Latest React version
- TypeScript 5.8.3 - Type-safe JavaScript development
- Custom Java Native Code - Platform-specific optimizations
- Expo Prebuild - Custom native code integration
- Node.js - Server-side runtime
- MongoDB Atlas - Cloud database for session data
- Express.js - Web application framework
- Redux Toolkit - State management
- Async Storage - Local data persistence
- MediaPipe Gestures - Real-time gesture recognition
- MediaPipe HandPose - Hand landmark detection
- MediaPipe FaceMesh - Eye tracking and facial analysis
- Google ML Kit Face Detection - Enhanced face tracking capabilities
- Custom Java Native Modules - Optimized ML processing
- TensorFlow Lite - On-device machine learning
- AssemblyAI - Voice recognition and speech-to-text
MediaPipe Hand Landmarker & Gesture Recognizer
- Implementation:
com.google.mediapipe:tasks-vision:0.10.26.1 - Hand Landmarker: Google AI Documentation
- Gesture Recognizer: Google AI Documentation
- Model Card: Hand Gesture Classification with Fairness
- Hand Tracking Model: Hand Tracking (Lite/Full) with Fairness
Google ML Kit Face Detection
- Documentation: ML Kit Vision Face Detection
- Features: Real-time face mesh analysis, eye gaze tracking, blink detection
- Privacy: On-device processing with no data transmission
AssemblyAI Speech-to-Text
- Service: AssemblyAI Platform
- Features: Real-time transcription, noise filtering, multi-language support
- Usage: Voice command processing and accessibility features
PORT=8000MONGO_URI=your_mongodb_connection_stringTOKEN_KEY=your_jwt_secret_keyAPP_NAME=Gesture Smart ServerEmailHost=smtp.gmail.comEmailPort=587EmailSecured=falseEmailUser=your_email_addressEmailPass=your_email_passwordCLOUD_NAME=your_cloudinary_cloud_nameCLOUDINARY_KEY=your_cloudinary_api_keyCLOUDINARY_SECRET=your_cloudinary_api_secretENVIRONMENT=DEV- MongoDB Atlas - Database connection
- AssemblyAI - Voice recognition (Get API Key)
- Cloudinary - Image/media storage
- Email Service - Notification system
The system demonstrates high reliability across key performance indicators:
- Precision High accuracy in controlled environments
- Accuracy Consistent performance across different input modes
- Recall Effective detection of user intents
- TNR (True Negative Rate) Robust false positive filtering
- Fallback Reliability Seamless mode switching preserves usability
- Node.js (v18 or higher)
- Expo CLI (latest version)
- Android Studio / Xcode
- EAS CLI (for builds)
- MongoDB Atlas account
- AssemblyAI account
- Java Development Kit (for custom native modules)
Clone the repository
git clone https://github.com/ateebnoone/gesturesmart.git cd MobileAppInstall dependencies
npm install # or yarn installEnvironment Configuration
- Add your API keys in
src/constants/api_keys.ts - Configure server environment variables as shown above
- Add your API keys in
Prebuild with custom native code
yarn prebuild # This will backup native code, run expo prebuild, and restore custom native modulesStart the development server
yarn start # or npx expo startRun on device/simulator
# For Android yarn android
Using EAS Build (Recommended)
# Build for Android
yarn buildCloud┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Expo/RN App │ │ Node.js │ │ MongoDB Atlas │
│ (TypeScript) │◄──►│ Backend │◄──►│ Database │
│ + Custom Java │ │ (Express.js) │ │ │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Native Modules│ │ Cloud Services │
│ • MP Gestures │ │ • AssemblyAI │
│ • MP HandPose │ │ • EAS Build │
│ • MP FaceMesh │ │ • Expo Services │
│ • Custom Java │ │ • Push Notifs │
└─────────────────┘ └─────────────────┘
┌─────────────────────────────────────────────────────────┐
│ React Native App │
├─────────────────────────────────────────────────────────┤
│ Navigation Layer (React Navigation + Drawer) │
├─────────────────────────────────────────────────────────┤
│ State Management (Redux Toolkit + Async Storage) │
├─────────────────────────────────────────────────────────┤
│ Feature Modules │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ Gesture │ │ Voice │ │ Eye Track │ │
│ │ Detection │ │ Commands │ │ (FaceMesh)│ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
├─────────────────────────────────────────────────────────┤
│ Native Bridge (Custom Java + Expo Modules) │
├─────────────────────────────────────────────────────────┤
│ Platform APIs (Camera, Microphone, Sensors) │
└─────────────────────────────────────────────────────────┘
- Gesture Recognition Annotated video frames of hand movements for swipes, taps, and zoom actions using MediaPipe HandPose landmarks
- Voice Commands Pre-recorded datasets with varying accents and environmental conditions processed through AssemblyAI
- Eye Tracking Facial landmark datasets for gaze direction and blinking pattern detection using MediaPipe FaceMesh
- Preprocessing Data normalization and augmentation
- Framework MediaPipe Gestures, HandPose, and FaceMesh with deep learning integration
- Training Metrics Accuracy, Precision, Recall, and F1-Score optimization
GestureSmart utilizes Expo's managed workflow with custom native code integration:
- Expo SDK 53 Latest managed workflow benefits
- Custom Java Modules Performance-critical ML processing
- Automated Build Scripts Backup/restore native code during prebuild
- EAS Build Cloud-based compilation for production builds
# Custom scripts for native code management
yarn backup-native # Backup custom native modules
yarn restore-native # Restore after prebuild
yarn prebuild # Full prebuild with native code preservation- Redux Toolkit Centralized state management
- Async Storage Persistent local data storage
- React Navigation Multi-screen navigation with drawer
- Background Actions Continuous gesture recognition
- Individuals with motor impairments
- Visually impaired users
- Users with speech difficulties
- Elderly users seeking simplified interfaces
- Noisy Environments Voice recognition with noise filtering
- Low-Light Conditions Gesture and eye-tracking alternatives
- Dynamic Settings Adaptive mode switching based on context
- Offline Scenarios Core functionality without internet dependency
- Enable users to create personalized hand gestures
- On-device ML training for privacy and adaptability
- Tailored gestures based on individual mobility needs
- Vibration-based tactile cues to simulate Braille
- Navigate interface without visual or audio support
- Enhanced accessibility for blind users
- Lightweight, personalized models
- Improved accuracy and responsiveness
- Reduced cloud dependency for better privacy
- Regional language support (Urdu, Hindi, Pashto, etc.)
- Right-to-left (RTL) text support
- Localized content for global accessibility
The system underwent extensive testing across:
- Controlled Environments Laboratory conditions with optimal lighting and minimal noise
- Real-World Scenarios Varied lighting, background noise, and user movement
- Accessibility Compliance WCAG 2.1 guidelines adherence
- Cross-Platform Compatibility iOS and Android device testing
We welcome contributions to make GestureSmart more accessible and inclusive! Please read our Contributing Guidelines before submitting pull requests.
- Follow Expo and React Native best practices
- Ensure accessibility compliance in all features
- Test custom native modules on multiple devices
- Use TypeScript for type safety
- Maintain clean Redux state architecture
- Document native code integrations
- Test prebuild process before major releases
This project is licensed under the MIT License - see the LICENSE file for details.
Developer: Atib Ur Rehman
- 📧 Email: ateebnoone@gmail.com
- 💬 WhatsApp: +923182553930
Special thanks to all the testers and accessibility advocates who provided valuable feedback during development and testing phases. Your insights were crucial in making GestureSmart truly inclusive.
GestureSmart is actively maintained and continuously improved. The system has demonstrated technical and commercial viability as a scalable accessibility solution through extensive real-world testing.
Building technology that empowers everyone to interact with the digital world, regardless of their physical abilities.