A comprehensive, modular, and production-ready AI-powered exam proctoring system with advanced behavioral pattern detection. Built with a decoupled architecture for maximum flexibility and extensibility.
- ✅ Face Detection - Detect no face, single face, or multiple faces
- ✅ Gaze Tracking - Iris-based eye tracking to detect looking away
- ✅ Head Pose Estimation - Track head rotation (yaw, pitch, roll)
- ✅ Mouth Movement Detection - Detect talking and whispering
- ✅ Mouth Covering Detection - Identify attempts to hide mouth
- ✅ Suspicious Object Detection - Detect phones, books, laptops, tablets
- ✅ Talking Detection - Identify normal speech above threshold
- ✅ Whispering Detection - Detect subtle audio below talking threshold
- ✅ Audio Level Tracking - Continuous audio level monitoring
- ✅ Silence Detection - Track periods of silence
- ✅ Duration Tracking - Measure total talking/whispering time
- ✅ Multi-Modal Analysis - Correlate visual + audio events
- ✅ Time-Windowed Patterns - Detect patterns within configurable windows
- ✅ Suspicious Correlations - Identify cheating behaviors:
- Looking away + talking + mouth moving
- Looking left/right + whispering
- Mouth covered + audio detected
- Suspicious object + looking away
- Multiple faces + audio detected
- Rapid eye movements (reading notes)
- Repeated tab switches
- ✅ Tab Switch Detection - Monitor when student leaves exam tab
- ✅ Focus Tracking - Detect window focus loss/gain
- ✅ Clipboard Monitoring - Track copy/paste/cut attempts
- ✅ Key Press Detection - Identify suspicious keyboard shortcuts
- ✅ Fullscreen Monitoring - Detect fullscreen exit
- ✅ Mouse Tracking - Monitor mouse leaving window
- ✅ Modular Design - Independent, swappable modules
- ✅ Event-Driven - Reactive architecture with event system
- ✅ State Management - Centralized state with subscription system
- ✅ Configurable - Extensive configuration options
- ✅ Extensible - Easy to add custom patterns and modules
- ✅ Model Cache - Models are cached for faster loading
npm install @timadey/proctorOr with yarn:
yarn add @timadey/proctorimport{ProctoringEngine}from'@timadey/proctor';// 1. Initialize the engineconstengine=ProctoringEngine.getInstance({// Enable/disable modulesenableVisualDetection: true,enableAudioMonitoring: true,enablePatternDetection: true,enableBrowserTelemetry: true,// CallbacksonEvent: (event)=>{console.log('Proctoring event:',event);// Send to your backend},onBehavioralPattern: (pattern)=>{console.warn('Suspicious pattern detected:',pattern);// Alert supervisor}});// 2. Initialize modulesawaitengine.initialize();// 3. Start proctoringconstvideoElement=document.getElementById('webcam');engine.start(videoElement);// 4. Get session summary anytimeconstsummary=engine.getSessionSummary();console.log('Suspicious score:',summary.suspiciousScore);// 5. Stop when exam endsengine.stop();- Installation
- Quick Start
- Architecture
- Modules
- Configuration
- Events
- Patterns
- API Reference
- Examples
- Browser Support
- Performance
- Security
- Contributing
- License
The system uses a decoupled, modular architecture where each component operates independently:
┌────────────────────────────────────────┐
│ ProctoringEngine (Orchestrator) │
│ ┌─────────────┐ ┌──────────────┐ │
│ │EventManager │ │StateManager │ │
│ └─────────────┘ └──────────────┘ │
└────────────┬───────────────────────────┘
│
┌────────┼────────┬────────┐
▼ ▼ ▼ ▼
┌────────┐ ┌────┐ ┌────┐ ┌────────┐
│Visual │ │Audio│ │Pat │ │Browser │
│Module │ │Mod │ │Mod │ │Telemetry│
└────────┘ └────┘ └────┘ └────────┘
- ProctoringEngine - Main orchestrator coordinating all modules
- VisualDetectionModule - Computer vision and face tracking
- AudioMonitoringModule - Audio analysis and detection
- PatternDetectionModule - Behavioral pattern recognition
- BrowserTelemetryModule - Browser interaction monitoring
- EventManager - Centralized event handling and logging
- StateManager - Application state management
Handles all computer vision tasks using MediaPipe.
// Access directly if neededconstvisualState=engine.visualModule.getState();console.log('Current gaze:',visualState.currentGazeDirection);console.log('Number of faces:',visualState.numFaces);console.log('Mouth moving:',visualState.isMouthMoving);console.log('Object detected:',visualState.suspiciousObjectDetected);Features:
- Face landmarking with 468 facial points
- Real-time gaze direction (left, right, up, down, center)
- Head pose angles (yaw, pitch, roll)
- Mouth aspect ratio for speech detection
- Object detection for unauthorized materials
Monitors audio using Web Audio API.
// Access audio stateconstaudioState=engine.audioModule.getState();console.log('Is talking:',audioState.isTalking);console.log('Is whispering:',audioState.isWhispering);console.log('Audio level:',audioState.currentAudioLevel,'dB');console.log('Total talking time:',audioState.totalTalkingDuration,'ms');Features:
- RMS (Root Mean Square) audio level calculation
- Talking detection (configurable threshold)
- Whispering detection (lower threshold)
- Audio level history tracking
- Silence duration tracking
Detects suspicious behavioral patterns through correlation.
// Get pattern summaryconstpatterns=engine.patternModule.getPatternSummary();console.log('Suspicious patterns detected:',patterns);Detected Patterns:
suspiciousTriplePattern- Looking away + talking + mouth movinglookingLeftWhispering- Looking left while whisperinglookingRightWhispering- Looking right while whisperingmouthCoveredWithAudio- Mouth covered while audio detectedlookingAwayAndTalking- Looking away while talkingobjectAndLookingAway- Suspicious object + looking awaymultipleFacesWithAudio- Multiple people + audioheadTurnedTalking- Head turned + talking
Monitors all browser interactions.
// Get telemetry summaryconsttelemetry=engine.telemetryModule.getSummary();console.log('Tab switches:',telemetry.tabSwitches);console.log('Copy attempts:',telemetry.copyAttempts);console.log('Paste attempts:',telemetry.pasteAttempts);Monitored Actions:
- Tab visibility changes
- Window focus/blur events
- Copy/paste/cut operations
- Suspicious keyboard shortcuts (F12, Ctrl+C, etc.)
- Fullscreen changes
- Right-click attempts
- Mouse leaving window
constengine=ProctoringEngine.getInstance({// ===== Module Toggles =====enableVisualDetection: true,enableAudioMonitoring: true,enablePatternDetection: true,enableBrowserTelemetry: true,// ===== Visual Detection Options =====detectionFPS: 10,// Frame processing rate (5-30)stabilityFrames: 15,// Frames before event triggersgazeThreshold: 20,// Degrees for gaze deviationyawThreshold: 25,// Degrees for head rotationpitchThreshold: 20,// Degrees for head tiltprolongedGazeAwayDuration: 5000,// ms for prolonged gazemouthOpenRatioThreshold: 0.15,// Mouth aspect ratio threshold// ===== Audio Monitoring Options =====talkingThreshold: -45,// dB for talking detectionwhisperThreshold: -55,// dB for whisper detectionaudioSampleInterval: 100,// Audio check interval (ms)prolongedTalkingDuration: 3000,// ms for prolonged talking// ===== Pattern Detection Options =====suspiciousPatternThreshold: 3,// Events to trigger patternpatternDetectionWindow: 10000,// Time window (ms)// ===== Callbacks =====onEvent: (event)=>{// Handle individual eventsconsole.log('Event:',event);},onBehavioralPattern: (pattern)=>{// Handle detected patterns (critical)console.warn('Pattern:',pattern);},onStatusChange: (status)=>{// Engine status: 'initializing', 'loading-models', 'ready', 'error'console.log('Status:',status);},onError: (error)=>{// Handle errorsconsole.error('Error:',error);}});// Lightweight mode - only browser telemetryconstlightEngine=ProctoringEngine.getInstance({enableVisualDetection: false,enableAudioMonitoring: false,enablePatternDetection: false,enableBrowserTelemetry: true});// Heavy mode - full monitoringconstheavyEngine=ProctoringEngine.getInstance({enableVisualDetection: true,enableAudioMonitoring: true,enablePatternDetection: true,enableBrowserTelemetry: true,detectionFPS: 15// Higher FPS for more accuracy});// Custom mode - visual + patterns onlyconstcustomEngine=ProctoringEngine.getInstance({enableVisualDetection: true,enableAudioMonitoring: false,enablePatternDetection: true,enableBrowserTelemetry: true});// Update configuration during sessionengine.updateOptions({gazeThreshold: 30,// More lenientdetectionFPS: 5// Reduce CPU usage});// Update individual modulesengine.visualModule.updateOptions({detectionFPS: 8});engine.audioModule.updateOptions({talkingThreshold: -40});{event: 'TALKING_DETECTED',// Event typelv: 8,// Severity level (1-10)ts: 1703098765432,// Timestamp (Unix ms)source: 'audio',// Module sourcesessionDuration: 123456,// Time since session start (ms)// Event-specific metadataduration: 5000,// Duration of behavior (ms)level: -40,// Audio level (dB)direction: 'left',// Direction (for gaze/head)severity: 'high',// Human-readable severityextractedFeatures: {}// The face and hand features extracted from the frame}NO_FACE- No face detectedMULTIPLE_FACES- Multiple people in framePERSON_LEFT- Student left for extended periodSUSPICIOUS_OBJECT- Unauthorized object detectedTAB_SWITCHED- Tab switch detectedPASTE_ATTEMPT- Paste operationPATTERN_*- Behavioral pattern detected
GAZE_AWAY- Looking away from screenPROLONGED_GAZE_AWAY- Extended gaze awayHEAD_TURNED- Head significantly rotatedPROLONGED_MOUTH_MOVEMENT- Extended mouth movementTALKING_DETECTED- Speech detectedWHISPERING_DETECTED- Whispering detectedWINDOW_FOCUS_LOST- Window lost focusEXITED_FULLSCREEN- Fullscreen exitedCOPY_ATTEMPT- Copy operation
MOUTH_MOVING- Mouth movement detectedMOUTH_COVERED- Mouth appears coveredEYES_OFF_SCREEN- Eyes looking off-screenRIGHT_CLICK- Right-click attemptSUSPICIOUS_KEY_PRESS- Suspicious keyboard shortcut
MOUSE_LEFT_WINDOW- Mouse left windowWINDOW_FOCUS_RESTORED- Focus restoredTAB_RETURNED- Returned to tab
Patterns are critical alerts indicating high probability of cheating.
When: Student is looking away + talking + mouth moving simultaneously
Severity: 10 (Critical)
Interpretation: Likely communicating with someone off-screen
onBehavioralPattern: (pattern)=>{if(pattern.pattern==='suspiciousTriplePattern'){// This is extremely suspiciousalertSupervisor('Student likely cheating');flagExamForReview();}}When: Student looking to side while whispering
Severity: 8 (High)
Interpretation: Possibly communicating with nearby person
When: Mouth covered but audio detected
Severity: 9 (High)
Interpretation: Attempting to hide speaking
When: Suspicious object detected + looking away from screen
Severity: 9 (High)
Interpretation: Using unauthorized materials
When: Multiple people + audio detected
Severity: 10 (Critical)
Interpretation: Multiple people taking exam together
Add your own patterns:
// Add custom patternengine.patternModule.patterns.myCustomPattern={name: 'myCustomPattern',severity: 8,events: [],count: 0,lastTriggered: 0,check: (visualState,audioState)=>{// Your custom logicreturnvisualState.numFaces===0&&audioState.isTalking;}};Get singleton instance.
constengine=ProctoringEngine.getInstance(options);Initialize all enabled modules.
awaitengine.initialize();Start proctoring with video element.
constvideo=document.getElementById('webcam');engine.start(video);Stop proctoring.
engine.stop();Update configuration at runtime.
engine.updateOptions({detectionFPS: 5});Get comprehensive session summary.
constsummary=engine.getSessionSummary();/*{ sessionDuration: 1800000, sessionStartTime: 1703098765432, sessionEndTime: 1703100565432, totalEvents: 45, eventCounts: {...}, eventsBySeverity: {...}, patterns: {...}, visualState: {...}, audioState: {...}, suspiciousScore: 127}*/Get all event logs.
constlogs=engine.getLogs();Clear all logs and patterns.
engine.clearLogs();Calculate overall suspicious score (0-1000).
constscore=engine.calculateSuspiciousScore();// 0-50: Normal// 51-100: Some suspicious activity// 101-200: Concerning behavior// 201+: High probability of cheatingCleanup and destroy engine.
engine.destroy();Subscribe to state changes.
constunsubscribe=engine.stateManager.subscribe((state)=>{console.log('State updated:',state);});// Unsubscribe laterunsubscribe();Get complete current state.
conststate=engine.stateManager.getCompleteState();Get visual detection state.
constvisual=engine.stateManager.getVisualState();Get audio monitoring state.
constaudio=engine.stateManager.getAudioState();Get all recorded events.
constevents=engine.eventManager.getAllEvents();Get events of specific type.
consttabSwitches=engine.eventManager.getEventsByType('TAB_SWITCHED');Get events above severity threshold.
constcritical=engine.eventManager.getEventsBySeverity(9);Get event summary statistics.
constsummary=engine.eventManager.getSummary();import{ProctoringEngine}from'./ProctoringEngine.js';classSimpleProctor{constructor(){this.engine=ProctoringEngine.getInstance({onEvent: (e)=>console.log('Event:',e.event),onBehavioralPattern: (p)=>alert(`Warning: ${p.pattern}`)});}asyncstart(){// Get cameraconststream=awaitnavigator.mediaDevices.getUserMedia({video: true});constvideo=document.getElementById('video');video.srcObject=stream;awaitvideo.play();// Start proctoringawaitthis.engine.initialize();this.engine.start(video);}stop(){constsummary=this.engine.getSessionSummary();console.log('Final score:',summary.suspiciousScore);this.engine.stop();}}constproctor=newSimpleProctor();awaitproctor.start();classAdvancedProctor{constructor(examId,studentId){this.examId=examId;this.studentId=studentId;this.ws=null;// WebSocket connectionthis.engine=ProctoringEngine.getInstance({onEvent: (event)=>this.handleEvent(event),onBehavioralPattern: (pattern)=>this.handlePattern(pattern)});}asyncstart(){// Connect to backend via WebSocketthis.ws=newWebSocket('wss://api.example.com/proctoring');// Setup cameraconststream=awaitnavigator.mediaDevices.getUserMedia({video: {width: 1280,height: 720}});constvideo=document.getElementById('video');video.srcObject=stream;awaitvideo.play();// Initialize and startawaitthis.engine.initialize();this.engine.start(video);// Subscribe to state for real-time updatesthis.engine.stateManager.subscribe((state)=>{this.sendStateUpdate(state);});// Periodic summariesthis.summaryInterval=setInterval(()=>{this.sendSummary();},30000);// Every 30 seconds}handleEvent(event){// Send event to backend via RESTfetch('/api/proctoring/event',{method: 'POST',headers: {'Content-Type': 'application/json'},body: JSON.stringify({examId: this.examId,studentId: this.studentId,event: event})});// Send via WebSocket for real-time monitoringif(this.ws&&this.ws.readyState===WebSocket.OPEN){this.ws.send(JSON.stringify({type: 'EVENT',data: event}));}// Show to student if criticalif(event.lv>=8){this.showWarning(event.event);}}handlePattern(pattern){// Critical alert - send immediatelyfetch('/api/proctoring/critical',{method: 'POST',headers: {'Content-Type': 'application/json'},body: JSON.stringify({examId: this.examId,studentId: this.studentId,pattern: pattern,timestamp: Date.now()}),keepalive: true// Ensure delivery});// Alert supervisor via WebSocketif(this.ws&&this.ws.readyState===WebSocket.OPEN){this.ws.send(JSON.stringify({type: 'CRITICAL_PATTERN',data: pattern}));}// Show strong warning to studentthis.showCriticalWarning(pattern.pattern);}sendStateUpdate(state){if(this.ws&&this.ws.readyState===WebSocket.OPEN){this.ws.send(JSON.stringify({type: 'STATE_UPDATE',data: {examId: this.examId,studentId: this.studentId,state: state}}));}}sendSummary(){constsummary=this.engine.getSessionSummary();fetch('/api/proctoring/summary',{method: 'POST',headers: {'Content-Type': 'application/json'},body: JSON.stringify({examId: this.examId,studentId: this.studentId,summary: summary})});}showWarning(eventType){constwarnings={'TALKING_DETECTED': 'Please remain quiet during the exam.','TAB_SWITCHED': 'Do not switch tabs during the exam.','MULTIPLE_FACES': 'Multiple people detected. Only you should be visible.','PERSON_LEFT': 'You have left the exam area.',};constmessage=warnings[eventType]||'Suspicious activity detected.';constwarning=document.getElementById('warning');warning.textContent=message;warning.classList.add('show');setTimeout(()=>{warning.classList.remove('show');},5000);}showCriticalWarning(patternName){constmodal=document.getElementById('critical-modal');modal.querySelector('.message').textContent=`Critical violation detected: ${patternName}. This exam may be flagged for review.`;modal.classList.add('show');setTimeout(()=>{modal.classList.remove('show');},10000);}asyncstop(){// Clear intervalif(this.summaryInterval){clearInterval(this.summaryInterval);}// Get final dataconstsummary=this.engine.getSessionSummary();constlogs=this.engine.getLogs();// Send final reportconstfinalData={examId: this.examId,studentId: this.studentId,summary: summary,logs: logs,endTime: Date.now()};// Use sendBeacon for reliability during unloadconstblob=newBlob([JSON.stringify(finalData)],{type: 'application/json'});navigator.sendBeacon('/api/proctoring/finalize',blob);// Close WebSocketif(this.ws){this.ws.close();}// Destroy enginethis.engine.destroy();}}// Usageconstproctor=newAdvancedProctor('exam-123','student-456');awaitproctor.start();// On exam submitdocument.getElementById('submit-btn').addEventListener('click',async()=>{awaitproctor.stop();// Submit exam answers...});// On page unloadwindow.addEventListener('beforeunload',()=>{proctor.stop();});importReact,{useEffect,useRef,useState}from'react';import{ProctoringEngine}from'./ProctoringEngine';functionExamProctoring({ examId, studentId }){constvideoRef=useRef(null);constengineRef=useRef(null);const[status,setStatus]=useState('initializing');const[events,setEvents]=useState([]);const[score,setScore]=useState(0);const[warning,setWarning]=useState('');useEffect(()=>{letmounted=true;constinitialize=async()=>{try{// Setup cameraconststream=awaitnavigator.mediaDevices.getUserMedia({video: {width: 1280,height: 720}});if(videoRef.current){videoRef.current.srcObject=stream;awaitvideoRef.current.play();}// Initialize engineconstengine=ProctoringEngine.getInstance({onEvent: (event)=>{if(mounted){setEvents(prev=>[event, ...prev].slice(0,20));}},onBehavioralPattern: (pattern)=>{if(mounted){setWarning(`⚠️ ${pattern.pattern} detected`);setTimeout(()=>setWarning(''),5000);}},onStatusChange: (newStatus)=>{if(mounted)setStatus(newStatus);}});engineRef.current=engine;awaitengine.initialize();if(videoRef.current){engine.start(videoRef.current);}// Update score periodicallyconstinterval=setInterval(()=>{if(engineRef.current&&mounted){constsummary=engineRef.current.getSessionSummary();setScore(summary.suspiciousScore);}},5000);return()=>{clearInterval(interval);};}catch(error){console.error('Initialization failed:',error);if(mounted)setStatus('error');}};initialize();return()=>{mounted=false;if(engineRef.current){engineRef.current.stop();engineRef.current.destroy();}};},[examId,studentId]);return(<divclassName="proctoring-container"><videoref={videoRef}autoPlayplaysInlinemutedclassName="proctoring-video"/><divclassName="proctoring-info"><divclassName="status">
Status: <spanclassName={status}>{status}</span></div><divclassName="score">
Suspicious Score: <span>{score}</span></div></div>{warning&&(<divclassName="warning-banner">{warning}</div>)}<divclassName="event-log"><h3>Recent Events</h3>{events.map((event,i)=>(<divkey={i}className={`event severity-${event.lv}`}>{newDate(event.ts).toLocaleTimeString()} - {event.event}</div>))}</div></div>);}exportdefaultExamProctoring;| Browser | Version | Support |
|---|---|---|
| Chrome | 90+ | ✅ Full |
| Firefox | 88+ | ✅ Full |
| Safari | 14+ | ✅ Full |
| Edge | 90+ | ✅ Full |
| Opera | 76+ | ✅ Full |
- WebRTC - For camera access
- Web Audio API - For microphone access
- WebGL - For MediaPipe GPU acceleration
- ES6+ - Modern JavaScript features
// Camera permissionawaitnavigator.mediaDevices.getUserMedia({video: true});// Microphone permission (handled internally)// Requested automatically by AudioMonitoringModule- Adjust Detection FPS
// Lower FPS for better performanceengine.updateOptions({detectionFPS: 5});- Increase Stability Frames
// Fewer false positives, better performanceengine.updateOptions({stabilityFrames: 20});- Selective Modules
// Only enable what you needProctoringEngine.getInstance({enableVisualDetection: true,enableAudioMonitoring: false,// Disable if not neededenablePatternDetection: true,enableBrowserTelemetry: true});- GPU Acceleration Ensure WebGL is enabled for MediaPipe GPU acceleration.
| Configuration | CPU Usage | Memory | Accuracy |
|---|---|---|---|
| Low (5 FPS) | ~10% | ~150MB | Good |
| Medium (10 FPS) | ~20% | ~200MB | Better |
| High (15 FPS) | ~30% | ~250MB | Best |
Tested on Intel i5, 8GB RAM, Chrome 120
- ✅ Client-Side Processing - All detection runs in browser
- ✅ No Cloud Dependencies - MediaPipe models loaded from CDN
- ✅ Secure Transmission - Use HTTPS for backend communication
- ✅ No Recording - Video/audio analyzed in real-time, not stored
- ✅ Configurable - Choose which data to send to backend
- Obtain Explicit Consent
// Show consent dialog before startingconstconsent=awaitshowConsentDialog();if(consent){awaitproctor.start();}- Use HTTPS
// Always use secure connectionsfetch('https://api.example.com/proctoring/event',{method: 'POST',// ...});- Implement Data Retention Policies
// Clear logs after examwindow.addEventListener('beforeunload',()=>{engine.clearLogs();});- Provide Accommodations
// Adjust for students with disabilitiesengine.updateOptions({gazeThreshold: 35,// More lenientprolongedGazeAwayDuration: 10000});// Check permissionnavigator.permissions.query({name: 'camera'}).then(result=>{console.log('Camera permission:',result.state);if(result.state==='denied'){alert('Please allow camera access');}});// Check microphone permissionnavigator.permissions.query({name: 'microphone'}).then(result=>{console.log('Microphone permission:',result.state);});// Check if audio module initializedif(!engine.audioModule.isSetup){console.warn('Audio monitoring not available');}// Reduce detection FPSengine.updateOptions({detectionFPS: 5});// Disable unnecessary modulesengine.updateOptions({enableAudioMonitoring: false});// Check network connection// MediaPipe models loaded from CDN// Ensure CDN is accessible: https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/Contributions are welcome! Please follow these guidelines:
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
# Clone repository
git clone https://github.com/yourusername/ai-proctoring-engine.git
# Install dependenciescd ai-proctoring-engine
npm install
# Run tests
npm test# Build
npm run buildThis project is licensed under the MIT License - see the LICENSE file for details.
- MediaPipe - Computer vision framework
- Web Audio API - Audio processing
- Issues: GitHub Issues
- Discussions: GitHub Discussions
- Email: support@example.com
- TypeScript support
- Hand tracking for gesture detection
- Mobile support
- Offline mode with model caching
- Dashboard for supervisors
- API for third-party integrations
- Automated report generation
- Multi-language support
Made with ❤️ for fair and secure online examinations