J.A.R.V.I.S

Just A Rather Very Intelligent System

A low-latency voice AI assistant built with LiveKit Agents, featuring:

• Sub-1 second end-to-end latency
• Hybrid STT: Deepgram Nova-3 (primary) + faster-whisper (fallback)
• Natural TTS: ElevenLabs (low-latency, high quality)
• Intelligent LLM: Auto-fallback between Claude and OpenAI (latency-first)
• Wake word: "Jarvis" via Picovoice Porcupine
• Extensible tools: System control, web search, and more

Quick Start

1. Clone and Install

cd J.A.R.V.I.S
cp .env.example .env
# Edit .env with your API keys
make install

2. Configure API Keys

Edit .env with your keys:

# Required
LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=your_key
LIVEKIT_API_SECRET=your_secret
DEEPGRAM_API_KEY=your_key
ELEVENLABS_API_KEY=your_key
# At least one LLM
ANTHROPIC_API_KEY=your_key # For Claude
OPENAI_API_KEY=your_key # For GPT-4# Wake word (optional for LiveKit mode)
PICOVOICE_ACCESS_KEY=your_key
JARVIS_WAKE_WORDS=jarvis
JARVIS_WAKE_SENSITIVITY=0.5
# Optional integrations
TWILIO_ACCOUNT_SID=your_sid
TWILIO_AUTH_TOKEN=your_token
TWILIO_FROM_NUMBER=+15551234567
HOME_ASSISTANT_URL=https://homeassistant.local:8123
HOME_ASSISTANT_TOKEN=your_token
MS_CLIENT_ID=your_client_id
MS_TENANT_ID=common
MS_CALENDAR_ID=
# UI (recommended)
JARVIS_UI_TOKEN=change_me
JARVIS_UI_HOST=127.0.0.1
JARVIS_UI_PORT=8080
JARVIS_UI_ALLOW_REMOTE=false

3. Run

# LiveKit mode (recommended)
make run
# Standalone mode with wake word
make run-standalone
# Or via installed script
jarvis-standalone
# Text command mode (tool runner)
python -m jarvis.text_mode
# Or via installed script
jarvis-text
# Web UI dashboard (local)
jarvis-ui
# macOS menu bar launcher (optional)
jarvis-tray

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ J.A.R.V.I.S │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │
│ │ Wake Word │───▶│ VAD │───▶│ STT │ │
│ │ Porcupine │ │ Silero │ │ Deepgram / Whisper │ │
│ │ ("Jarvis") │ │ (<1ms) │ │ (streaming) │ │
│ └─────────────┘ └─────────────┘ └──────────┬──────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ LLM BRAIN ││
│ │ ┌─────────────────────────────────────────────────────┐ ││
│ │ │ Auto: Claude / OpenAI (latency-first) │ ││
│ │ └─────────────────────────────────────────────────────┘ ││
│ │ │ ││
│ │ ┌─────────────────────────────────────────────────────┐ ││
│ │ │ Tools: System │ Web Search │ Weather │ ... │ ││
│ │ └─────────────────────────────────────────────────────┘ ││
│ └─────────────────────────────────────────────────────────────┘│
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐│
│ │ TTS (streaming) ││
│ │ ElevenLabs (low-latency) ││
│ └─────────────────────────────────────────────────────────────┘│
│ │
└─────────────────────────────────────────────────────────────────┘

Latency Budget

Component	Target	Actual
Wake word	<50ms	~20ms
VAD	<10ms	<1ms
STT (Deepgram)	<300ms	~200ms
LLM (Claude)	<500ms	~400ms
TTS (ElevenLabs)	<150ms	~100ms
Total	<1s	~660ms

Project Structure

J.A.R.V.I.S/
├── jarvis/
│ ├── main.py # Entry point
│ ├── agent.py # LiveKit voice agent
│ ├── config.py # Configuration
│ ├── audio/ # Wake word, VAD
│ ├── stt/ # Speech-to-text
│ ├── llm/ # LLM integration
│ ├── tts/ # Text-to-speech
│ └── tools/ # Agent tools
├── tests/
├── .env.example
├── pyproject.toml
└── Makefile

Available Tools

System Tools

• get_current_time - Get time in any timezone
• open_application - Open apps on your computer
• run_shell_command - Execute shell commands (with safety checks)
• get_system_info - System information
• set_volume - Control system volume
• toggle_dark_mode - Toggle dark mode (macOS)

Web Tools

• web_search - Search the web
• fetch_url - Fetch content from URLs
• get_weather - Get weather for any city
• get_definition - Look up word definitions

Memory / Notes / Tasks

• remember / recall_memory - Long-term memory
• add_note / list_notes - Notes storage
• add_task / list_tasks / complete_task - Task management
• forget_memory_by_tag / forget_memory_before - Memory hygiene

Calendar

• add_calendar_event / list_calendar_events - Local calendar events
• outlook_list_events / outlook_create_event / outlook_delete_event - Outlook calendar

Daily Briefing

• daily_brief - Weather + tasks + calendar snapshot

Smart Home (Home Assistant)

• get_device_state / set_device_state / toggle_device

Routines

• add_routine / list_routines / run_routine - Automation sequences

Contacts

• add_contact / list_contacts / find_contact

Files

• list_files / read_file / write_file / search_files

Code Analysis

• analyze_code / get_project_structure / count_lines / find_todos / diff_files / explain_code

GitHub

• github_list_repos / github_read_file / github_search_code / github_list_issues / github_list_prs

Sandboxed Commands

• run_quick_command - fast allowlisted host execution
• run_sandboxed_command - Docker sandbox (network off + read-only by default)

Calling

• place_call / send_sms (Twilio)
• Calls can use saved contact names when available.

Alarms

• add_alarm / list_alarms / cancel_alarm

macOS

• get_battery_status / get_active_app / reveal_in_finder
• send_notification
• play_music / pause_music / next_track / previous_track / now_playing
• list_apple_calendars / create_apple_calendar_event
• create_apple_reminder

API Key Sources

Service	Free Tier	Sign Up
LiveKit Cloud	50 hrs/month	cloud.livekit.io
Deepgram	$200 credit	console.deepgram.com
ElevenLabs	Paid subscription	elevenlabs.io
Twilio	Pay-as-you-go	twilio.com
Home Assistant	Self-hosted	home-assistant.io
Anthropic	Pay-as-you-go	console.anthropic.com
OpenAI	Pay-as-you-go	platform.openai.com
Picovoice	Free tier	console.picovoice.ai
Microsoft Graph	Free tier	learn.microsoft.com/graph

Development

# Install dev dependencies
make dev
# Run tests
make test# Lint and format
make lint
make format
# Run with debug logging
make run-debug

Benchmarking Models

Run interactive benchmarks (tool calling + speed + your scoring):

make install
jarvis-bench --judge \
--model openai:gpt-4o-mini \
--model openai:gpt-5-nano \
--model openai:gpt-5-mini \
--model anthropic:claude-sonnet-4-20250514

By default, benchmarks run with a focused toolset (only the expected tools for each scenario) to reduce noise and provider payload limits. To benchmark with all tools available:

jarvis-bench --toolset full --model openai:gpt-4o-mini

If jarvis-bench is not on your PATH, run:

python3 -m jarvis.bench.runner --judge --model openai:gpt-4o-mini

Summarize results:

jarvis-bench-report

Auto-judge (LLM-as-judge) example:

jarvis-bench \
--model openai:gpt-4o-mini \
--auto-judge anthropic:claude-sonnet-4-20250514

Notes:

• GPT-5 models may require different parameters; the benchmark runner handles this automatically.
• Auto pass for tool scenarios requires both calling the expected tool and producing a final user-facing answer.

Roadmap

• Smart home integration (Home Assistant)
• Local calendar and notes/tasks
• Phone calling (Twilio)
• Persistent memory/context (local SQLite)
• macOS Calendar + Reminders + Notifications
• Desktop UI
• Mobile app

License

MIT