OmniAgent

Your AI representative across communication channels.

OmniAgent is a personal AI assistant that routes messages across multiple communication platforms, processes them via an AI agent, and responds on your behalf.

Features

💬 Multi-Channel Support - Telegram, Discord, Slack, WhatsApp, and more
🤖 AI-Powered Responses - Powered by omnillm (Claude, GPT, Gemini, etc.)
🎤 Voice Notes - Transcribe incoming voice, respond with synthesized speech via OmniVoice
📞 Full-Duplex Phone Calls - Real-time phone conversations via Twilio Media Streams
⚡ Native Voice-to-Voice - Ultra-low latency (~100-300ms) via OpenAI Realtime, Gemini Live, or Deepgram Agent APIs
🧩 Skills System - Markdown skills (OpenClaw compatible) and compiled Go skills
💾 Persistent Sessions - Conversation history with SQLite storage via omnistorage-core
🧠 Semantic Memory - Multi-backend memory with automatic recall via omnimemory
⏰ Scheduled Jobs - Cron expressions, intervals, and one-time job scheduling
🔒 Secure Sandboxing - WASM and Docker isolation with GPU passthrough
🌐 Browser Automation - Built-in browser control with dialog handling via Rod
🔌 WebSocket Gateway - Real-time control plane with tools RPC endpoint
📊 Observability - Integrated tracing via omniobserve
🎭 Agent Profiles - Bootstrap profiles and lean mode for resource optimization
🛡️ Access Policies - Per-sender tool access control and channel conformance
🔐 Vault Credentials - Secure credential storage via 1Password, Bitwarden, file, or environment
🔑 Skill Secrets - GitHub-style requires.secrets declaration in SKILL.md, global/per-skill config bindings, per-skill-type injection (MCP, OpenAPI, compiled), required-secret gating, and log/output redaction
🔗 OpenAI-Compatible API - Drop-in replacement for OpenAI client libraries with SSE streaming
🖼️ Image Generation - AI image generation via OpenAI (DALL-E) or Fal AI (FLUX)
👥 Multi-Agent Support - Run multiple agents with different models and configurations
🏢 Team Mode - Multi-user deployments with magic-link, email+password, or Google/GitHub sign-in, an admin UI, private/group chats, and PostgreSQL row-level isolation
🪪 Virtual Agents - Named personas with per-agent skills, agent-scoped secrets, owner/maintainer roles, and a discoverable catalog
🛒 Agent Marketplace Primitives - Reusable agent and skill listings, filters, and provider interfaces for host apps such as UIForge
📈 Usage Analytics - Token usage tracking, tool call statistics, and cost estimation
🔧 Tool Visualization - Real-time tool call display with arguments and results in web UI

Ways to Connect

OmniAgent supports multiple connection methods for different use cases:

Method	Protocol	Use Case	Latency
WhatsApp	WebSocket	Personal messaging, voice notes	Text: instant, Voice: ~2s
Telegram/Discord/Slack	Bot API	Team messaging, notifications	Instant
OpenAI-Compatible API	HTTP/SSE	Programmatic access, web apps	Streaming
Phone Calls	Twilio/Telnyx	PSTN voice conversations	Traditional: 500ms+, Realtime: ~100ms
LiveKit (WebRTC)	WebRTC	Browser/mobile voice, meetings	~100-300ms

Quick links:

WhatsApp Setup - Scan QR code to connect
OpenAI API - Use any OpenAI client library
Voice Gateway - Phone calls via Twilio/Telnyx
LiveKit Agents - WebRTC voice in meetings

Screenshots

Stock Skill via OmniAgent Web Interface

Search Skill via WhatsApp

Installation

go install github.com/plexusone/omniagent/cmd/omniagent@latest

Quick Start

WhatsApp + OpenAI

The fastest way to get started is with WhatsApp and OpenAI:

# Set your OpenAI API keyexport OPENAI_API_KEY="sk-..."# Run with WhatsApp enabled
OMNIAGENT_AGENT_PROVIDER=openai \
OMNIAGENT_AGENT_MODEL=gpt-4o \
WHATSAPP_ENABLED=true \
omniagent gateway run

A QR code will appear in your terminal. Scan it with WhatsApp (Settings -> Linked Devices -> Link a Device) to connect.

Configuration File

For more control, create a configuration file:

# omniagent.yamlgateway:
address: "127.0.0.1:18789"agent:
provider: openai # or: anthropic, geminimodel: gpt-4o # or: claude-sonnet-4-20250514, gemini-2.0-flashapi_key: ${OPENAI_API_KEY}system_prompt: "You are OmniAgent, responding on behalf of the user."channels:
whatsapp:
enabled: truedb_path: "whatsapp.db"# Session storagetelegram:
enabled: falsetoken: ${TELEGRAM_BOT_TOKEN}discord:
enabled: falsetoken: ${DISCORD_BOT_TOKEN}twilio_sms:
enabled: falseaccount_sid: ${TWILIO_ACCOUNT_SID}auth_token: ${TWILIO_AUTH_TOKEN}phone_number: ${TWILIO_PHONE_NUMBER}messaging_service_sid: ${TWILIO_MESSAGING_SERVICE_SID} # Optional: enables RCS with SMS/MMS fallbackwebhook_path: /webhook/twilio/sms # Supports SMS, MMS, and RCSvoice:
enabled: trueresponse_mode: auto # auto, always, never# Option 1: Native voice-to-voice (lowest latency, ~100-300ms)realtime:
provider: openai # or: gemini, deepgramvoice: alloy # OpenAI: alloy, nova, etc. Gemini: Puck, Charon, etc. Deepgram: aura-2-thalia-en, etc.# Option 2: Traditional pipeline (custom STT/TTS providers)# stt:# provider: deepgram# model: nova-2# tts:# provider: elevenlabs# voice_id: your-voice-idskills:
enabled: truepaths: # Additional skill directories
- ~/.omniagent/skillsmax_injected: 20# Max skills to inject into prompt

Run with the config file:

omniagent gateway run --config omniagent.yaml

OpenAI-Compatible API

OmniAgent exposes an OpenAI-compatible REST API, allowing you to use standard OpenAI client libraries:

# Start the gateway with API enabled
omniagent gateway run --config omniagent.yaml
# Use with any OpenAI client
curl http://localhost:18789/openai/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $API_KEY" \
-d '{ "model": "omniagent", "messages": [{"role": "user", "content": "Hello!"}], "stream": true }'

API Endpoints

Endpoint	Description
`POST /openai/v1/chat/completions`	Chat completions with SSE streaming
`GET /openai/v1/models`	List available models
`POST /openai/v1/images/generations`	Generate images from prompt
`POST /openai/v1/images/edits`	Edit images with mask
`POST /openai/v1/images/variations`	Create image variations
`GET /api/v1/tools`	List registered tools
`GET /api/v1/agents`	List configured agents
`POST /api/v1/agents`	Create a new agent
`GET /api/v1/cron/jobs`	List scheduled jobs
`GET /api/health`	Health check
`GET /docs`	Interactive API documentation (Scalar)
`GET /api/openapi.json`	OpenAPI 3.1 specification

Python Example

fromopenaiimportOpenAIclient=OpenAI(
base_url="http://localhost:18789/openai/v1",
api_key="your-api-key"
)
response=client.chat.completions.create(
model="omniagent",
messages=[{"role": "user", "content": "What tools do you have?"}],
stream=True
)
forchunkinresponse:
print(chunk.choices[0].delta.content, end="")

See OpenAI API Guide for detailed documentation.

Skills

OmniAgent supports skills compatible with the OpenClaw SKILL.md format. Skills extend the agent's capabilities by injecting domain-specific instructions into the system prompt.

Managing Skills

# List all discovered skills
omniagent skills list
# Show details for a specific skill
omniagent skills info sonoscli
# Check requirements for all skills
omniagent skills check

Skill Format

Skills are defined in SKILL.md files with YAML frontmatter:

---name: weatherdescription: Get weather forecastsmetadata:
emoji: "🌤️"requires:
bins: ["curl"]secrets:
- name: WEATHER_API_KEYdescription: API key for the weather providerrequired: trueinstall:
- name: curlbrew: curlapt: curl---# Weather Skill
You can check the weather using the `curl` command...

Skill Discovery

Skills are discovered from:

Built-in skills directory
~/.omniagent/skills/
Custom paths via skills.paths config

Skills with missing requirements (binaries, env vars) are automatically skipped. A skill that declares a required secret (requires.secrets) with no matching binding — see Secrets below — is skipped the same way, with a logged reason instead of failing later at call time.

Compiled Skills

For better performance and type safety, register Go functions as LLM tools:

import (
"github.com/plexusone/omniagent/agent""github.com/plexusone/omniagent/skills/compiled"
)
// Create a skill with toolstypeWeatherSkillstruct{}
func (s*WeatherSkill) Name() string { return"weather" }
func (s*WeatherSkill) Description() string { return"Weather forecasts" }
func (s*WeatherSkill) Tools() []compiled.Tool {
return []compiled.Tool{{
Name: "get_weather",
Description: "Get weather for a location",
Parameters: map[string]compiled.Parameter{
"location": {Type: "string", Required: true},
},
Handler: func(ctx context.Context, paramsmap[string]any) (any, error) {
returnfetchWeather(params["location"].(string))
},
}}
}
func (s*WeatherSkill) Init(ctx context.Context) error { returnnil }
func (s*WeatherSkill) Close() error { returnnil }
// Register with agentagent.New(config, agent.WithCompiledSkill(&WeatherSkill{}))

Remote Skills

Remote skills connect to external services and expose their capabilities as agent tools.

MCP Skills - Spawn external MCP servers and expose their tools:

import"github.com/plexusone/omniagent/skills/remote/mcp"agent, err:=agent.New(config,
agent.WithMCPSkill(mcp.Config{
Name: "github",
Command: []string{"npx", "-y", "@modelcontextprotocol/server-github"},
Env: map[string]string{
"GITHUB_TOKEN": os.Getenv("GITHUB_TOKEN"),
},
}),
)

OpenAPI Skills - Parse OpenAPI 3.x specs and expose operations as tools:

import openapi "github.com/plexusone/omniagent/skills/remote/openapi"agent, err:=agent.New(config,
agent.WithOpenAPISkill(openapi.Config{
Name: "petstore",
SpecURL: "https://petstore3.swagger.io/api/v3/openapi.json",
Auth: openapi.AuthConfig{
Type: openapi.AuthBearer,
Token: os.Getenv("API_TOKEN"),
},
}),
)

See the Skills Guide for configuration options.

Skill Secrets

A skill declares the secrets it needs in SKILL.md frontmatter (requires.secrets, shown above); OmniAgent supplies them, GitHub-Actions style. Values come from global or per-skill config bindings, resolved the same way as any other credential (plain values or op:///bw:/// file:///env:// vault URIs):

# omniagent.yamlsecrets:
GITHUB_TOKEN: "op://Shared/github/token"# global bindingskills:
config:
github:
secrets:
GITHUB_TOKEN: "env://GITHUB_TOKEN_OVERRIDE"# wins over the global binding

Resolved values are injected into MCP subprocess environments, OpenAPI auth, and compiled skills via agent.WithSecretEnv; a skill with an unmet required secret is excluded from the loaded set (with a logged reason) instead of loading and failing later. Every resolved value — vault-backed or plain — is masked out of log output, including omniagent config show. In team mode, secrets are instead managed per-agent from the web UI (see Team Mode below); superadmins get a read-only view of the global config bindings under Admin. Full reference: Secrets.

Roles

Roles are high-level agent personas that combine skills, workflows, and system prompts into cohesive behaviors. They separate organizational responsibilities from runtime implementations.

import (
"github.com/plexusone/omniagent/agent""github.com/plexusone/omniagent/agent/roles"
facilitator "github.com/plexusone/omnirole-facilitator"
)
// Create a role with configurationpmRole:=facilitator.New(facilitator.Config{
DefaultConfluenceSpace: "TEAM",
EnableActionTracking: true,
})
// Create role manager with skillsmgr, _:=roles.NewManager(pmRole, meetingSkill, googleSkill, confluenceSkill)
mgr.Init(ctx)
defermgr.Close()
// Access role capabilitiesprompt, _:=mgr.SystemPrompt(ctx)
workflows:=mgr.Workflows()
spec:=mgr.Spec()
// Use policy enforcementiferr:=mgr.CheckToolAccess(ctx, "confluence_publish"); err!=nil {
// Tool access denied by policy
}
// Context-aware behaviorsmgr.SetBehaviorContext(role.BehaviorContextMeeting)
behaviors:=mgr.GetActiveBehaviors(ctx)
// Track metricsmgr.RecordMetric(ctx, "meetings-facilitated", 1)

Available Roles

Role	Package	Description
Meeting PM	`github.com/plexusone/omnirole-facilitator`	Meeting facilitation, notes, action tracking

Role Features

Feature	Description
Behaviors	Context-aware actions (meeting, chat, autonomous)
Policies	Tool access control, data access, rate limits
Metrics	KPIs and success measurements
Delegation	Sub-agent orchestration
Workflows	Structured multi-step operations

See the Roles Guide for complete documentation.

Team Mode

Team mode turns a single-operator deployment into a multi-user one: real accounts, allowlist-closed sign-in (magic-link email, email+password, or Google/GitHub), private and group chats, and virtual agents that people discover in a catalog and chat with.

# omniagent.yamlteam:
enabled: truesuperadmin_email: you@example.combase_url: https://team.example.comdatabase:
app_dsn: postgres://omniagent_app:pw@db:5432/omniagent_teammigrate_dsn: postgres://owner:pw@db:5432/omniagent_teamsmtp:
host: smtp.example.comport: 587from: agent@example.comweb:
enabled: trueagent:
provider: anthropicmodel: claude-sonnet-5# api_key: set via ANTHROPIC_API_KEY (loadEnv falls back to the# provider-specific env var when api_key is left unset here)

omniagent gateway run --config omniagent.yaml

The gateway migrates the database (schema and PostgreSQL row-level security), serves the SPA at /, and emails magic links (or logs them in dev when no SMTP is set). The account in superadmin_email bootstraps as superadmin on first sign-in, and gets an Admin tab to manage the allowlist and members.

Virtual agents. In team mode an agent is a first-class entity: a persona + a chosen subset of the deployment's skills + agent-scoped secrets, with per-agent owner/maintainer roles (independent of chat membership — conversing never grants configuration) and a private/listed + featured registry. Owners configure agents under My Agents, anyone browses the Catalog and starts a DM or group, and superadmins promote agents under Curation.

Uses PostgreSQL for production (row-level security isolation); a non-postgres://app_dsn selects SQLite for local trials only.
Agent-scoped secrets (team.secrets) are namespaced per agent so two agents load disjoint secrets with no cross-leak; superadmins get a read-only view of the deployment-wide config bindings (names and set-state, never values) under Admin.
Email+password sign-in (argon2id-hashed) is available alongside magic-link, seeded by an optional team.superadmin_password bootstrap.
Optional Google OIDC and GitHub OAuth sign-in (team.sso.*), additive to magic-link email — an SSO identity links by verified email to an existing account rather than creating a duplicate.
A production Docker Compose stack (Caddy + omniagent + PostgreSQL) for a single-VM deployment lives under deploy/team/prod/.

See the Team Mode guide, Virtual Agents guide, and Agent Marketplace guide, Team Deployment guide for the full walkthrough.

Sessions

OmniAgent supports persistent conversation sessions. omniagent gateway run wires this up from config — no code required:

storage:
type: sqlite # or redis, memorypath: /data/omniagent.dbsessions:
enabled: true

Embedding binaries that build their own *agent.Agent use the library API:

import (
"github.com/plexusone/omniagent/agent""github.com/plexusone/omnistorage-core/kvs/backend/sqlite"
)
// Create storage backendbackend, _:=sqlite.New(sqlite.Config{Path: "omniagent.db"})
// Create agent with sessionsa, _:=agent.New(config,
agent.WithSessionsFromStorage(backend),
)
// Process with conversation historyresponse1, _:=a.ProcessWithSession(ctx, "user-123", "My name is Alice")
response2, _:=a.ProcessWithSession(ctx, "user-123", "What's my name?")
// Agent remembers: "Your name is Alice"

See Sessions Guide for details.

Scheduled Jobs

OmniAgent supports scheduled job execution via the cron package. omniagent gateway run starts the scheduler automatically in single-agent mode, using the same storage: config as Sessions above. The library API, for embedding binaries:

import (
"github.com/plexusone/omniagent/agent""github.com/plexusone/omnistorage-core/kvs/backend/sqlite"
)
// Create agent with cron supportbackend, _:=sqlite.New(sqlite.Config{Path: "omniagent.db"})
a, _:=agent.New(config,
agent.WithSessionsFromStorage(backend),
agent.WithCronScheduler(),
)

The LLM can then create scheduled jobs via tool calls:

Tool	Description
`cron_create`	Create a new scheduled job
`cron_list`	List all jobs (filterable by status)
`cron_get`	Get job details
`cron_delete`	Delete a job
`cron_enable`	Enable a disabled job
`cron_disable`	Disable without deleting
`cron_trigger`	Run job immediately

Schedule types:

Cron expressions: 0 0 9 * * * (9am daily, with seconds)
Intervals: 1h, 30m, 24h
One-time: RFC3339 timestamp for single execution

Action types:

send_message - Send a message to a session
call_webhook - Make an HTTP request
call_tool - Invoke a registered tool

See Cron Guide for details.

Sandboxing

OmniAgent provides layered security for tool execution:

App-Level Permissions

Capability-based permissions control what tools can do:

fs_read - Read files from allowed paths
fs_write - Write files to allowed paths
net_http - Make HTTP requests to allowed hosts
exec_run - Execute allowed commands

Docker Isolation

For OS-level isolation, tools can run inside Docker containers:

sandbox, _:=sandbox.NewDockerSandbox(ctx, sandbox.DockerConfig{
Image: "alpine:latest",
NetworkMode: "none", // No network accessCapDrop: []string{"ALL"}, // Drop all capabilitiesMounts: []sandbox.DockerMount{
{HostPath: "/tmp/data", ContainerPath: "/data", ReadOnly: true},
},
}, &appConfig)
result, _:=sandbox.Run(ctx, "cat", []string{"/data/file.txt"})

GPU Passthrough

For GPU-accelerated workloads, enable NVIDIA GPU passthrough:

sandbox, _:=sandbox.NewDockerSandbox(ctx, sandbox.DockerConfig{
Image: "nvidia/cuda:12.0-base",
GPU: &sandbox.GPUConfig{
Enabled: true,
DeviceIDs: []string{"0"},
Capabilities: []string{"compute", "utility"},
},
})

WASM Runtime

For lightweight isolation, tools can run in a WASM sandbox (wazero):

runtime, _:=sandbox.NewRuntime(ctx, sandbox.Config{
Capabilities: []sandbox.Capability{sandbox.CapFSRead},
MemoryLimitMB: 16,
Timeout: 30*time.Second,
AllowedPaths: []string{"/tmp/data"},
})

Agent Profiles

Profiles customize agent behavior for different use cases:

import"github.com/plexusone/omniagent/agent/profiles"profile:=&profiles.BootstrapProfile{
Name: "customer-support",
SystemPromptPrefix: "You are a customer support agent.\n",
AllowedTools: []string{"search_kb", "create_ticket"},
DeniedTools: []string{"shell", "browser"},
}
a, _:=agent.New(config, agent.WithProfile(profile))

Lean Mode

Optimize for constrained environments:

leanMode:=profiles.NewLeanMode(profiles.LeanLevelModerate)
a, _:=agent.New(config, agent.WithLeanMode(leanMode))

Level	Memory Reduction	Use Case
`Off`	None	Default operation
`Light`	~15%	Slightly constrained
`Moderate`	~35%	Mobile/embedded
`Aggressive`	~60%	Severely constrained

See Agent Profiles Guide for details.

Access Policies

Tool Policies

Control which tools are available per sender:

import"github.com/plexusone/omniagent/tools/policy"manager:=policy.NewManager()
manager.SetPolicy("guest", &policy.Policy{
AllowedTools: []string{"search", "weather"},
DeniedTools: []string{"shell", "browser"},
RateLimit: &policy.RateLimit{
MaxCalls: 10,
Window: time.Minute,
},
})

Channel Policies

Validate messages against content rules:

import"github.com/plexusone/omniagent/channels/policy"checker:=policy.NewConformanceChecker(config)
checker.AddRule(policy.ConformanceRule{
Name: "rate-limit",
Action: policy.ActionRateLimit,
RateLimit: &policy.RateLimit{MaxMessages: 60, Window: time.Minute},
})

See Access Policies Guide for details.

Environment Variables

Variable	Description
`OPENAI_API_KEY`	OpenAI API key
`ANTHROPIC_API_KEY`	Anthropic API key
`GEMINI_API_KEY`	Google Gemini API key
`OPENROUTER_API_KEY`	OpenRouter API key
`OMNIAGENT_AGENT_PROVIDER`	LLM provider: `openai`, `anthropic`, `gemini`, `openrouter`
`OMNIAGENT_AGENT_MODEL`	Model name (e.g., `gpt-4o`, `claude-sonnet-4-20250514`)
`WHATSAPP_ENABLED`	Set to `true` to enable WhatsApp
`WHATSAPP_DB_PATH`	WhatsApp session storage path
`TELEGRAM_BOT_TOKEN`	Telegram bot token (auto-enables Telegram)
`DISCORD_BOT_TOKEN`	Discord bot token (auto-enables Discord)
`TWILIO_ACCOUNT_SID`	Twilio Account SID (auto-enables SMS/MMS/RCS)
`TWILIO_AUTH_TOKEN`	Twilio Auth Token
`TWILIO_PHONE_NUMBER`	Twilio phone number in E.164 format
`TWILIO_MESSAGING_SERVICE_SID`	Messaging Service SID for RCS (enables RCS with SMS/MMS fallback)
`TWILIO_WEBHOOK_PATH`	SMS webhook path (default: `/webhook/twilio/sms`)
`SERPER_API_KEY`	Serper API key for web search
`SERPAPI_API_KEY`	SerpAPI key for web search (alternative)
`DEEPGRAM_API_KEY`	Deepgram API key for voice STT/TTS (traditional)
`OMNIAGENT_VOICE_ENABLED`	Set to `true` to enable voice processing
`OMNIAGENT_VOICE_RESPONSE_MODE`	Voice response mode: `auto`, `always`, `never`
`OMNIAGENT_VOICE_REALTIME_PROVIDER`	Native voice-to-voice: `openai`, `gemini`, `deepgram`
`ELEVENLABS_API_KEY`	ElevenLabs API key for voice TTS (traditional)
`GOOGLE_API_KEY`	Google API key for Gemini Live (native voice-to-voice)
`IMAGE_ENABLED`	Set to `true` to enable image generation
`IMAGE_PROVIDER`	Image provider: `openai`, `fal` (default: `openai`)
`IMAGE_MODEL`	Default image model (e.g., `gpt-image-2`, `fal-ai/flux-pro`)
`FAL_KEY`	Fal AI API key for image generation
`LIVEKIT_URL`	LiveKit server URL (e.g., `wss://your-project.livekit.cloud`)
`LIVEKIT_API_KEY`	LiveKit API key
`LIVEKIT_API_SECRET`	LiveKit API secret
`REALTIME_PROVIDER`	Realtime voice provider: `openai`, `gemini`, `deepgram`
`REALTIME_VOICE`	Voice for realtime API (provider-specific)
`AVATAR_PROVIDER`	Avatar mode: `""` (none), `static`, `tavus`
`AVATAR_IMAGE_PATH`	Static avatar image path (for `static` mode)
`TAVUS_API_KEY`	Tavus API key (for `tavus` mode)
`TAVUS_PAL_ID`	Tavus PAL ID (optional, uses default if not set)
`TAVUS_FACE_ID`	Tavus Face ID override (optional)

Vault-Backed Credentials

OmniAgent supports storing credentials in password managers via omnivault and omnitoken.

Supported Vault Providers

Provider	URI Scheme	Environment Variable
1Password	`op://`	`OP_SERVICE_ACCOUNT_TOKEN`
Bitwarden	`bw://`	`BW_ACCESS_TOKEN`, `BW_ORGANIZATION_ID`
File	`file://`	-
Environment	`env://`	-

Static Credentials

API keys and tokens can be stored in vaults instead of config files:

# omniagent.yamlagent:
provider: anthropicmodel: claude-sonnet-4-20250514api_key: "op://MyVault/anthropic/api-key"# Resolved from 1Passwordchannels:
telegram:
enabled: truetoken: "bw://org-id/telegram-bot-token"# Resolved from Bitwardendiscord:
enabled: truetoken: "env://DISCORD_BOT_TOKEN"# Resolved from environmentvoice:
enabled: truestt:
provider: deepgramapi_key: "op://MyVault/deepgram/api-key"tts:
provider: deepgramapi_key: "op://MyVault/deepgram/api-key"

Credentials are resolved once at startup. Plain string values still work for development.

OAuth Token Management

For services requiring OAuth token refresh (Google, Zoom, RingCentral), use the tokens configuration:

# omniagent.yamltokens:
vault_uri: "op://MyVault"services:
google:
credentials_name: "google-service-account"scopes:
- "https://www.googleapis.com/auth/calendar"zoom:
credentials_name: "zoom-oauth"ringcentral:
credentials_name: "ringcentral-oauth"

The token manager handles:

In-memory token caching
Automatic refresh when tokens expire
Vault coordination for multi-process deployments
Refresh token persistence

Vault Environment Variables

Variable	Provider	Description
`OP_SERVICE_ACCOUNT_TOKEN`	1Password	Service account token (starts with `ops_`)
`BW_ACCESS_TOKEN`	Bitwarden	Access token
`BW_ORGANIZATION_ID`	Bitwarden	Organization ID
`BW_API_URL`	Bitwarden	Custom API URL (self-hosted)
`BW_IDENTITY_URL`	Bitwarden	Custom Identity URL (self-hosted)

CLI Commands

# Gateway
omniagent gateway run # Start the gateway server# Setup & Diagnostics
omniagent setup # Interactive setup wizard
omniagent doctor # Diagnose configuration and connectivity# Voice (Full-Duplex Phone Calls)
omniagent voice serve # Start the voice gateway server
omniagent voice status # Show voice configuration status
omniagent voice call NUM # Make an outbound call to NUM# Skills
omniagent skills list # List all discovered skills
omniagent skills info NAME # Show skill details
omniagent skills check # Validate skill requirements# Sessions
omniagent sessions list # List conversation sessions
omniagent sessions show ID # Show session details
omniagent sessions delete ID # Delete a session# Channels
omniagent channels list # List registered channels
omniagent channels status # Show channel connection status# OpenAI API
omniagent openai spec # Generate OpenAPI specification# Config
omniagent config show # Display current configuration# Version
omniagent version # Show version information

Voice Gateway

Start a full-duplex voice gateway for phone calls:

# Native voice-to-voice (lowest latency, ~100ms)
omniagent voice serve \
--listen :8081 \
--public-url https://your-server.com \
--realtime openai \
--realtime-voice alloy
# Traditional pipeline (custom STT/TTS)
omniagent voice serve \
--listen :8081 \
--public-url https://your-server.com \
--stt deepgram \
--tts elevenlabs \
--llm anthropic \
--model claude-sonnet-4-20250514

Configure Twilio webhooks:

Voice URL: https://your-server.com/voice/inbound
Status Callback: https://your-server.com/voice/status

Local Development with ngrok

For local development, use ngrok to expose your local server to Twilio:

# Set ngrok auth tokenexport NGROK_AUTHTOKEN=your-ngrok-token
# Start voice server with ngrok tunnel (auto-generates public URL)
omniagent voice serve \
--listen :8081 \
--ngrok \
--stt deepgram \
--tts elevenlabs \
--llm anthropic

The ngrok public URL will be displayed on startup. Configure this URL in your Twilio webhook settings.

With a custom ngrok domain (requires paid plan):

omniagent voice serve \
--listen :8081 \
--ngrok \
--ngrok-domain myapp.ngrok.io

LiveKit Voice Agents

Run OmniAgent as a voice participant in LiveKit meetings:

# Set credentialsexport LIVEKIT_URL="wss://your-project.livekit.cloud"export LIVEKIT_API_KEY="your-api-key"export LIVEKIT_API_SECRET="your-api-secret"# Option 1: Realtime mode (lowest latency, ~100-300ms)export REALTIME_PROVIDER="deepgram"# or: openai, geminiexport DEEPGRAM_API_KEY="your-deepgram-key"# Option 2: Traditional pipeline (STT→LLM→TTS)# export ANTHROPIC_API_KEY="your-anthropic-key"# export STT_PROVIDER="deepgram"# export STT_API_KEY="your-deepgram-key"# export TTS_PROVIDER="openai"# export TTS_API_KEY="your-openai-key"# Run the generic voice agent
go run ./cmd/livekit-agent
# Or run the meeting facilitator (with Meeting PM role)
go run ./cmd/livekit-agent-facilitator
# Or run multi-agent panel discussions
go run ./cmd/livekit-agent-panel

Command	Description
`cmd/livekit-agent`	Generic voice agent with web search
`cmd/livekit-agent-facilitator`	Meeting facilitator with Meeting PM role, supports realtime mode
`cmd/livekit-agent-panel`	Multi-agent panel discussions with HeyGen avatars, JSON scheduling, slides, and recording

Avatar Configuration

Agents can display visual avatars in video tiles. Multiple providers are supported:

Mode	`AVATAR_PROVIDER`	Description
None	`""` (empty)	Audio-only, no video tile
Static	`static`	Display static image (640x360)
Tavus	`tavus`	Real-time lip-sync via Tavus
HeyGen	`heygen`	Real-time lip-sync via HeyGen (panel agents)

# Static image avatarexport AVATAR_PROVIDER="static"export AVATAR_IMAGE_PATH="./avatar.png"# Optional, has default# Tavus live avatar (lip-sync video)export AVATAR_PROVIDER="tavus"export TAVUS_API_KEY="your-tavus-key"export TAVUS_PAL_ID="your-pal-id"# Optional, uses default# HeyGen avatars (panel agents)export HEYGEN_API_KEY="your-api-key"export HEYGEN_SANDBOX=true # Use sandbox for testingexport MODERATOR_AVATAR_ID="avatar-id-1"export PANELIST_1_AVATAR_ID="avatar-id-2"# Legacy: AGENT_AVATAR=true maps to AVATAR_PROVIDER=static

See omni-livekit avatar documentation and Panel Discussions Guide for detailed setup instructions.

These agents use omni-livekit for LiveKit transport.

Architecture

+-------------------------------------------------------------+
| Messaging Channels |
| Telegram | Discord | Slack | WhatsApp | ... |
+---------------------------+---------------------------------+
|
+---------------------------v---------------------------------+
| Gateway (WebSocket Control Plane) |
| ws://127.0.0.1:18789 |
+---------------------------+---------------------------------+
|
+---------------------------v---------------------------------+
| Agent Runtime |
| +------------------+ +------------------+ |
| | Skills | | Sandbox | |
| | (SKILL.md) | | (WASM/Docker) | |
| +------------------+ +------------------+ |
| - omnillm (LLM providers) |
| - omnivoice (STT/TTS) |
| - omniobserve (tracing) |
| - Tools (browser, shell, http) |
+-------------------------------------------------------------+

Configuration Reference

Gateway

Field	Type	Default	Description
`gateway.address`	string	`127.0.0.1:18789`	WebSocket server address
`gateway.read_timeout`	duration	`30s`	Read timeout
`gateway.write_timeout`	duration	`30s`	Write timeout
`gateway.ping_interval`	duration	`30s`	WebSocket ping interval

Agent

Field	Type	Default	Description
`agent.provider`	string	`anthropic`	LLM provider
`agent.model`	string	`claude-sonnet-4-20250514`	Model name
`agent.api_key`	string	-	API key (or use env var)
`agent.temperature`	float	`0.7`	Sampling temperature
`agent.max_tokens`	int	`4096`	Max response tokens
`agent.system_prompt`	string	-	Custom system prompt

Skills

Field	Type	Default	Description
`skills.enabled`	bool	`true`	Enable skill loading
`skills.paths`	[]string	`[]`	Additional skill directories
`skills.disabled`	[]string	`[]`	Skills to skip
`skills.max_injected`	int	`20`	Max skills in prompt

Voice

Field	Type	Default	Description
`voice.enabled`	bool	`false`	Enable voice processing
`voice.response_mode`	string	`auto`	`auto`, `always`, `never`
`voice.realtime.provider`	string	-	Native voice-to-voice: `openai`, `gemini`, `deepgram`
`voice.realtime.voice`	string	-	Voice for realtime API
`voice.stt.provider`	string	-	STT provider (traditional): `deepgram`, `whisper`
`voice.tts.provider`	string	-	TTS provider (traditional): `elevenlabs`, `deepgram`

Omni* Library Ecosystem

OmniAgent is built on a modular ecosystem of omni* libraries:

 OmniAgent
(Agent Runtime)
┌────────┬────────┬────────┬────────┬────────┬────────┬────────┐
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
omnichat omnillm omnivoice omniimage omniobserve omniserp omnistorage ...
│ │ │ │ │
│ ┌────┴────┐ ┌─┴──┐ │ ┌────────────┴────────┐
│ │ │ │ │ │ │ │
▼ ▼ ▼ ▼ ▼ ▼ ▼ ▼
omnillm-core omnivoice-core omnistorage-core
├── /object (files)
└── /kvs (sessions)
│ │ │ │ │
└────────┴────────┴─────────┴──────────────────────────┘
│
Provider Modules
┌───────────────────┼───────────────────┐
▼ ▼ ▼
omni-aws omni-google omni-github
├── /omnillm ├── /omnillm └── /omnistorage
├── /omnistorage └── /omnistorage
└── /omnivoice

See Architecture Overview for detailed documentation.

Dependencies

Omni* Libraries

Package	Purpose
omnichat	Unified messaging (WhatsApp, Telegram, Discord)
omnillm	Multi-provider LLM abstraction
omnivoice	Voice STT/TTS interfaces
omni-livekit	LiveKit WebRTC voice transport
omni-twilio	Full-duplex voice gateway via Twilio
omni-deepgram	Deepgram STT/TTS and realtime voice
omnimemory	Semantic memory with vector retrieval
omniobserve	LLM observability
omniserp	Web search via Serper/SerpAPI
omniimage	Image generation (OpenAI, Fal AI)
omnistorage-core	Object and key-value storage
omnivault	Secure credential storage
omnitoken	OAuth token management

Infrastructure

Package	Purpose
wazero	WASM runtime for sandboxing
moby	Docker SDK for container isolation
Rod	Browser automation
gorilla/websocket	WebSocket server

Related Projects

omnichat - Unified messaging provider interface
omnillm - Multi-provider LLM abstraction
omnivoice - Voice interactions (STT/TTS)
omni-livekit - LiveKit WebRTC voice transport
omnimemory - Semantic memory with vector retrieval
omniimage - Image generation (OpenAI, Fal AI)
omnirole-facilitator - Meeting PM role
OpenClaw - Compatible skill format

License

MIT License - see LICENSE for details.

Name		Name	Last commit message	Last commit date
.github		.github
agent		agent
api/openai		api/openai
autoreply		autoreply
channels/policy		channels/policy
cmd		cmd
config		config
context		context
cron		cron
deploy		deploy
docs		docs
examples		examples
gateway		gateway
hooks		hooks
internal		internal
marketplace		marketplace
memscope		memscope
openai		openai
platform		platform
sandbox		sandbox
sessions		sessions
skills		skills
team		team
testdata/skills		testdata/skills
tools		tools
voice		voice
web		web
.gitignore		.gitignore
.golangci.yaml		.golangci.yaml
CHANGELOG.json		CHANGELOG.json
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Dockerfile		Dockerfile
Dockerfile.example		Dockerfile.example
ENGINEERING.md		ENGINEERING.md
LICENSE		LICENSE
PLAN.md		PLAN.md
PRD.md		PRD.md
README.md		README.md
ROADMAP.md		ROADMAP.md
TASKS.md		TASKS.md
TRD.md		TRD.md
go.mod		go.mod
go.sum		go.sum
mkdocs.yml		mkdocs.yml

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

OmniAgent

Features

Ways to Connect

Screenshots

Stock Skill via OmniAgent Web Interface

Search Skill via WhatsApp

Installation

Quick Start

WhatsApp + OpenAI

Configuration File

OpenAI-Compatible API

API Endpoints

Python Example

Skills

Managing Skills

Skill Format

Skill Discovery

Compiled Skills

Remote Skills

Skill Secrets

Roles

Available Roles

Role Features

Team Mode

Sessions

Scheduled Jobs

Sandboxing

App-Level Permissions

Docker Isolation

GPU Passthrough

WASM Runtime

Agent Profiles

Lean Mode

Access Policies

Tool Policies

Channel Policies

Environment Variables

Vault-Backed Credentials

Supported Vault Providers

Static Credentials

OAuth Token Management

Vault Environment Variables

CLI Commands

Voice Gateway

Local Development with ngrok

LiveKit Voice Agents

Avatar Configuration

Architecture

Configuration Reference

Gateway

Agent

Skills

Voice

Omni* Library Ecosystem

Dependencies

Omni* Libraries

Infrastructure

Related Projects

License

About

Resources

Stars

Watchers

Forks

Releases

Contributors

Languages