🐟 Billy Bass AI

An AI-powered animatronic Big Mouth Billy Bass that listens, thinks, and talks back — with synchronized lip movements, head gestures, and a sarcastic personality.

Built on an ESP32 microcontroller with a Python relay server that bridges to ElevenLabs Conversational AI for real-time voice conversations. Also works as a standalone Bluetooth speaker with automatic dance choreography.

Features

• AI Conversations — Say "Hey Jarvis" (or press the button) and Billy responds with real-time voice via ElevenLabs Conversational AI (ASR + LLM + TTS in one pipeline)
• Bluetooth Speaker — Pairs as a standard A2DP receiver with real-time lip-sync to any music you stream
• Adaptive Lip-Sync — RMS-based envelope tracking with fast attack / smooth release for natural mouth movement on both speech and music
• Dance Mode — Detects sustained music playback and triggers head-bobbing dance bursts (10s on, 15s rest) — only after 30s of actual audio energy, not silence
• Web Setup Portal — Captive portal (Billy_Setup AP) for WiFi config, mode selection, and server settings from any phone
• Button Override — Short press = wake/interrupt; long press (4s) = factory reset back to setup portal

Architecture

┌──────────────────────── ESP32 ───────────────────────┐
│ │
│ Mic (INMP441) ──I2S──► Audio Task ──WS──► Server │
│ │
│ Server ──WS──► Audio Task ──I2S──► Amp (MAX98357A) │
│ │
│ Motor Task (60Hz) ──► Mouth PWM (lip-sync) │
│ ──► Head GPIO (dance/gestures) │
│ │
│ State: PORTAL │ BT_STREAMING │ AI_IDLE/LISTEN/REPLY │
└──────────────────────────────────────────────────────┘
│ WebSocket (16kHz PCM + JSON)
▼
┌──────────────── Relay Server ──────────────────┐
│ │
│ openWakeWord ──► ElevenLabs ConvAI (WebSocket) │
│ │ ASR + LLM + TTS │
│ ▼ │
│ PCM audio ─────────► ESP32 │
└──────────────────────────────────────────────────────┘

Hardware

Component	Part	Notes
MCU	ESP32-WROOM-32D	No PSRAM — 520KB SRAM, carefully budgeted
Microphone	INMP441	I2S input, 3.3V
Speaker Amp	MAX98357A	I2S output, 5V
Motor Driver	L298N H-Bridge	Mouth (PWM lip-sync) + Head (directional)
Power	3.7V Li-Po → TP4056 → MT3608	Boosted to 5V; star grounding + decoupling caps
Button	Tactile switch on GPIO 33	INPUT_PULLUP, debounced

Pin Map

Microphone (INMP441) Amplifier (MAX98357A)
SCK = GPIO 14 BCLK = GPIO 26
WS = GPIO 15 LRC = GPIO 25
SD = GPIO 32 DIN = GPIO 22
Motor Driver (L298N) Button
Mouth IN1 = GPIO 18 (PWM) GPIO 33
Mouth IN2 = GPIO 19 (PWM)
Head IN3 = GPIO 21
Head IN4 = GPIO 23

Getting Started

Prerequisites

• PlatformIO (VS Code extension or CLI)
• Python 3.10+
• No wake word API key needed — openWakeWord is fully free
• An ElevenLabs API key + Conversational AI Agent configured in the dashboard

1. Flash the Firmware

git clone https://github.com/serplay/FishAI.git
cd FishAI
pio run --target upload # Build and flash
pio device monitor # Serial output (115200 baud)

2. Configure the Fish

On first boot the ESP32 creates an open WiFi AP called Billy_Setup.

1. Connect to Billy_Setup from your phone — a captive portal appears
2. Pick your WiFi network and enter the password
3. Choose mode: AI Agent or Bluetooth Speaker
4. For AI mode, enter your server's local IP (default port 8765)
5. Save & Reboot

3. Start the Server (AI Agent Mode)

cd server
python -m venv venv &&source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Fill in your ElevenLabs API key and Agent ID in .env
python main.py

The server listens on ws://0.0.0.0:8765/ws by default. The ESP32 connects automatically after reboot.

4. Bluetooth Speaker Mode

No server needed — just pair your phone to "Billy Bass" and play music. The fish lip-syncs automatically and starts dancing after 30 seconds of continuous playback.

Project Structure

FishAI/
├── include/ # ESP32 headers
│ ├── config.h # Pin map, timing, enums, constants
│ ├── audio_manager.h # I2S mic/amp driver
│ ├── motor_control.h # Lip-sync + head motor controller
│ ├── network_manager.h # WiFi, portal, WebSocket client
│ └── button_handler.h # Debounced button (short/long press)
├── src/ # ESP32 sources
│ ├── main.cpp # State machine, FreeRTOS tasks, A2DP
│ ├── audio_manager.cpp # I2S init, read/write
│ ├── motor_control.cpp # RMS engine, PWM control, dance logic
│ ├── network_manager.cpp # Web server routes, WS events, NVS
│ └── button_handler.cpp # ISR-based polling
├── server/ # Python relay server
│ ├── main.py # WS server, session manager
│ ├── pipeline.py # ElevenLabs ConvAI bridge
│ ├── wake_word.py # openWakeWord wake word engine
│ ├── mock_client.py # Desktop mock ESP32 for testing
│ ├── config.py # Env vars and constants
│ ├── requirements.txt # websockets, openwakeword, aiohttp
│ └── .env.example # API key template
└── platformio.ini # Build config, libs, partition table

Configuration

Firmware Constants

Key tuning parameters in include/config.h:

Constant	Default	What it does
RMS_SILENCE_THRESHOLD	80	Audio energy floor — below this, mouth stays closed
LIPSYNC_PEAK_RATIO	1.6	Signal must exceed baseline by 60% to open
LIPSYNC_ATTACK_ALPHA	0.7	Mouth open speed (higher = snappier)
LIPSYNC_RELEASE_ALPHA	0.45	Mouth close speed
BT_DANCE_START_MS	30000	Continuous audio before dance unlocks
BT_DANCE_BURST_MS	10000	Dance duration per burst
BT_DANCE_REST_MS	15000	Rest between bursts

Server Environment

See server/.env.example for all options. Key variables:

Variable	Required	Description
ELEVENLABS_API_KEY	✅	ElevenLabs API key
ELEVENLABS_AGENT_ID	✅	Conversational AI agent ID (from dashboard)
OWW_MODEL_NAMES		Wake word model (default: hey_jarvis_v0.1)
OWW_THRESHOLD		Detection sensitivity (default: 0.5)
SERVER_PORT		Default: 8765

Firmware Dependencies

Managed automatically by PlatformIO:

Library	Version	Purpose
ESP32-A2DP	v1.8.3	Bluetooth A2DP Sink
arduino-audio-tools	v1.0.2	I2S / A2DP integration
ESPAsyncWebServer	latest	Captive portal web server
WebSockets	≥2.6.1	WebSocket client (AI mode)
ArduinoJson	≥7.3.0	JSON parsing

Constraints & Design Decisions

• No PSRAM — all buffers are statically sized or stack-allocated. No malloc in hot paths.
• WiFi ⊕ Bluetooth — mutually exclusive due to RAM. Mode is stored in NVS and requires reboot to switch.
• ~400KB usable heap after FreeRTOS + radio stack. The firmware uses huge_app.csv partitions (~3MB app, no OTA).
• ElevenLabs ConvAI — the server is a thin bridge; ASR, LLM, TTS, and turn-taking are all handled by ElevenLabs. Agent personality is configured in the dashboard.
• Star grounding + decoupling caps — prevents motor-induced brownouts from crashing the ESP32 on battery power.