Recall

Recall is a RAG system that checks its episodic memory before hitting the document store. If it already has a confident, fresh answer, it skips retrieval and answers in about 266 ms instead of 49 s.

Core advantages

Speed. When memory hits, the full retrieval‑generation pipeline is bypassed:

Path	Measured latency	What happens
Memory hit	~260 ms	Embed → Chroma lookup → return cached answer
Retrieval + generation	~49 s (cold), ~6 s (warm)	Embed → HyDE expand → doc search → LLM generation via OpenRouter
Reconciled	~5–8 s	Both paths run → LLM arbiter picks the better answer

The first query is always a bit slow – it has to spin up the embedding model and LLM API. The second, identical (or very similar) query hits memory and is 180× faster. That's the core value: repeated questions become almost free.

Scope awareness. A cached answer to “list all state capitals” won’t be served for “capital of Karnataka.” Recall tags each memory with a scope (single entity vs. set) and checks that tag at query time. If the scopes don’t line up, it falls back to the full retrieval path, even if the similarity score is high.

No hallucination on unknowns. When the source docs don’t contain an answer, Recall says so instead of making something up. It also filters out safety‑filter responses (e.g. “User Safety: safe”) so they never get cached.

No extra infrastructure. Chroma runs in‑process as a local persistent store – no Docker, no external vector DB, just a couple of API keys.

Self‑correcting. Source document content is hashed. If the underlying docs change, the memory entry is marked stale and refreshed. A lightweight logistic‑regression confidence model predicts answer quality from similarity, correction history, age, and scope match – it doesn’t rely on raw cosine distance alone.

Observable. Every query returns a latency_ms field in the JSON response. search_memory logs collection size and query latency. When the collection exceeds MAX_MEMORY_ENTRIES, older low‑confidence entries are pruned.

Why this matters

Standard RAG re‑fetches everything on every query. That’s wasteful when you asked the same thing yesterday. Recall treats retrieval as an expensive fallback, not the default – just like you wouldn’t reread a manual you just consulted.

Typical use cases

• Support bots that get the same question over and over – answer from memory, not from scratch.
• Research workflows where you keep asking follow‑up questions about the same PDF.
• Internal knowledge bases where employees repeatedly ask similar questions.
• Any RAG setup where queries overlap significantly.

How it works

query → embed → search_memory → [confident?] → check_staleness → [fresh?] → answer_from_memory
\→ [not confident] → hyde_expand → retrieve_docs → [memory also present?] → reconcile
\→ answer_from_retrieval
(every path) → write_memory (skip if junk) → append to session → END

Every answer gets written to episodic memory with a scope label (“single” or “set”). The next time a similar question arrives, it checks memory first – no retrieval needed. When both memory and documents have something to say, a reconcile node picks the better answer and flags the other as corrected.

Junk filtering. Safety‑filter messages, fallback strings, and error messages are detected and never cached. If a junk answer somehow slips in, it’s evicted at search time and the query falls back to fresh retrieval.

Scope‑mismatch handling. If the query scope doesn’t match the cached memory (e.g. a broad list vs. a narrow request), the memory confidence is forced below the threshold, so the system falls back to retrieval and reconciliation. No LLM arbitration needed for scope mismatches.

Confidence scores combine similarity, correction history, age, and scope match via a logistic‑regression model. Staleness is detected by hashing source docs – if they change, memory is flagged stale and refreshed.

Stack

Component	Role
LangGraph	Async orchestration
Chroma	Two collections: episodic_memory, documents (local, persistent)
Groq	HyDE expansion + reconcile decisions
OpenRouter	Generation pass (streaming)
Langfuse	Tracing (optional, skipped if keys missing)
FastAPI	Async serving, SSE streaming, web UI, PDF/URL upload
Streamlit	Optional chat UI with progressive streaming
Sessions	JSON‑backed conversation persistence under data/sessions/

Setup

uv venv --python 3.11 .venv
uv pip install -r requirements.txt
cp .env.example .env # fill in your API keys

Chroma data lives under CHROMA_PATH (default ./data/chroma). No Docker needed.

Model config

In .env, set your generation model:

# Recommended – reliable, no safety‑filter issues:
OPENROUTER_MODEL=openai/gpt-4o-mini
# Free tier – may hit safety filters on some queries:
OPENROUTER_MODEL=openrouter/free

The HyDE and reconcile nodes use Groq (GROQ_MODEL=llama-3.3-70b-versatile).

Running the web UI (default)

uv run uvicorn app.api:app --reload --reload-exclude '.venv'

Open http://localhost:8000. You’ll see a dark‑mode chat UI with session management in the sidebar. Ingest text, PDFs, or URLs directly from the UI.

Ingest the demo PDF

uv run python scripts/ingest_pdf_demo.py

That pulls down arXiv:2410.12837 (a RAG survey, ~52 KB, 117 chunks).

Demo flow

After you ingest the demo PDF, try these queries in order:

#	Query	Expected	What to watch
1	“What is Retrieval‑Augmented Generation?”	Retrieval path, ~49 s	First query – cold start, full pipeline
2	“What is Retrieval‑Augmented Generation?”	Memory hit, ~260 ms	Exact repeat – cache hit
3	“What is RAG?”	Memory hit, ~375 ms	Near‑duplicate – high similarity to cached memory
4	“What is the capital of France?”	“I don’t have information…”	Unrelated – graceful decline, no hallucination
5	“List all state capitals” then “Capital of Karnataka?”	Broad cached, narrow retrieval	Scope mismatch – broad answer can’t serve narrow query

Key moment: Compare the latency_ms field between query 1 (~49 000 ms) and query 2 (~260 ms). That’s the 180× speedup from memory.

Streamlit UI (optional)

uv run streamlit run app/ui.py

Open http://localhost:8501. The UI streams answers token‑by‑token via SSE and shows latency, confidence, similarity, and a source badge for each message.

Threshold sweep

uv run python eval/thresholds_sweep.py --eval-set eval/eval_set.jsonl

Sweeps confidence thresholds against labeled Q/A pairs using an LLM judge.

Memory consolidation

uv run python scripts/consolidate_memory.py

Finds near‑duplicate memories (cosine > 0.92) and merges them. When the collection exceeds MAX_MEMORY_ENTRIES (default 5000), low‑confidence old entries are pruned automatically.

Latency numbers (local machine)

Collection size	Memory search latency
100 entries	~5 ms
1 000 entries	~15 ms
10 000 entries	~40 ms

The bottleneck is the LLM API calls (HyDE + generation), not the Chroma search. Memory hits skip those calls entirely.

API reference

Method	Path	Description
GET	/	Web UI
POST	/query	Ask a question (blocking, returns full response with latency_ms)
POST	/query/stream	Ask a question (SSE streaming, token‑by‑token)
POST	/docs	Ingest plain text
POST	/docs/pdf	Ingest a PDF
POST	/docs/url	Ingest a URL
GET	/docs	List all ingested source documents
DELETE	/docs/{doc_id}	Delete a source and all its chunks
GET	/health	Health check + Chroma counts
GET	/sessions	List sessions
POST	/sessions	Create a session
GET	/sessions/{id}	Session detail + messages
PATCH	/sessions/{id}	Rename a session
DELETE	/sessions/{id}	Delete a session

Response format (JSON)

{
"answer": "Retrieval‑Augmented Generation (RAG) is a hybrid architecture…",
"source": "memory",
"memory_confidence": 0.999,
"memory_similarity": 0.999,
"is_stale": false,
"reconciled": false,
"latency_ms": 266.0,
"session_id": "..."
}

Streaming format (SSE)

data: {"type": "token", "content": "Retrieval"}
... (more token events) ...
data: {"type": "done", "source": "memory", "memory_confidence": 0.999, "memory_similarity": 0.999, "is_stale": false, "reconciled": false, "latency_ms": 266.0}

Tests

uv run pytest -q