rag-cli

A small RAG tool in Go: embed markdown files into Postgres/pgvector, ask questions against them via any OpenAI-compatible API, and grade the answers with an LLM judge.

Requirements

• Go 1.26+
• Docker (for the pgvector database)
• An OpenAI-compatible endpoint (defaults assume LM Studio at http://localhost:1234/v1)

Quick start

docker compose up -d # start pgvector on :5432
make migrate-up # create the schema
make build # build ./rag-cli
make embed # embed markdown from ./source
./rag-cli rag "your question"

Migrations

The schema lives in migrations/ and is managed by golang-migrate — the Go code never creates or alters tables, it only fails fast if the schema is missing. Install the CLI once:

go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

Target	Description
make migrate-up	Apply pending migrations
make migrate-down	Roll back one migration (N=2, or N=all)
make migrate-reset	Roll everything back and re-apply it — a fresh, empty table
make migrate-version	Print the current schema version
make migrate-force V=1	Clear a dirty state after a failed migration
make NAME=add_index migrate-create	Scaffold a new .up.sql/.down.sql pair

embed assumes it is filling a fresh table, so re-embedding from scratch is make migrate-reset && make embed.

Commands

Command	Description
rag-cli embed [dir]	Chunk and embed markdown files (default ./output). Assumes a fresh table.
rag-cli rag <query>	Retrieve context and answer a single question. -j for JSON output.
rag-cli batch -f questions.json -o answers.json	Answer every question in a file.
rag-cli evaluate -f answers.json -g questions.json	Grade answers against ground truth.

Unanswered questions

Every JSON result carries an error field. It is empty when the model answered, and holds the reason when it did not — a timeout, a transport failure, an empty response — with answer left blank:

{
"question": "What is the capital of France?",
"answer": "",
"documents": ["<document source=\"geo.md\">…</document>"],
"error": "failed to generate response: attempt 2 exceeded 1m0s: context deadline exceeded"
}

Retrieval still ran, so documents shows what the question was going to be answered from. A failure earlier than generation — embedding the query, reaching the store — is a hard error instead and stops the command.

batch records the failure against that question and keeps going; rag -j prints the JSON and exits non-zero. evaluate fails such a question on all four metrics without spending a judge call on it, so it stays in result_count and is counted in error_count.

Make targets

make embed, make batch, and make evaluate wrap the above with sensible timeouts and model settings. Override anything on the command line:

make ANSWER_MODEL=qwen/qwen3.5-9b QUESTIONS=test/questions.json batch
make help# full list of variables

Configuration

Settings come from environment variables or flags — flags win. Every setting below has a matching lower-case flag (TOP_K → --top-k); rag-cli --help lists them with their defaults.

Env var	Default	Description
DATABASE_URL	—	Postgres connection string (required)
LLM_BASE_URL	http://localhost:1234/v1	OpenAI-compatible endpoint
LLM_TOKEN	not-needed	API key, if your endpoint wants one
EMBED_MODEL	text-embedding-nomic-embed-text-v1.5	Embedding model
ANSWER_MODEL	qwen/qwen3.5-9b	Model used to answer
EVAL_MODEL	—	Model used to grade
CHUNK_SIZE / CHUNK_OVERLAP	500 / 50	Chunking
TOP_K	10	Chunks retrieved per query
CONCURRENCY	4	Parallel embedding/answering
MAX_TOKENS	4096	Generation cap
REQUEST_TIMEOUT	60s	Per-attempt budget
MAX_RETRIES / RETRY_BACKOFF	3 / 2s	Retry policy (backoff doubles)

Evaluation

questions.json maps each question to its ground truth answer:

{
"What is the capital of France?": "Paris."
}

evaluate runs four LLM-judged metrics per question — correctness, relevance, groundedness, and retrieval relevance — each returning a pass/fail verdict with reasoning, written to the output JSON.

Comparing runs

An .eval file is mostly judge reasoning, so it is far too large to read. compare reduces any number of them to the numbers that matter and lines them up:

rag-cli compare test/answers-prompt_v1.json.eval test/answers-prompt_v2.json.eval
make compare # same thing over test/*.eval
make compare EVAL_FILES="a.eval b.eval"

Output goes to stdout, so it pipes into jq; -o writes it to a file instead.

{
"baseline": "answers-prompt_v1",
"evals": [
{ "label": "answers-prompt_v1", "result_count": 50, "error_count": 0,
"metrics": { "correctness": { "passed": 16, "total": 50, "rate": 0.32 } } }
],
"deltas": [ {}, { "correctness": 0.12, "retrieval_relevance": -0.08 } ],
"best": { "correctness": "answers-prompt_v2" }
}

The first file given is the baseline every delta is measured against, so ordering the arguments chooses what you are comparing to. Counts are recomputed from the per-question results rather than trusted from the file header. Two caveats worth knowing: errored questions stay in the denominator, and a judge call that itself failed is stored as a false verdict, so a rate mixes "the judge said no" with "the judge never answered". warnings calls out runs that used a different judge, embedding or answering model, or a different number of questions.