Agent Runtime

A minimal but structurally complete single-agent runtime: a task goes in, the model reasons and calls tools, the runtime validates and executes those tools, verifies the result against the real filesystem, persists every event, and streams the whole process over SSE.

POST /api/tasks -> model call -> tool call -> schema validation -> policy check
-> tool execution -> tool result back to the model -> verification -> evidence
-> events persisted + streamed -> final answer

By default there is no Kubernetes and no Redis, but PostgreSQL is required — see docker-compose.yml for a local instance. Setting REDIS_URL additionally switches the same code to a cross-replica event bus.

Agent Runtime architecture (learning roadmap)

This is the shape of an Agent Runtime as an architectural pattern — the thing that turns a stateless model call into a system with a lifecycle, memory, safety and recovery. It generalises past this specific codebase: swap the model provider, the tool set or the storage backend and the boxes below stay the same. Where a box is grounded in this repo, the file is noted; for a line-by-line walkthrough of this repo's own module layout, see ARCHITECTURE.md.

 Boundary / contract
(what may the outside world assume?
create -> stream -> resume -> cancel)
|
task/goal in, events + answer out
v
┌───────────────────────┐
│ AGENT LOOP │ the runtime's core control flow:
│ reason -> act -> │ reason -> act -> observe -> repeat
│ observe -> repeat │ until a stop condition is reached
└──┬─────┬─────┬─────┬───┘
| | | |
┌──────────▼┐ ┌──▼───┐ ┌───▼─────┐ ┌───▼─────────────┐
│ Reasoning │ │Action │ │ Safety │ │ State & memory │
│ pluggable │ │tools = │ │policy │ │ - lifecycle │
│ model │ │capability│ risk │ │ state machine │
│ behind a │ │surface │ │gate + │ │ - durable event │
│ protocol │ │ │ │sandbox │ │ log (replay) │
└────────────┘ └───────┘ └─────────┘ │ - context budget │
└────────┬─────────┘
|
┌────────────────────────────────────────▼──────────┐
│ Termination gate: verification, not silence │
│ ("model stopped calling tools" != "task is done") │
└────────────────────────────────────────┬──────────┘
|
┌───────────────────────┬─────────────────────────────┬─▼───────────────────┐
│ Human-in-the-loop │ Execution ownership │ Extended cognition │
│ suspend/resume is a │ who may execute this run │ sub-agents, agent-to-│
│ first-class state, │ when there is more than one │ agent protocols, │
│ not an exception │ replica (lease + fencing) │ retrieval, sandboxed │
│ │ │ code execution │
└───────────────────────┴─────────────────────────────┴──────────┬───────────┘
|
┌───────────────────▼──────────┐
│ Continuous improvement │
│ (execution -> lesson -> gated │
│ promotion -> next run's prompt)│
└────────────────────────────────┘
cross-cutting, present at every layer, shaping none of the above logic:
observability (metrics + tracing) · security (secrets, workspace isolation)

Where each concept is grounded in this repo (secondary — use this to jump into code once the concept above makes sense, not the other way round):

Concept	This repo
Boundary / contract	app/api/{routes,sse}.py
Agent loop	app/runtime/loop.py
Reasoning (model provider)	app/providers/{base,dashscope,mock}.py
Action (tools)	app/tools/{registry,base}.py + individual tools
Safety (policy + sandbox)	app/policy/engine.py, app/tools/workspace.py
Lifecycle state machine	app/domain/enums.py
Durable event log	app/runtime/events.py, app/persistence/
Context budget	app/runtime/context.py
Verification	app/runtime/verification.py
Human-in-the-loop	app/tools/ask_user.py, checkpoint/resume in app/runtime/manager.py
Execution ownership	app/runtime/manager.py (lease + fencing token)
Extended cognition	app/tools/{subagent,remote_agent,knowledge,program}.py
Continuous improvement	app/evolution/
Observability	app/observability/, app/web/

Suggested reading order — each stage answers one question before moving on; this is also roughly the order features were built in this repo (see the git history and ARCHITECTURE.md sections of the same names):

Stage	Question to be able to answer	Read
0. Vocabulary	What states can a run be in, and who enforces legal transitions?	app/domain/enums.py, app/domain/models.py
1. Single-process loop	How does one goal become "model calls a tool, tool result goes back to the model"?	app/runtime/loop.py, app/providers/base.py+mock.py, app/tools/registry.py+base.py, app/policy/engine.py
2. Events & persistence	Why must the event be published before a terminal status is written?	app/runtime/events.py, app/persistence/{db,repository,tables}.py
3. The API contract	How does a client create, stream and resume a run without re-executing side effects?	app/api/{routes,sse}.py, and Subscribe to the event stream below
4. Human in the loop	Why does a suspended run hold no lease?	app/tools/ask_user.py, checkpoint/resume in app/runtime/manager.py
5. Distributed execution	What stops two replicas from both executing the same run?	app/runtime/manager.py (lease + fencing token), docker-compose.yml, scripts/check_infra.py
6. Advanced tools	What changes when the "sub-agent" is a different process entirely?	app/tools/{subagent,remote_agent,knowledge,program}.py, Talking to other agents in ARCHITECTURE.md
7. Self-evolution	Why is PROPOSED a security boundary and not just a review step?	app/evolution/*, Self-evolution in ARCHITECTURE.md
8. Observability & console	How do metrics/tracing/UI attach without becoming part of the critical path?	app/observability/*, app/web/*, Metrics / Tracing below

Requirements

• Python 3.11+
• PostgreSQL (a local instance via docker compose up -d postgres is enough)
• git (optional; only used for get_git_diff and diff-based evidence)

Install

uv sync --extra dev

Configure

Exactly four environment variables are required:

cp .env.example .env
docker compose up -d postgres # or point DATABASE_URL at any PostgreSQL you already run

Variable	Meaning
DASHSCOPE_API_KEY	API key for the OpenAI-compatible endpoint
DASHSCOPE_BASE_URL	Full OpenAI-compatible base URL, e.g. https://dashscope.aliyuncs.com/compatible-mode/v1
MODEL_ID	Model identifier used for every request
DATABASE_URL	PostgreSQL connection string, e.g. postgresql://user:pass@localhost:5432/db (a plain postgresql:// URL is upgraded to the async driver automatically)

Everything else (iteration budget, timeouts, output caps, workspace) has a default and is changed in code — see RuntimeLimits in app/config.py.

The key is loaded into the runtime process only. It is never logged, never persisted, never sent to a tool's environment and never placed in the model context.

Before starting the app for the first time, apply the schema:

alembic upgrade head

See Schema migrations below for what that command does and why nothing is auto-created at startup.

Trusted-local and secure mode

Authentication is deliberately dual-mode. With RUNTIME_API_KEYS_JSON unset, the runtime accepts local requests as tenant local, keeps the configured workspace root, and does not enforce tenant quotas. This compatibility mode is for a trusted developer machine only.

Setting a tenant-to-secret map enables secure mode:

RUNTIME_API_KEYS_JSON='{"tenant-a":"replace-with-a-long-random-secret"}'
SANDBOX_IMAGE='your-agent-sandbox:latest'

Every /api operation except health then requires Authorization: Bearer <secret>. Repository queries are tenant-scoped, and tenant workspaces live at WORKSPACE_DIR/<tenant-id>. Request and active-run quotas are PostgreSQL-backed and therefore apply across replicas; configure them with TENANT_REQUESTS_PER_MINUTE and TENANT_MAX_ACTIVE_RUNS.

Optional: run as more than one replica

docker compose up -d # postgres, redis, qdrant, meilisearch, prometheus, grafana
python scripts/check_infra.py # connectivity + two distributed-behaviour experiments

Then set the remaining variable in .env:

Variable	Effect when set
REDIS_URL	Events fan out to every replica instead of one process
REPLICA_ID	Labels this process in metrics and /api/health

Schema migrations

The schema is managed exclusively by Alembic — nothing is auto-created at startup, for any environment:

alembic upgrade head # fresh or existing database
alembic stamp 0001 && alembic upgrade head # a database created by create_all before migrations existed
alembic revision --autogenerate -m "..."# after changing app/persistence/tables.py

The URL comes from the same .env the app reads (ALEMBIC_DATABASE_URL overrides it), and migrations run on the async driver already in the dependencies. create_all is deliberately never used: it creates missing tables and never adds a column to a table that exists, which is the classic "works on a fresh database, fails on the deployed one" bug.

Both are needed together. PostgreSQL alone gives shared state but a stream opened on replica B will not see live events from a run executing on replica A; Redis alone gives fan-out over storage that is not shared.

What Redis is not for: it does not make delivery reliable. The database holds the authoritative, sequenced event log and a client recovers by replaying with Last-Event-ID. Redis only removes the latency of polling.

Run two replicas locally against the same infrastructure:

REPLICA_ID=a uvicorn app.main:app --port 8000 &
REPLICA_ID=b uvicorn app.main:app --port 8001 &# start a run on A, stream it from B:
curl -N http://127.0.0.1:8001/api/runs/<run_id>/stream

Durable dispatch and execution ownership

Creating an auto-start task, starting a parked run, answering ask_user, or approving reconciliation records dispatch_requested_at in the same database transaction as the state change. Each replica polls and claims these durable requests up to MAX_CONCURRENT_RUNS_PER_REPLICA; a crash after the API commit therefore leaves work claimable instead of losing an in-process task.

A run is owned by whichever replica holds a live lease on it, renewed every lease_ttl_s / 3. A lease alone is not mutual exclusion: a replica that stalls past the TTL loses the run without noticing, and keeps writing until its next renewal fails.

So every acquisition also increments lease_epoch, and the owner writes run state with the epoch it was granted. A stale owner's writes no longer match and are rejected with LEASE_LOST — safety stops depending on the old owner realising anything. Sub-agent runs are fenced together with their parent.

The fence is applied per transaction, not per statement: database.repository( fence=(run_id, epoch)) checks the epoch before the first write that touches that run, so events, checkpoints, tool records and status changes all roll back together if the run has been taken over.

lease_ttl_s then trades two measurable things against each other: it must exceed the worst normal pause (GC, slow IO) or healthy replicas get declared dead, and stay under the recovery delay you can tolerate, since a crashed replica's runs are unclaimable until the lease expires.

Metrics

Endpoint	For
GET /api/metrics	Prometheus text format (machine)
GET /api/metrics/view	The same numbers with an explanation of each (human)

Metrics are derived from the event stream (agent_runs_total, agent_tool_calls_total, agent_model_tokens_total, agent_run_duration_seconds, agent_dropped_events_total, ...) plus HTTP-level counters (agent_http_requests_total, agent_http_request_duration_seconds).

Prometheus at http://localhost:9090 scrapes /api/metrics; Grafana at http://localhost:3000 (admin/admin) is provisioned with the datasource and an Agent Runtime dashboard from ops/grafana. The dashboard only shows data while the app is running — Prometheus cannot scrape a stopped process, and a target that was never up produces empty panels.

Tracing (Langfuse, optional)

Set LANGFUSE_BASE_URL, LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY — all three, or tracing stays off. Each run becomes one trace: a generation per model call (output, usage, latency) and a span per tool execution, plus the verification result. It hangs off the same hook metrics use, so nothing is sprinkled through the loop.

Metrics and traces are not alternatives: Prometheus answers "how many, how fast, how much" across all runs; a trace answers "what happened in this run". Observations are buffered and shipped when the run reaches a terminal event, so a crashed run leaves no trace rather than half of one. The prompt itself is not traced — it is not in the event stream, and putting a whole context window there would make the event table a data channel.

API conventions

• Every response carries X-Request-ID; supply one to correlate your own logs.
• In secure mode every non-health API request requires a configured Bearer token.
• Supply Idempotency-Key when creating a task so a client retry returns the same tenant-scoped task/run instead of creating another one.
• Every failure has the same body: {"error_code", "message", "request_id"}.
• Collections are wrapped in an object ({"run_id", "events"}), never a bare JSON array, so pagination can be added without breaking clients.
• GET /openapi.json and /docs describe the whole JSON API. The Prometheus endpoints are excluded: they are not part of that contract.

Check model capabilities

python scripts/check_model.py

Connection: PASS
Text completion: PASS
Streaming: PASS
Function calling: PASS
Echo tool round trip: PASS
Model: <MODEL_ID>

If the model cannot do function calling the script says so explicitly and exits non-zero. There is no fallback that simulates tool calls with regexes or prompt parsing.

Run the service

python -m app # fails fast if the 4 variables are missing# or
uvicorn app.main:app --reload # starts even when unconfigured

uvicorn deliberately starts without configuration so that history endpoints still work; the first attempt to start a run then returns 503 CONFIGURATION_ERROR.

Create a task and run it

curl -s -X POST http://127.0.0.1:8000/api/tasks \
-H 'content-type: application/json' \
-H 'Idempotency-Key: example-task-1' \
-d '{"goal": "List the Python files in the workspace and summarise them."}'# -> {"task_id":"task_...","run_id":"run_...","status":"PENDING"}
curl -s -X POST http://127.0.0.1:8000/api/runs/<run_id>/start
curl -s http://127.0.0.1:8000/api/runs/<run_id>

Pass "auto_start": true to POST /api/tasks to skip the explicit start call.

Subscribe to the event stream

curl -N http://127.0.0.1:8000/api/runs/<run_id>/stream
# resume after a disconnect (either header or query parameter):
curl -N -H 'Last-Event-ID: 12' http://127.0.0.1:8000/api/runs/<run_id>/stream
curl -N 'http://127.0.0.1:8000/api/runs/<run_id>/stream?after_sequence=12'# skip the history entirely and follow from now on:
curl -N 'http://127.0.0.1:8000/api/runs/<run_id>/stream?replay=false'# keep the connection open across an ask_user suspension (default: it closes):
curl -N 'http://127.0.0.1:8000/api/runs/<run_id>/stream?follow_suspended=true'

The stream also ends on input.requested, not only on a terminal event. A run waiting for a person is not over, but nobody is executing it either and the answer may take hours — holding an HTTP connection for that mismatches two time scales. Reconnect with the sequence you already have once you have replied.

Disconnecting a stream never changes run state. On reconnect the runtime replays persisted events first and then continues with live ones; sequence numbers are per-run, gapless and strictly increasing.

The resume point is the client's to state, not the server's: several clients may watch the same run, so no single "last delivered" position exists on the server. Last-Event-ID and after_sequence are the same mechanism through two doors — the header exists because a browser's native EventSource reconnects on its own and only sends that header.

String fields in an event payload are clipped at max_event_payload_chars. When that happens the payload carries truncated_fields naming them, so a client can tell a clipped value from a complete one without parsing it — never re-parse a field listed there.

model.delta is transient: it is delivered but never persisted, so it has "transient": true, sequence: 0 and no SSE id: field. Its content is repeated in full by the model.completed event that follows, so logging every increment would multiply the event table to store what the next row already says. Clients must ignore transient events when tracking their resume position — reconnecting replays the durable log, not the typing.

Endpoints

Method	Path	Purpose
POST	/api/tasks	Create a task + its run
POST	/api/runs/{run_id}/start	Start execution
GET	/api/runs/{run_id}	Run state + completion evidence
GET	/api/runs/{run_id}/events	Persisted events (after_sequence, limit)
GET	/api/runs/{run_id}/stream	SSE stream (replay + live)
POST	/api/runs/{run_id}/cancel	Cancel a run
POST	/api/runs/{run_id}/reply	Answer a pending ask_user question
POST	/api/runs/{run_id}/reconcile	Resolve an uncertain non-idempotent tool call
GET	/api/runs/{run_id}/questions	Questions asked during the run
GET	/api/health	Configuration/tool/policy/backend summary
GET	/api/metrics	Prometheus metrics
GET	/api/metrics/view	The same metrics, annotated for humans

Human in the loop

The agent can call ask_user when it genuinely cannot proceed. The run then suspends: no task is running, and GET /api/runs/{id} reports WAITING_FOR_INPUT together with the pending question.

curl -s http://127.0.0.1:8000/api/runs/<run_id>| jq .pending_question
curl -s -X POST http://127.0.0.1:8000/api/runs/<run_id>/reply \
-H 'content-type: application/json' -d '{"answer": "notes.txt"}'

Because the suspended run lives only in the database, the answer may arrive minutes later and on a different replica than the one that asked. Resuming replays the checkpoint rather than the tools, so no side effect happens twice. The iteration budget is not reset by a question.

A checkpoint also stores a workspace_digest: the sha256 of every file the run read or wrote. On resume the digests are re-checked, and any file that changed in the meantime is named to the model in a SYSTEM NOTE before it continues. Nothing is rolled back — external side effects cannot be undone. The point is narrower and achievable: the agent must not keep treating a stale read as current.

Tools

list_files, read_file, write_file, search_text, get_git_diff, echo, run_tests (fixed command only), ask_user, write_todos, delegate. There is no arbitrary shell, no network tool and no git push. All paths pass through WorkspaceGuard.

write_todos holds no state: the model re-sends the whole plan and gets it rendered back, so the current plan is always among the newest messages the context manager keeps. At most one item may be in_progress.

read_file returns a window of lines and always states the file's total line count, so a large file can be paged (offset, max_lines) instead of silently truncated. Knowing the total is the part that matters: it is what lets the model choose between paging on and using search_text to jump straight to a section.

delegate starts a sub-agent as a real child run — its own run_id, event stream, checkpoints, verification and iteration budget — and returns only its final answer to the parent's conversation. That is the point: the sub-task's reading never enters the parent's context. The child inherits the parent's lease rather than claiming one (it executes inside the parent's task, so its liveness is the parent's liveness) and shares its cancellation token. It never receives delegate or ask_user, which keeps the tree one level deep. Find the child from the parent's tool.completed event (payload.metadata.child_run_id) or from parent_run_id on the run itself.

Tool recovery and reconciliation

Every built-in tool declares READ_ONLY, IDEMPOTENT, or NON_IDEMPOTENT. The loop persists the assistant's tool-call intent and a deterministic run_id:tool_call_id idempotency key before execution. After a replica crash, unfinished read-only/idempotent calls can be replayed with that same key. Supplying a key does not make an external system idempotent; the tool implementation must honor it.

An unfinished non-idempotent call moves the run to RECONCILIATION_REQUIRED instead of guessing whether its side effect happened. An operator uses POST /api/runs/{run_id}/reconcile with accept plus an observation, explicit retry, or fail. The run holds no lease while waiting.

Optional tools (off unless configured)

Tool	Switched on by	Adds
search_knowledge	QDRANT_URL + MEILISEARCH_URL + MEILISEARCH_API_KEY	Two network dependencies
run_program	ENABLE_PROGRAM_TOOL=true	Model-written Python through the configured execution backend

search_knowledge fuses Meilisearch (lexical) and Qdrant (semantic) with Reciprocal Rank Fusion, because BM25 scores and cosine similarity are not on a comparable scale. The query is embedded with HYBRID_EMBED_MODEL, which must be the model the corpus was indexed with — a mismatch returns plausible-looking noise rather than an error. Seed the corpus with python scripts/seed_hybrid_search.py.

In trusted-local mode, commands use a subprocess with a minimal environment allowlist and a timeout; that is not containment. In secure mode execution tools fail closed unless SANDBOX_IMAGE is configured. The Docker backend uses argv without a shell, no network, a read-only root, dropped capabilities, no-new-privileges, CPU/memory/PID limits, tmpfs, and only the tenant workspace mounted at /workspace.

Context management

The history is budgeted in estimated tokens, not messages — one read_file result can outweigh fifty short turns. context_strategy selects the policy:

Strategy	Behaviour	Cost
sliding-window (default)	Keep the system prompt, the goal and the newest turns	Free, but older turns are lost
summarising	Replace older turns with one model-written summary	One extra model call, and the summary is lossy

Both guarantee a tool message never appears without the assistant message that requested it. Summarising degrades to the window if the summary call fails: context management must never be the reason a run fails.

Current limitations

• Single agent, single run per process-managed task; no multi-agent orchestration.
• Recovery is only automatic for tools whose declared semantics permit replay. Generic external tools do not become idempotent merely because the runtime supplies a key; uncertain non-idempotent work requires an operator decision.
• Verification is intentionally narrow: applicable successful checks produce PASSED, contradicted evidence produces FAILED, and answer-only work or no applicable check produces terminal UNVERIFIED. It does not judge arbitrary semantic correctness.
• No automatic schema creation. The schema is managed exclusively by Alembic; nothing is created at startup, so alembic upgrade head must be run against the database before starting the app for the first time and after every migration (see Schema migrations above).
• Trusted-local mode intentionally has no HTTP authentication, tenant quotas, or OS containment. Do not expose that compatibility mode to untrusted users.
• The Docker backend is a stronger boundary, not a complete hostile-code platform: production deployments should consider image provenance, daemon isolation, seccomp/AppArmor, rootless execution, and a microVM/gVisor boundary.
• Data retention/partitioning, provider backoff/fallback, cumulative token/cost budgets, and durable telemetry projection remain out of scope.