Loom

Leased orchestration for high-utilization Agent evaluation.

Loom is an inventory-driven task control plane for Agent evaluation that makes existing remote capacity do more useful work. Rather than making a fixed cloud instance the unit of scheduling, Loom leases individual evaluation attempts to shared, operator-owned workers with explicit resource, concurrency, cache, and recovery contracts. The Loom Hub owns scheduling, leases, retries, and result intake; Loom Runners execute ordered task packages and return compact, queryable result ZIPs.

Why Loom | Remote quick start | Manifest | Architecture | v0.5 release and protocol versions | Source cache and affinity | Docker Agent protocol | Oracle, trajectory, and reward | Release contract | Support scope

The Name

Loom is the short name for Leased Orchestration for Observable Model work. A loom brings separate threads into one ordered fabric; Loom brings task definitions, available hosts, leases, and evidence packages into one recoverable execution flow.

Why Loom

Long-running Agent evaluations often waste more wall-clock time waiting for available capacity than doing useful work. A whole cloud instance or worker slot is frequently treated as the smallest schedulable unit, leaving CPU, memory, accelerators, and already-fetched source idle while another experiment waits.

Loom makes a task attempt the unit of control on hosts you already operate:

• Reuse shared capacity deliberately. A task declares CPU, memory, disk, and accelerator reservations; the Hub leases it only when the shared worker has declared capacity. shared and exclusive placement make coexistence an explicit scheduling decision.
• Keep the useful work moving. Per-worker concurrency, leases, retries, and fine-grained case/run/attempt identities let independent evaluation work use available slots without losing its execution history.
• Avoid repeated setup work. Pinned Git sources can stay in a bounded Runner-local cache, while every attempt still receives a fresh writable worktree. Cache affinity prefers a matching worker without making a cache miss block the queue.
• Reuse Agent runtimes without re-sending them. A Linux Agent image can be admitted once as a SHA-256-verified tar, then its case/run configuration moves as files. Runners prefer a matching local image or archive cache while each attempt keeps fresh input, output, and workspace directories.

This is not VM provisioning or cloud autoscaling. Loom begins once hosts exist and focuses on turning their remaining, heterogeneous capacity into observable, recoverable evaluation throughput.

What You Get

• Explicit benchmark work units. Model every runnable unit as a campaign, case, setting, and run instead of handing a worker an ambiguous instruction.
• Controller-owned scheduling. Workers advertise a hard capacity, while the controller controls leases, desired concurrency, retries, and recovery.
• Resource-aware shared-host admission. A task can reserve CPU, memory, disk, and accelerators before it is leased, while shared and exclusive placement make host multiplexing explicit instead of accidental.
• Frozen Core Preview v0.5 contracts. Versioned inventory, manifest, dispatch, token, CLI, and capability-query contracts keep downstream automation on a documented surface rather than internal Python helpers.
• Remote worker connections that persist. Bootstrap workers once over SSH, use long-polling for quieter idle periods, or expose an authenticated worker control API. Direct Runners can pull work or receive Hub-leased exact tasks; the task queue and state machine always stay in the controller.
• Versioned phase contracts. A repository task can declare named prepare, evaluate, and collect phases with phase-specific arguments, environment, timeouts, artifact paths, and immutable runtime IDs.
• Recoverable evidence. Each attempt keeps its task ID, attempt number, worker identity, logs, explicit artifacts, and result ZIP. A later successful retry does not erase the earlier failure package.
• Independent semantic evaluation. An Oracle can consume a retained execution ZIP on its own capable worker, report pass/fail/error/inconclusive, and retry without rerunning the original Agent attempt. Opt-in trajectory exports are redacted and size-bounded; Oracle-owned rewards stay structured and queryable without turning Loom into an RL trainer.
• Integration-owned metadata. Optional namespaced extensions flow from a campaign into recovered task and worker-result records without becoming a hidden scheduling or provider contract.
• Cache-aware source reuse. Immutable Git commits can reuse a bounded Runner-local mirror while every attempt still receives an isolated writable workspace. Hub treats a local cache hit as a scheduling preference, never as a requirement.
• Docker Agent admission. Adapter-owned linux/amd64 images pass a versioned label contract before Hub accepts their tar archive. Container tasks receive non-secret config files, declared secret environment-variable names, bounded archive-cache reuse, resource-derived Docker limits, and normal attempt ZIP recovery.
• A small operating footprint. The core Hub and Runner use the Python standard library and a SQLite-backed controller; Docker is an optional host prerequisite only for runner: "container" tasks.

Supported Boundary

Important

This project starts after controller and worker hosts already exist. Automatic cloud resource creation, resizing, billing, teardown, and provider credential management are intentionally unsupported and are not on the roadmap.

Supply an operator-owned inventory, then use Loom Hub to deploy or connect Loom Runners on those hosts. The retained tools/loom_tencent_provision_reference.py, tools/loom_tencent_e2e_reference.py, and tools/loom_aws_smoke_reference.py files are historical/community references, not supported interfaces. A maintained provisioning integration needs a contributor-owned pull request with provider-specific tests, security, cost, failure-recovery, and cleanup behavior.

Read the complete support scope before changing the infrastructure boundary.

Remote Quick Start

This is the supported path for dispatching work to an existing remote fleet. Run these commands from an operator control environment; Loom Matrix coordinates the actual work on the hosts named in the inventory.

1. Start from examples/loom-inventory.example.json and replace its sample addresses, users, SSH key paths, controller URLs, and worker capabilities with your own existing hosts.

2. Define the campaign as explicit case/run records. The task input manual covers the schema, private source repositories, artifacts, retries, and expected outcomes.

3. Normalize the handoff into a controller dispatch specification:

   python3 tools/loom_manifest.py campaign.json \
   --operator my-team \
   --output campaign.dispatch.json

4. Deploy or connect to the inventory, dispatch the specification, and wait for remote results:

   export LOOM_HUB_TOKEN='generate-and-store-outside-the-inventory'export LOOM_RUNNER_TOKEN='separate-direct-runner-token'
   python3 tools/loom_matrix.py \
   --inventory /path/to/operator-owned/inventory.json \
   --dispatch-spec campaign.dispatch.json \
   --output remote-run-summary.json

For a private source repository, add --forward-env SOURCE_REPO_TOKEN and make that variable available only in the operator environment. The worker uses it through GIT_ASKPASS; the token is not placed in task JSON, command logs, or result ZIPs. Matrix forwards the Hub and Direct Runner token environment variables through temporary 0600 remote files and records only their names.

loom_matrix.py deploys Loom Hub and Loom Runner scripts only. It does not create, resize, stop, or delete cloud resources.

How A Run Moves

campaign manifest -> normalized dispatch spec -> controller -> workers -> result ZIPs -> query, retry, or recover

Role	Owns
Operator	Existing hosts, network policy, credentials, and infrastructure lifecycle.
Loom Hub	Task dispatch, leases, state transitions, desired concurrency, result intake, audit logs, and data queries.
Loom Runner	Capability registration, heartbeats, task execution, artifact collection, and result upload.

Loom Hub is the single source of truth for task state. Loom Runners report execution facts; they do not operate an independent queue.

Connection Modes

Choose a connection mode per host in the inventory:

Mode	Use it when
ssh-start	SSH should bootstrap a long-lived Loom Runner, after which scheduling uses the Hub HTTP API rather than a fresh SSH session for each task.
long-poll	An idle Loom Runner should keep a claim request open briefly instead of repeatedly polling the Hub.
direct-worker-api	A Runner-side authenticated HTTP endpoint. Select direct_api_dispatch_mode: "pull" for a Runner pull loop or "push" for Hub-leased exact task delivery. Task state still belongs to Loom Hub.

Inventory-level ssh_control_persist supports SSH ControlMaster/ ControlPersist reuse during setup. See the architecture guide for placement and networking details.

Task Model And Recovery

Every runnable task has four mandatory identifiers:

campaign_id + case_id + setting_id + run_id

Loom Manifest derives a stable task ID from those values. That makes an individual run queryable, cancellable, retryable, and inspectable without disturbing adjacent work. Optional retry policies can limit retries to known transient categories and require the retry to land on a different capable worker.

Repository tasks describe a source checkout, ordered commands, timeouts, and an explicit artifact allowlist. Loom Runners materialize the source in a per-task workspace, then upload only metadata, command logs, and requested artifacts. Full source checkouts are deliberately excluded from result packages. Pinned Git sources may use a Runner-local cache; see Source Cache And Cache Affinity for the descriptor and scheduling contract. For Dockerized Agent runtimes, see Docker Agent Protocol.

See Loom Manifest for the full JSON/JSONL schema and Architecture for the delivery protocol.

Documentation

Guide	When to read it
Loom Scope	Before changing host, provider, or resource-lifecycle behavior.
Versioning	When deciding whether a change belongs in a patch, product release, or protocol version.
Loom Manifest	When preparing a campaign, retry policy, private source, or expected-result contract.
Resource Admission	When sharing existing workers safely with declared task resource requests.
Cache Affinity	When reusing pinned Git sources or admitted Docker images and inspecting cache-local scheduling.
Docker Agent Protocol	When admitting a Linux Agent image, publishing its tar, or configuring container tasks.
EnvTrustBench Docker adapters	When adapting a named EnvTrustBench scaffold to the Docker protocol.
Docker Gate Evidence	When inspecting the redacted remote Docker/Flash protocol acceptance receipt.
Source-Cache Gate Evidence	When inspecting the redacted remote cache-affinity release receipt.
Oracle, Trajectory, And Reward	When adding a semantic judge, redacted trace export, reward data, or selected result recovery.
Oracle Contract Example	When adapting the smallest execution-to-Oracle flow to an operator-owned remote fleet.
Release Contract	When changing phases, Direct Runner delivery, authentication, result retention, or release gates.
AgentDojo Release Fixture	When running or inspecting the fixed 2-case x 2-run x 2-attempt remote regression.
Architecture	When integrating the Hub, Runner, connection modes, concurrency behavior, or result APIs.
Loom Remote Validation	When validating the inventory-driven remote path on operator-supplied Tencent hosts.

Repository Map

tools/loom_hub.py # Loom Hub API, SQLite state, and CLI
tools/loom_cache.py # Immutable Git source descriptors
tools/loom_container.py # Docker Agent admission and archive publishing
tools/loom_evaluation.py # Oracle, trajectory, and reward contracts
tools/loom_export.py # Selector-based SHA-256 result recovery
tools/loom_runner.py # Loom Runner
tools/loom_manifest.py # Loom Manifest normalizer
tools/loom_matrix.py # Loom Matrix remote runner
examples/loom-inventory.example.json
docs/

Contributing

Contributions are welcome. Please keep the ownership boundary intact: this project coordinates work on supplied hosts, while infrastructure lifecycle belongs to the operator or a separate infrastructure system. A proposal to maintain automatic provisioning must include its provider-specific validation, security, cost, recovery, and cleanup contract.

License

MIT.