Uh oh!
There was an error while loading. Please reload this page.
fix(master): refuse tmpfs options Swarm cannot express - #65
Closed
echobt wants to merge 605 commits into
Closed
Conversation
Two bugs kept base-supervisor from staying up after bootstrap: 1. Type=notify readiness: `uv run` launches the supervisor as a child (uv stays the unit main PID), so the sd_notify READY came from the child and systemd (NotifyAccess=main) ignored it, leaving the unit stuck `activating` -> crash-loop at start timeout. Set NotifyAccess=all so systemd accepts the child's READY/WATCHDOG pings. 2. image-updater false timeout: the forward roll issues `docker service update --image` WITHOUT --detach, so the CLI blocks through Swarm's health-gated stop-first rollout (pull + 40s health-start + 45s monitor), exceeding the 60s subprocess timeout and raising a false SwarmBackendError that bypassed the convergence/rollback path. Raise DEFAULT_COMMAND_TIMEOUT_SECONDS to 300s; the updater runs on its own daemon thread so a long roll never starves the other loops.
The host systemd supervisor runs outside the swarm overlay and cannot resolve the overlay service DNS in docker.broker_url (http://base-docker-broker:8082), so the broker-health probe failed forever and permanently tripped the BrokerHealthGate. That false trip gates self-update's pre-swap check (and weight submission) OFF, silently disabling supervisor self-update even though the broker is healthy. Add supervisor.broker_health_url so the host probe targets the broker's host-published port (127.0.0.1:8082 on the manager); None falls back to docker.broker_url for in-overlay callers (proxy). Set it in the canonical master.yaml. Reaper/config-sync/image-updater already ignore the gate.
Two durability gaps in the auto-update pipeline: 1. config-sync EROFS: ProtectSystem=full mounts /etc read-only, so config-sync's atomic write of the node-local master.yaml (+ .digest sidecar) under /etc/base failed with OSError Errno 30. Latent until a real canonical change (broker_health_url) forced the first write. Add ReadWritePaths=/etc/base so config-sync can converge the host config. 2. Manifest publish flake: the github-actions[bot] force-push to the release branch intermittently 403s on the first attempt (a re-run clears it). Wrap the push in a bounded retry-with-backoff so the self-update manifest publishes without a manual re-run.
Add an offline network-egress guard (pytest plugin -p no_external_egress) plus its unit test so both default suites can be proven to run with ZERO real egress to lium.io / api.targon.com (loopback + AF_UNIX stay allowed). Wire a worker_plane_enabled toggle into the mission master: with the flag OFF no capability is owned by the worker plane, so gpu units route to online gpu validators byte-for-byte as pre-mission (VAL-MASTER-013). Document the exact reproducible flags-OFF verification procedure in the harness docs.
The master's agent-challenge eval job + analyzer run on base_jobs_internal (--internal, no egress), so the gateway PUBLIC IP is unreachable from there; the proxy is multi-homed onto that overlay precisely so they can reach it by service name. Point CHALLENGE_LLM_GATEWAY_BASE_URL at the internal overlay service URL http://base-master-proxy:19080 (install-swarm.sh static path + cli_app dynamic own_runner env), fixing zero-gateway-call evals.
…erminate (VAL-CROSS-005) Add an opt-in (BASE_LIVE_PROVIDER_TESTS=1) live validation that provisions a real Lium pod via the real `base worker deploy --provider lium` CLI, runs a base worker agent inside the pod (pod_worker_agent.py) that enrolls with a LOCAL mission master over a reverse SSH tunnel, pulls+executes a gpu unit on the CPU stub, and posts an ExecutionProof stamped with the lium provider + real pod id; then DELETE + verify gone + GET /pods empty + balance delta <= $2. Pod is terminated on every path (try/finally) so a failure never leaks a billable pod.
… image) base worker deploy --provider lium hit two real gaps: Lium's edge WAF 403s any POST body carrying a loopback URL (baked into the template env), and the private-namespace placeholder WORKER_IMAGE fails pod creation (CREATION_FAILED). - is_loopback_url() + build_worker_pod_env/LiumClient.ensure_template now strip loopback master/broker/gateway URLs from the WAF-sensitive template body; the agent resolves master_url at runtime from config. - require_worker_image() makes worker.deploy.image + image_digest a required, digest-pinned config for provider deploys (fail-fast, clear error); no more silent pin of an un-pullable private image. - docs/miner/worker-plane.md: publish + digest-pin procedure (flagged as a user/release action) + WAF/CREATION_FAILED troubleshooting; config example. Offline respx/unit tests only; full base gate green.
Confirmed live (single green-lit deploy) that Targon has no single
POST /workloads/deploy route (returns 405); the real flow is two-step
register-then-deploy: POST /workloads (-> uid, state registered) then
POST /workloads/{uid}/deploy (-> provisioning). Refactor TargonClient.deploy
to create-then-deploy, surfacing an insufficient-credit failure at either
step as the typed InsufficientCreditsError (never retried). Send the correct
uppercase type RENTAL. Mirror the LiumClient sub-1h lifetime guard into
provision: 0<max_lifetime_hours<1 raises CostGuardrailError instead of
int-truncating termination_hours to 0. respx tests updated; base gate green
(1540 passed, cov 88.4%, mypy/ruff clean).…gging churn Attach a recording handler directly to the base.supervisor.image_updater logger (mirroring test_supervisor_weights.py / test_supervisor_weight_submit.py) instead of relying on caplog. Importing bittensor raises every already-created logger to CRITICAL, which filtered the WARNING/ERROR records before caplog could capture them, making these tests order-dependent in the full suite. No product code change.
Add one-command blank-server bring-up to install-swarm.sh (+ install-worker.sh):
- ensure_docker(): idempotent Docker Engine auto-install (get-docker|apt),
SKIP_DOCKER_INSTALL opt-out, dry-run safe, runs before preflight.
- --auto-secrets: auto-generate/derive/mint all 13 internal secrets
(openssl rand, asyncpg DSNs, Fernet key, central-gate token via
mint-central-gate-token); YUNWU_API_KEY stays the sole external input.
Persisted mode-600 to SECRETS_ENV_FILE for rotation coherence.
- ensure_validator_wallet() + auto-seed MOCK_METAGRAPH from the derived
ss58 hotkey (validator_permit) via the public master image.
- --skip-ghcr-login + {}-config footgun fix so broker/proxy /root/.docker
binds resolve on public-image deploys.
- --turnkey umbrella flag ties it together (blank box -> full validator).
Tests: ensure_docker, auto-secrets (DSN derivation/persistence/Fernet/
idempotency/mint-plan), wallet+metagraph, worker ensure_docker, ghcr-skip
footgun; docs updated. Full suite green (coverage 89%).… + flush harness master logs worker_unit_status keyed faulted_units by work_unit_id alone; harden to the (challenge_slug, work_unit_id) tuple so same-id units under different challenge slugs never collide (regression test added). Line-buffer + basicConfig the mission_master harness so its drill logs are non-empty/inspectable after SIGTERM teardown.
base-supervisor.service ExecStart=/usr/local/bin/uv run ...; a blank box has no uv. Add idempotent, dry-run-safe ensure_uv() (astral installer, UV_INSTALL_DIR pinned to /usr/local/bin, SKIP_UV_INSTALL opt-out), gated on INSTALL_SUPERVISOR, wired before preflight in both main() and main_validator_node(). +6 tests.
Two bugs surfaced installing --turnkey on a fresh non-master server:
- daemon.json had live-restore:true, which docker rejects at swarm init/join
('incompatible with swarm mode') and is inert on swarm nodes anyway. Removed
from all three daemon templates (validator/cpu-worker/gpu-worker).
- ADVERTISE_ADDR defaulted to the hardcoded master IP; now auto-detects this
host's primary IPv4 (default-route src) with a swarm_init guard. Kept a
PRODUCTION_ADVERTISE_ADDR constant for the canonical-config consistency test.
Tests: daemon-no-live-restore regression + advertise-addr auto-detect/override;
updated master-config consistency test; README updated. Full suite green.…ount fallback - lium.provision: move pod-id extraction inside the try so a 2xx rent with a non-JSON body still terminates+verifies the just-rented pod (no leaked pod); _extract_pod_id raises a typed LiumError on an unparseable body. - targon._extract_gpu_count: fall back to the top-level numeric gpu_count when spec.gpu_count is 0/unknown, so per-GPU price (and max_price filtering) is correct for multi-GPU shapes. - worker.deploy.image_tag documented as informational-only (never consumed by the digest-pinned deploy path) in settings, config, and the miner docs.
# Conflicts: # src/base/cli_app/main.py # tests/unit/test_image_updater.py
PRISM Compute Plane: miner-funded GPU worker plane (base)
…h auto-update Add an authoritative Compute Requirements table to the validator guide (submit-only 2 vCPU/4 GB; base agent-challenge 8 vCPU/32 GB; PRISM needs no extra compute because GPU eval is delegated to the miner-funded worker plane) and reconcile the minimum-requirements FAQ to point at it. Document deploy/swarm/install-swarm.sh as the automatic one-command install: dry-run by default (--apply executes), --validator-node brings up an auto-updatable base-validator-agent Swarm service plus a node-local base-supervisor image-updater (the auto-update), --install-supervisor enables the systemd control-plane unit. Include a dry-run-then-apply quick-start with required env and VALIDATOR_CAPABILITIES cpu vs delegated-PRISM selection.
…s root Turnkey one-shot base-master CLI runs (central-gate token mint, validator wallet gen, runtime-uid inspect) now use IMAGE_MASTER_CLI (mutable :latest) instead of the digest-pinned IMAGE_MASTER, so newly-added CLI subcommands (e.g. mint-central-gate-token) always exist. Deployed services keep the pinned IMAGE_MASTER (the supervisor image-updater rolls them forward). The mint docker run now uses --user 0:0 so it can read the root-owned mode-600 mint.yaml + gateway secret temp files. Adds regression tests for the ephemeral-vs-service image split and the --user 0:0 root flag.
docs(validator): compute-requirements table + turnkey install with auto-update
…ickstart to docs/deploy.md Reduce the README to a one-screen overview (banner, nav, architecture + weight mermaid, roles/worker-plane/docs tables of <a href> links) and move the long deploy walkthrough into docs/deploy.md. Tighten every docs/ page and add a coordination-flow mermaid to architecture.md. All doc-contract tests preserved.
…wlist fix(master): allowlist public FE agent-challenge reads on proxy
Align package docs with joinbase unattested scoring and miner day-1 flow. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Allow bare GET /benchmarks (metadata object) in the enabled-mode Agent Challenge proxy allowlist. The frontend already requests this path, but only /benchmarks/tasks was permitted. The two endpoints return different shapes, so the frontend path cannot simply be swapped, and joinbase kept rendering a Base 404 after the public submissions allowlist shipped. Extend the proxy forward matrix and the FE public GET path list to cover the bare benchmarks route. Deny coverage for internal, owner, evidence and TEE neighbors is unchanged. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
In-process terminal_bench own_runner argv omitted --cache-root and --digest-manifest that runner.py already wires. pydantic-settings does not export CHALLENGE_* into the subprocess env, so production trials fell back to broken defaults. Mirror runner.py before the optional --model block.
…default parents[3] only worked in the src checkout; installed wheels overshot into site-packages' grandparent and lacked dataset-digest.json. force-include the golden file and resolve via importlib.resources plus an ancestor walk.
hatch force-include needs golden/dataset-digest.json at /app during pip install; both runtime and terminal-bench-runner stages now COPY it before install so CI image builds stop failing FileNotFoundError. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…esolution fix(agent-challenge): resolve TB digest manifest in installed wheel Pass --cache-root/--digest-manifest from own_runner, ship golden/dataset-digest.json in the wheel, and COPY it into Docker build context so installed images stop failing TaskDefNotFoundError.
Task guests incorrectly inherited the DooD job-client --read-only posture, so verifier upload_tests failed with "mkdir: cannot create directory '/tests': Read-only file system" while the container was still running. Keep cap-drop/nnp/pids hardening; leave rootfs writable for harbor paths and apt-based test.sh. Wire DooD env into upload_tests/collect_verifier_dir.
fix(agent-challenge): stop RO-rootfs on own_runner task containers
Regression guard for the compose long-syntax tmpfs.mode decimal footgun that made /tmp mode 3361 and blocked uid 1000 Terminal-Bench builds.
Compose long-syntax tmpfs.mode: 1777 is decimal (octal 3361). Short syntax /tmp:size=256m,mode=1777 matches the validator compose and yields sticky world-writable /tmp for the master validator process.
…ng secrets Lock both directions: count-dataset-tokens stays verbatim and genuine credential shapes remain scrubbed from public task-event text.
Match secret/token as hyphen-delimited segments and shield the known Terminal-Bench task-id catalog so ids like count-dataset-tokens are not replaced with [REDACTED_SECRET] in public task events.
Contract that score 0.0 serializes passed=false with status completed, score 1.0 serializes passed=true, and terminal event metadata carries the same flag when score is present.
Additive boolean derived from score >= 1.0 (same rule as passed_tasks). Lifecycle status stays completed/failed for compatibility; consumers no longer treat a zero-score completed task as a pass.
Flip isolation invariants so own_runner task containers assert cap-drop is absent while no-new-privileges, pids-limit, and tmpfs remain.
…ucceed Deliberate isolation trade-off already running in prod as a hotpatch: own_runner task guests need capabilities for apt/chmod inside verifiers. Master compose cap_drop ALL is unchanged. Keep no-new-privileges, pids limit, /tmp tmpfs, and writable workspace volume.
Narrow tmpfs options to dict[str, Any] before .get so union-attr is clean.
…action-passed-capdrop fix(tb): scoring tmpfs, redaction, passed flag, cap-drop
…ate (#61) * fix(proxy): allow miner env/launch routes under attested allowlist Prod enables agent_challenge_attested_routes_enabled, which blocked GET|PUT /env, POST /env/confirm-empty, and POST /launch with local 404. Treat env/launch as signed routes flag-independently via the existing _is_agent_challenge_env_route SSOT so the enabled-mode allowlist admits them and miner X-Hotkey/X-Signature/X-Nonce/X-Timestamp headers are preserved (otherwise signed PUT residual 401 after allowlisting). * fix(agent-challenge): start submissions unconfirmed for miner env gate _persist_submission hardcoded env_confirmed_empty=True, so analysis allow always auto-enqueued credential-less evaluation and waiting_miner_env was dead. New rows start unconfirmed; miners must PUT /env or POST /env/confirm-empty before evaluation. Legacy backfill paths untouched. * fix(agent-challenge): fail-closed evaluation enqueue status gate _validate_evaluation_enqueue_status previously fell through for unknown statuses (including accidental enqueue from non-ready states). Explicitly allow in-flight TB statuses, waiting_miner_env when confirmed, legacy analysis_allowed, and terminal re-eval labels; raise ValueError otherwise.
Host-local evaluation has egress to PyPI; --no-index caused instant empty-index failures for real agent deps (e.g. litellm>=1.55.0). Drop offline-only pip flags and use network-sane retries/timeouts.
…pypi-index fix(agent-challenge): allow PyPI index for agent dependency install
Every Terminal-Bench evaluation scored 0 and every result.json carried "model_name": null. The packaged agent fails closed without a concrete model id: agent run failed: A concrete model id is required: set LLM_MODEL (the measured review harness supplies it under .rules). Three independent layers dropped the model id, so no submission could ever score: 1. the runner never set LLM_MODEL -- the platform only holds it as CHALLENGE_LLM_MODEL (settings.llm_model), which the agent never reads; 2. sanitize_miner_env_for_job stripped a miner-supplied LLM_MODEL because the name is not token/key shaped; 3. AGENT_ENV_ALLOWLIST admitted only LLM_COST_LIMIT and OPENROUTER_API_KEY. The trial died in ~5s per task, before any LLM call, which is why the whole leaderboard sat at zero while OPENROUTER_API_KEY was reaching the agent correctly. _terminal_bench_env now publishes settings.llm_model as LLM_MODEL, ahead of the sanitized miner merge so a miner may override it; the model name joins both the miner product allowlist and the agent sandbox allowlist. A miner can therefore bring their own key and their own model. When an operator blanks llm_model nothing is injected, so the agent still fails loudly rather than running an unmeasured default. Updated pins keep their intent: gateway/URL/host/proxy names stay excluded, secret scrubbing still asserted, and the ex-gateway test now asserts LLM_MODEL comes from settings and that no gateway base_url or token leaks into the job env. Verified: 7 new tests RED -> GREEN; full agent-challenge suite shows zero regressions (26 pre-existing failures on main, 26 with this change, identical sets; 2286 -> 2293 passing).
fix(agent-challenge): supply LLM_MODEL to the evaluated agent
…asted jobs Four production causes made Terminal-Bench score 0 for infrastructure reasons rather than merit, and hid why. - allow exec on the runner /tmp tmpfs: pip installs the miner agent into /tmp/.local under a read-only rootfs, and Docker mounts tmpfs noexec by default, so loading failed with 'failed to map segment from shared object'. - surface per-trial failure diagnostics: the broker never bind-mounts the job directory back, so only runner stdout/stderr survives; the cause now travels on that channel. - keep the real cause of agent construction failures: a constructor that shells out writes to inherited file descriptors, so CalledProcessError kept only argv plus exit status; fd 1 and 2 are captured and replayed. - stop a job once the agent cannot be constructed: construction is task-independent, so a broken package no longer burns 30 tasks of wall clock and LLM budget to rediscover one packaging error.
The Swarm backend translated a docker-run tmpfs spec by keeping size= and discarding everything else. The own_runner job container installs the miner agent into /tmp/.local under a read-only rootfs, so its spec carries exec; tmpfs is noexec unless told otherwise. Routing that workload through Swarm therefore produced a noexec /tmp, where the agent loads no shared object and fails with "failed to map segment from shared object" -- a symptom that took hours to trace back to a dropped mount flag. Swarm cannot express the flag at all: against Docker 29.2.1 a bare exec is rejected as a non key=value field and tmpfs-options is an unknown option, so only tmpfs-size and tmpfs-mode survive. Translate those two, accept the flags Swarm already applies by default, and refuse anything else -- exec loudest of all -- rather than dropping it and failing far from the cause.
Warning Review limit reached
Next review available in:6 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Swarm backend translated a docker-run tmpfs spec by keeping
size=and discarding every other option.The own_runner job container installs the miner agent into
/tmp/.localunder a read-only rootfs, so its spec is/tmp:rw,nosuid,exec,size=2g. tmpfs isnoexecunless told otherwise. Routing that workload through Swarm produced a noexec /tmp, where the agent loads no shared object and dies withfailed to map segment from shared object— the same symptom that already cost hours to trace on the broker path (fixed in #64).The broker path is what production runs today, so this was latent, not live. It becomes live the moment the Swarm path is enabled.
What
Swarm cannot express the flag at all. Verified against Docker 29.2.1 on the prod host:
type=tmpfs,...,execinvalid field 'exec' must be a key=value pairtype=tmpfs,...,tmpfs-options=execunknown option 'tmpfs-options'type=tmpfs,destination=/tmp,tmpfs-size=1048576So translation is impossible and silence is the worst option. This change:
size=andmode=totmpfs-size/tmpfs-mode;rw,noexec,nosuid,nodev);exec, and refuses any other untranslatable option, instead of dropping it.Tests
New
tests/unit/test_swarm_tmpfs_exec.py, written failing first: the two refusal tests failed withDID NOT RAISEbefore the change while the three translation tests already passed, pinning that existing specs keep working. 5 passed after.