Uh oh!
There was an error while loading. Please reload this page.
feat(prism): enable unattested Lium dispatch for PRISM jobs (T14) - #54
Closed
echobt wants to merge 572 commits into
Closed
feat(prism): enable unattested Lium dispatch for PRISM jobs (T14)#54echobt wants to merge 572 commits into
echobt wants to merge 572 commits into
Conversation
…r-validator on-chain weights
…oute)
Make the master LLM gateway provider-agnostic and config-driven per the
llm-yunwu contract, superseding all deepseek/openrouter/deepseek-v4-pro guidance.
- tokens: add optional source (wire s) + model (wire m) claims to
GatewayTokenClaims/issue/issue_central_gate/verify (backward compatible).
- providers: replace deepseek/openrouter constants with a config-driven
provider registry (build_providers keyed by name; mock+real).
- gateway: single POST /llm/v1/{path} route; resolve provider+model from the
token source via config (default yunwu/claude-opus-4-8); overwrite request
body model + inject provider key; drop _enforce_model/DEEPSEEK_REQUIRED_MODEL;
provider-agnostic build + secret redaction.
- settings: GatewaySettings gains providers registry + default_provider +
default_model + sources; deepseek/openrouter fields removed.
- executor + assignment_coordination: emit BASE_LLM_GATEWAY_URL (={root}/llm/v1)
+ BASE_GATEWAY_TOKEN, stamp source=agent on assignment tokens.
- cli: build service from the yunwu key; mint-central-gate-token gains --source.
- update all platform gateway tests to the new contract (provider_mode=mock).…l-swarm.sh) Render the config-driven, provider-agnostic yunwu gateway block into deploy/swarm/master.yaml + the install-swarm.sh master-config: providers.yunwu (base_url https://yunwu.ai/v1, api_key_file /run/secrets/yunwu_api_key), default_provider/default_model (claude-opus-4-8), and per-source routes (agent/llm_review). install-swarm.sh now creates + mounts the single base_gateway_yunwu_api_key secret at /run/secrets/yunwu_api_key on the master proxy (provider_mode=real), keeps the mandatory HMAC gateway_token_secret, routes both central gates through the master gateway /llm/v1 with the scoped central-gate token (source=llm_review), and drops all deepseek/openrouter secret creation + mounts. Update the swarm/secret/deploy unit tests + runbook to expect the yunwu mount + absence of deepseek/openrouter. No yunwu key value in git. VAL-LLM-CODE-010
Reframe yunwu branding in deploy/swarm/README.md so the gateway is described generically: provider-agnostic, with the active provider + model chosen in master.yaml and injected server-side from the token source claim. Operational secret identifiers (base_gateway_yunwu_api_key / YUNWU_API_KEY / /run/secrets/yunwu_api_key) are unchanged and kept where operators need them, reframed as the configured-provider key. Docs-only; no code/config/prod change.
Remove stale DeepSeek/OpenRouter direct-provider-key prose from README.md and docs/operations/validator.md. The master gateway resolves provider+model server-side from the token 'source' claim (agent / llm_review) per master.yaml; challenge/eval services + validators hold no raw provider key and authenticate with a scoped gateway token (base_gateway_token, source=llm_review, /llm/v1). The single provider key lives only on the master gateway at /run/secrets/yunwu_api_key. Docs-only; no code/config/test changes.
yunwu intermittently closes the connection mid-response (httpx.RemoteProtocolError) or returns a transient 5xx; the gateway had no retry so these collapsed to caller-facing 502s. HttpLLMProvider.forward now wraps the request in a bounded retry loop (default 3 attempts, each on a FRESH client) with exponential backoff (0.25s then 0.5s), retrying the enumerated transient transport exceptions and 502/503/504. A 429 / other 4xx (and 2xx/3xx) returns immediately and is never retried. Retrying is safe because the gateway is fully buffered (no partial bytes reach the caller before an attempt succeeds). Retry logs name only the exception class + attempt counters, so no injected provider key can leak. Retry policy is configurable via ProviderConfig with production-safe defaults.
Allow the frontend origins (joinbase.ai, www, localhost dev ports and platform Vercel previews) to call the single public proxy cross-origin. Public, cookie-less reads only: GET/HEAD/OPTIONS, allow_credentials=False, so token-gated /v1/admin/* and signed uploads stay unchanged. Middleware wraps the whole ASGI app, covering forwarded /challenges/* (incl. SSE) and the included admin/registry routes. Origins overridable via allowed_cors_origins.
The agent-challenge own_runner reads task definitions only from node-local named volumes and fails closed on any digest mismatch; the runner image does not bake the ~89 task trees. Previously the volumes were empty after a fresh deploy, so every terminal-bench evaluation failed with terminal_bench_failed. - Add download-terminal-bench-cache.sh: reads the byte-exact public source (repo + revision + task count) from the frozen golden/dataset-digest.json, clones at the pinned commit, and stages the task dirs (stripping only the task-root .gitignore, as harbor does at packaging time) as a --source dir. - Wire provision_agent_challenge_cache() into install-swarm.sh (STEP 9b, after deploy_master, before deploy_challenges): download then acquire-agent-challenge-cache.sh --apply to copy + digest-verify onto the volumes. Both halves honor the script's dry-run/--apply gate. No secrets are read or required.
Add src/base/compute/ package: - provider.py: ProviderClient protocol, InstanceSpec (mandatory max_lifetime_hours/max_price_per_hour bounds), Offer/Instance, typed ProviderError/CostGuardrailError. - lium.py: LiumClient over httpx (X-API-Key, base https://lium.io/api). list_offers price filtering; provision refuses unbounded/over-priced specs before any network call, always sends bounded termination_hours, and terminates+verifies on any post-rent failure (try/finally); idempotent ensure_ssh_key/ensure_template; idempotent terminate; verify_terminated via GET /pods; stream_logs; watchtower digest; GET /users/me balance. API key never logged/repr'd/in errors. 50 offline respx unit tests (VAL-PROV-001/003/004/005/011/017/018 + secret hygiene). Full base gate green: 1316 passed, coverage 89.1%.
…able log level Batch of tested master/broker hardening (previously staged in the working tree): - Broker capacity: add `broker_max_concurrent_global` (server-wide cap on total concurrent broker jobs across all slugs) and route both run paths through a single `_register_job` choke point that enforces the per-slug + global caps atomically under the workload-ledger lock (over-limit -> WorkloadCapacityError -> HTTP 429 docker_quota_exceeded). Default None = unlimited (behavior-preserving). - WorkloadLedger.register accepts `max_concurrent_global`, checked under the same lock as insertion so check-and-register stays atomic. - Broker output bound: add `broker_log_limit_bytes` (default 5MB) and broker job lifecycle logging (accepted/finished/timed-out/rejected; never logs secrets). - Observability: configurable `log_level` (case-insensitive, falls back to INFO). - Submitter + install-swarm deploy wiring and matching unit tests. Validation: ruff clean, ruff format clean, full unit suite 1234 passed.
…-client secret hygiene
Add build_lium_worker_template (Lium CustomTemplateRequest payload) and build_targon_worker_app (Targon app definition) in src/base/compute/ worker_deployment.py. Both pin the docker image BY DIGEST via a shared pinned_image_reference helper; the image+digest default to the published prism-evaluator digest (M1 placeholder) and are inputs so M2 swaps in docker/Dockerfile.worker with a one-line change. Well-formedness is enforced (required fields, internal ports incl. 22, env plumbing, is_private, no embedded secrets) and covered by offline respx tests that also assert the whole compute suite makes zero real network calls under respx strict mode and needs no provider credentials (VAL-PROV-009/010/016).
…TESTS Add scripts/live_lium_e2e.py driving the M1 provider clients against production: read-only reachability (Lium users/me + executors + watchtower/digest, Targon inventory + apps) plus one batched Lium rental cycle (ensure ssh key + template -> rent cheapest suitable executor -> poll RUNNING -> ssh nvidia-smi -> logs -> DELETE -> verify gone -> record balance delta). Pod is deleted in a finally on every path; opt-in via BASE_LIVE_PROVIDER_TESTS=1 so the default suite stays offline.
…gpu shape - LiumClient.provision rejects sub-1-hour max_lifetime_hours so termination_hours never truncates to 0 (auto-termination stays on) - widen post-rent cleanup guard to key off rent success, not pod-id resolution: a transient GET /pods failure during resolution still best-effort terminates + verifies the just-rented pod - normalize Targon Offer.price_per_hour to per-GPU (cost_per_hour / gpu_count) so the per-GPU max_price cap filters multi-GPU shapes correctly - default WORKER_GPU_SHAPE 'h100' -> live-valid 'h100-small'
Add the miner-funded GPU worker plane registry (architecture.md sec 3.3),
gated behind compute.worker_plane_enabled:
- alembic 0009: worker_registrations + worker_faults + worker_request_nonces
on the existing chain (0008 -> 0009), no legacy table altered.
- WorkerCoordinationService mirroring validator_coordination patterns:
POST /v1/workers/register (sr25519 miner binding verified against the mock
metagraph, binding-nonce replay protection, no silent cross-owner rebind),
POST /v1/workers/{id}/heartbeat, GET /v1/workers (fleet: status/owner/
provider/last-seen/faults, authenticated-but-not-admin), and
GET /v1/workers/active?hotkey= (admission surface).
- Lifecycle pending -> active -> stale -> retired with
compute.worker_heartbeat_ttl_seconds (default 120); retired is terminal
(no heartbeat resurrection); staleness derived on read + a background pass.
- worker_auth: binding message/verify, signed-request verifier + registered-
worker/validator eligibility, worker nonce store.
- Wired into create_proxy_app + the master proxy CLI (flag-off => unmounted).
Unit + Postgres integration tests (15433) cover VAL-MASTER-001/002/015/016/
018/021 and VAL-AGENT-015.…x spurious 502s Broaden RETRYABLE_STATUS_CODES to include Cloudflare 520/522/524, Anthropic 529 (overloaded) and generic 500 so a transient edge/upstream overload is retried instead of collapsed into an opaque 502. Add full-jitter backoff and a 4th attempt. Add a true streaming passthrough for stream=true callers (the eval agent): open the upstream stream with a per-chunk read timeout so a slow large completion no longer trips Cloudflare 524 or the 30s buffered read cap, retrying only pre-first-byte. Log the real upstream status (status+source only, never bodies/keys) so a surfaced failure is no longer an opaque 502. The buffered llm_reviewer safety-gate path response shape is unchanged.
Add the miner-funded WorkerAgent (src/base/worker/): register under a
miner-signed binding, heartbeat to stay active, pull gpu-only replicas,
execute via the AssignmentExecutor seam on its local broker, and post
results that always carry an ExecutionProof envelope (sr25519 over
sha256('{manifest_sha256}:{unit_id}'), pinned identically to prism).
Extract the shared agent-loop primitives into base/coordination/agent_loop
(BackoffPolicy, is_transient_error, AgentCycleSummary, sleep_until,
backoff_sleep); the validator agent now imports them with behavior unchanged.
Add the master worker assignment plane (worker_assignments table + 0010
migration, worker-authenticated pull/result routes gated on registration
and liveness, never a validator permit) and wire it behind
compute.worker_plane_enabled.
Fulfills VAL-AGENT-002/003/004/005/006/007/008/016/017/018.Add a top-level `base worker` Typer app distinct from the legacy `base master worker` Swarm group: - deploy --provider local starts a miner-funded agent against a local master and reports it active within 60s. - deploy --provider lium|targon requires the provider key env (actionable refusal before any network), bounds offer selection by --max-price preferring an exact gpu_count executor (next-cheapest fallback), and never transmits the provider key to the master (pod env excludes it). - status renders the fleet from GET /v1/workers (signed as the worker key). Thread startup_commands through InstanceSpec, LiumClient.ensure_template and build_lium_worker_template (validated metachar-free) per the live-learned Lium rent constraint; keep docker/Dockerfile.worker's exec-form entrypoint metachar-free. Add WorkerSettings, worker/miner keypair resolvers, signed list_workers, config/worker.example.yaml, and unit tests. All behind compute.worker_plane_enabled; full base gate green.
Add WorkerAssignmentEngine that materializes gpu work-unit replicas onto ACTIVE distinct-owner workers behind compute.worker_plane_enabled: R=2 with self-evaluation exclusion (unit waits under sole-capacity scarcity), graceful degradation to R=1 with a recorded warning, per-worker gpu concurrency 1, and per-replica deadline/reassignment bounded by max_attempts. The validator AssignmentService skips worker-plane capabilities when the flag is on, so flag-OFF gpu routing to validators stays byte-identical to legacy.
… behind flag Reconcile replicated gpu worker results (architecture.md 3.3): matching ExecutionProof.manifest_sha256 forwards exactly one result to the challenge; divergent hashes dispute the unit (never forwarded, before or after audit) and create a validator-executor audit unit whose outcome writes worker_faults for the divergent worker(s), visible in fleet status. Single-replica reporting terminates deterministically (accept-after-degrade with warning); late/foreign posts stay rejected with replica state intact. All gated by compute.worker_plane_enabled (reconciler is None when off).
Legacy AssignmentService.reclaim_stale_assignments treated a null assigned_validator_hotkey as 'offline validator => reassignable', so under the worker plane a worker-owned prism PRIMARY (ASSIGNED, null hotkey by design) churned back to PENDING every MasterOrchestrationDriver.run_once pass. _assign_pending_in_session already skipped such units via worker_plane_capabilities; _reclaim_in_session did not. Factor the guard into AssignmentService._worker_plane_owns and apply it in BOTH paths so a worker-owned primary is neither assigned nor reclaimed here, while a genuinely stale validator unit (cpu, or gpu AUDIT with executor_kind= validator) is still reclaimed. Flag OFF keeps reclaim byte-identical to legacy.
The static --static-challenges installer path built the agent-challenge api and worker service env WITHOUT CHALLENGE_TERMINAL_BENCH_LOG_STREAM_URL or CHALLENGE_DOCKER_BROKER_NETWORK, even though both services are already multi-homed onto base_jobs_internal for exactly this purpose. Without them the terminal-bench runner JOB lands on the default bridge network, cannot resolve challenge-agent-challenge by name, and live task.log streaming silently no-ops. Add both to the shared ac_eval_env array (applied to api+worker), mirroring the dynamic seed path (cli_app._agent_challenge_own_runner_env), and lock it in with a parity regression test.
…budget The eval task concurrency was hardcoded (broker_max_concurrent_global=30, CHALLENGE_EVALUATION_CONCURRENCY=15), over-committing a 62 GiB manager. Derive it at install time from system RAM at a 4 GB/task budget: EVAL_TASK_CONCURRENCY = max(4, floor((TOTAL_RAM_GB - reserve) / 4)), reserve 10 (-> 13 on the 62 GiB box), with EVAL_RAM_TOTAL_GB / EVAL_RAM_RESERVE_GB / EVAL_TASK_CONCURRENCY overrides. Render it into both broker_max_concurrent_global and CHALLENGE_EVALUATION_CONCURRENCY, and pin a durable 2g runner (own_runner DooD client) memory ceiling. The 4 GB divisor only sizes concurrency; the inner terminal-bench task container keeps its higher 8 GiB ceiling (low actual usage).
…ollback The master image-updater fired docker service update --detach and never checked convergence, so a broken :latest image silently degraded proxy/broker with no revert, no retry budget, and no alert. Make auto-update durable: - New retry.py: RetryPolicy (bounded exponential backoff + equal jitter) + per-target RetryState. - image_updater.py: record pre-update digest as last-known-good (persisted under the release root, restart-durable), issue the update WITHOUT --detach, poll UpdateStatus.State to convergence; on paused/rolled_back/timeout re-pin the last-known-good digest and schedule exponential backoff; after max_attempts emit an image_update_failed alert and wait for a new digest (which resets). - Operator freeze: supervisor.image_update_hold (global) + per-target hold skip a service entirely so a pinned known-good digest is never overridden. - settings knobs (max_attempts/backoff base+max/hold), new alert kind, alert hook wired into the image-updater task.
Allow GET /v1/workers/active to authenticate via the prism<->master bridge shared bearer (the same token base forwards results to prism with) in addition to the signed-request path, so prism's live admission check works end-to-end (VAL-CROSS-004). The full-fleet GET /v1/workers stays signed-request only; the internal bearer is rejected there. Flag OFF => router unmounted (404) unchanged.
Services were created with no rolling-update policy, so a broken image roll was left paused/degraded with no auto-rollback. Add a Swarm self-healing policy to every long-lived first-party service create (postgres, master proxy/broker, challenge api + workers) in both the dynamic (swarm_backend.py) and static (install-swarm.sh) paths: --update-failure-action rollback, --update-monitor 45s, --update-max-failure-ratio 0 (a task that crashes on start within the window triggers rollback with no healthcheck needed), --update-order stop-first (required for singleton services on fixed host ports / per-node volumes), --rollback-failure-action pause, --rollback best-effort. Add a conservative HTTP /health container healthcheck (python urllib, 40s start-period) to the proxy, broker, and challenge api for runtime liveness. Kept in sync across both paths.
…loop service locks - challenge_image_updater: capture pre-roll digest, roll back to previous image when restart/health-check fails, per-slug retry backoff, and a challenge_image_update_failed alert once the retry budget is exhausted. - self_update: bounded retry on manifest+tarball downloads; retry-before- blacklist via swap_attempts so a transient boot failure no longer permanently blacklists a possibly-good version; timing/retry knobs sourced from settings.supervisor (defaults equal prior module constants). - service_locks: shared per-service update lock registry handed to both the image-updater and config-sync so their independent loops never issue overlapping docker service update on the same shared service (one lock at a time -> deadlock-free). - cli_app: DockerRuntimeController.rollback helper for challenge rollback.
CI runs 'mypy src tests' (not covered by the local ruff+pytest gate), which flagged three type errors introduced across the auto-update hardening: - self_update manifest _fetch annotated -> Any (json.loads is Any) so the dict-shaped raw.get access type-checks as it did before the retry refactor. - _FakeResponse.__exit__ returns None (was bool, which mypy rejects for a context manager that never suppresses). - test_swarm_backend replicated/job 'common' kwargs dict typed dict[str, Any] so the heterogeneous **unpack into SwarmServicePlan type-checks.
…e tests Lock stage defaults (review tdx.small/20GB, eval tdx.xlarge/100GB), disk bounds/billing, lifecycle budget with disk, provision disk_size emission, and frozen eval/review compose hashes before any production sizing change.
…faults Split review (tdx.small/20GB) and eval (tdx.xlarge/100GB) defaults, add disk bounds and billing helpers, and fold optional disk into projected cost while keeping CpuShape free of a disk field.
Charge both stages for compute plus disk against the shared $20 money cap and update the ordered CLI lifecycle fixture for disk-aware totals.
Thread stage disk_size_gb on eval/review deployment plans and send disk_size as a sibling of compose_file without mutating measured compose documents.
… CLI Default eval to tdx.xlarge, add --review-disk-size-gb/--eval-disk-size-gb, pass disks into lifecycle budget checks, and harden offline deploy test doubles.
Record review tdx.small/20GB and eval tdx.xlarge/100GB defaults, disk rate, and unchanged CPU-only $20 money cap in miner and validator self-deploy docs.
Lock DELETE /cvms/{id} allowlist behavior, 204/404 success, unique app_id
resolution, and CLI teardown without a phala binary before the fix lands.Allow DELETE /cvms/{id} (204/404 success) and refuse ambiguous app_id
matches when resolving teardown identity without a phala binary.Route teardown through PhalaCloudClient.delete_cvm, accept optional --cvm-id/--app-id, and resolve unique app_id matches from GET /cvms.
Assert delete_cvm path, ambiguous app_id refusal, and docs no longer require a phala binary in default_phala_teardown.
Describe DELETE /cvms/{id} as the primary path and keep manual phala
cvms list/delete strings for VAL-DEPLOY-020 verification.Check --cvm-id/--app-id before constructing PhalaCloudClient so missing identity fails closed without requiring credentials.
Mirror review prepare recovery: when eval prepare returns token-less secret_delivery, cancel the current run and retry so deploy receives a fresh one-shot capability. Production residual on submission 3 left attempts 1 and 2 stuck after standalone prepare/retry spent the token. Never cancel when the token is already present; never invent tokens.
Assert token-present skips cancel/retry, token-absent recovers via
cancel+retry, sticky absence raises RouteClientError, and the
{env_key,token} secret_delivery shape stays fail-closed. Also pin
review deploy fixtures to the production REVIEW_API_BASE_URL.…UN_TOKEN Cover the production closed loop where eval deploy injects the one-time run token into the CVM but never surfaces it for eval result. Assert --emit-run-token / --token-output handoff, fail-closed without either on live deploy, dry-run exemption, --output hygiene, and prepare/status redaction regressions.
Live eval deploy now requires --token-output and/or --emit-run-token so the host can post via eval result. Always include eval_run_id on success stdout; write the token only to the secure 0600 file or optional stdout key. Never put the token in --output, redacted prepare/status, logs, or exception text. CVM encrypted_env injection is unchanged.
…pends it The handoff check depends only on argv, but ran after _obtain_eval_prepare_with_token had already consumed the single EVAL_RUN_TOKEN delivery. A miner who omitted the flags burnt the token and then hit the error, reproducing the unrecoverable state the handoff exists to prevent. Validate the destination before any remote mutation and cover it with a test asserting eval_prepare is never called.
Lock RED→GREEN contract for plan vs CLI shape mismatch and optional --expected-measurement rtmr0 pin check before Phala create.
…la create Name plan and CLI shapes plus vm_shape/instance_type/rtmr0, warn that a stale allowlist pin only surfaces as a generic key-release denial, and abort before spend. Optional --expected-measurement compares rtmr0 with truncated prefixes only.
Document guest emit-only design, token handoff flags, host scrape of BASE_BENCHMARK_RESULT, eval result posting, teardown, and the shape/pin footgun so miners can finish a run from the docs alone.
…-sizing feat(agent-challenge): CVM sizing, token handoff, and Phala eval deploy hardening
…push (#52) * feat(agent-challenge): add temporary NO_PHALA host-local unattested mode Opt-in host execution while Phala CVMs are disabled. Results are explicitly marked unattested and cannot be forged as attested. Attested path stays byte-identical when the flag is off. Contradiction with attestation flags fails closed at startup. * feat(agent-challenge): complete NO_PHALA offline pipeline with weight push Drive the dual-flag-off analysis+own_runner path under NO_PHALA through scores and authenticated raw-weight push, with unattested provenance on completion metadata and CRITICAL push logs. Attested gates stay untouched. * feat(agent-challenge): route NO_PHALA LLM review via OpenRouter Grok Add OpenRouterReviewProvider for analyzer LLM review when NO_PHALA is on, with key resolution (env then opencode auth.json), operator config keys, and fail-closed cost/error handling. Gateway path stays default when mode is off. Document embed.env keys and cover selection/parsing with unit tests. * fix(agent-challenge): load file token and host-local NO_PHALA bench Production mounts the challenge token via shared_token_file; raw-weight push now resolves it the same way auth does so the loop does not skip. NO_PHALA + cli docker backend runs own_runner in-process to avoid nested DooD path mismatch on master embed. Default epoch_seconds matches master sealer (360s).
* feat(compute): add lium capacity scheduler Queue on empty 1-GPU Blackwell inventory instead of terminal-failing capacity misses so training plane can wait for stock. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(compute): add lium orphan pod terminator Prefix-owned orphan cleanup guards against leaked paid pods after interrupted training runs. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(config): add LiumTrainingSettings and prism dispatch variant Expose training-plane env knobs and prism_dispatch_variant so Lium wiring and orchestration can fail closed without an API key. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(compute): add lium training wiring factories Build capacity/orphan/client helpers from LiumTrainingSettings with fail-closed behavior when credentials are missing. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(compute): extend lium client for training GPU lock Add for_prism_training surface and training GPU lock helpers used by the host landmine training plane. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(cli): wire lium training plane into master CLI Register training-plane settings and factories on the CLI entry path alongside existing constation wiring. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(constation): extend digest allowlist for training plane Port host allowlist repository deltas required by Lium training constation checks. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(master): wire lium training into orchestration Hook capacity, orphan cleanup, and training client factories into the master orchestration loop. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(prism): add pod_boot contract validators Pure validators for SHA/URL/env forbid lists used before Lium pod boot. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(prism): gate checkpoint publish and surface intake status Add PRISM_CHECKPOINT_UPLOAD_ENABLED gate, public top-prism repo default, CheckpointPublishError→502 mapping, and last_status/last_error intake observability from the host landmine ports. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(prism): no-op local queue when worker plane owns GPU When worker-plane is on, process_next becomes a no-op and container processing refuses so Lium owns GPU execution. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * feat(docker): default prism worker-plane policy in entrypoint Port host base-master-entrypoint defaults for worker-plane and plagiarism LLM env into docker/master-entrypoint.sh. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * style: ruff format/import and mypy fixes for landmine ports Satisfy CI ruff/format/mypy/prism-checks: format touched modules, sort queue imports, narrow orphan terminate arg, cast orchestration request in tests, wrap long entrypoint assertion lines. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: echobt <154886644+echobt@users.noreply.github.com> Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
T14 unattested path: execution_backend=lium no longer requires a constation bundle at PrismWorker construction. Bundle remains optional API-compat; remote_provider/local_cpu stay rejected. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Mock FakeLiumClient proves bridge+tick provisions a pod and empty inventory stays queued with capacity_wait. No live billable rentals. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Ops keys for BASE_LIUM_TRAINING__*, worker PRISM_EXECUTION_BACKEND=lium, fail-closed defaults, and live 1M e2e handoff. Not TEE. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
📝 WalkthroughWalkthroughThis change documents the PRISM-to-Lium dispatch flow, adds a ChangesLium dispatch
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
🔧 Fix failing CI
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/unit/test_lium_dispatch_unattested.py`:
- Around line 120-124: Fix the Ruff E501 violations in
test_dispatch_enqueue_and_tick_provisions_mocked_pod by wrapping the overlong
docstring or statements at lines 121 and 181–182 to stay within the configured
88-character limit, without changing test behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 854b0c70-5cbb-4403-b787-50db7ae20760
📒 Files selected for processing (4)
docs/lium-dispatch.mdpackages/challenges/prism/config.example.yamlpackages/challenges/prism/tests/test_execution_backend_constation_gate.pytests/unit/test_lium_dispatch_unattested.py
Comment on lines
+120
to
+124
| async def test_dispatch_enqueue_and_tick_provisions_mocked_pod() -> None: | ||
| """Given Prism pending work + fake Lium inventory, When bridge+tick, Then pod provisioned. | ||
| End-to-end master-owned dispatch without constation and without live API. | ||
| """ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Fix the Ruff E501 failures.
Lines 121 and 182 exceed the configured 88-character limit, blocking CI.
Proposed fix
- """Given Prism pending work + fake Lium inventory, When bridge+tick, Then pod provisioned.+ """Given pending Prism work and fake Lium inventory, bridge+tick provisions a pod.
...
- """Given empty Lium inventory, When tick, Then lease stays queued (capacity_wait)."""+ """Given empty Lium inventory, tick leaves the lease queued (capacity_wait)."""Also applies to: 181-182
🧰 Tools
🪛 GitHub Actions: CI / 8_ruff.txt
[error] 121-121: ruff check failed (E501): Line too long (94 > 88).
🪛 GitHub Actions: CI / ruff
[error] 121-121: ruff E501 Line too long (94 > 88).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/unit/test_lium_dispatch_unattested.py` around lines 120 - 124, Fix the
Ruff E501 violations in test_dispatch_enqueue_and_tick_provisions_mocked_pod by
wrapping the overlong docstring or statements at lines 121 and 181–182 to stay
within the configured 88-character limit, without changing test behavior.
Source: Pipeline failures
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make PRISM actually dispatch training/eval jobs to our Lium machines (plan todo #14).
execution_backend=liumis allowed withoutconstation_bundle(compute-only; not TEE).test_lium_without_bundle_rejectedremoved.LiumCapacityScheduler.tickprovisions viaFakeLiumClient(no live billable rentals).docs/lium-dispatch.mdwith ops config keys + T15 1M e2e handoff.How to trigger dispatch
Master path:
try_build_lium_capacity_scheduler→bridge_pending_workenqueue →run_oncetick → provision when inventory free.Changes
packages/challenges/prism/tests/test_execution_backend_constation_gate.pytests/unit/test_lium_dispatch_unattested.pydocs/lium-dispatch.mdpackages/challenges/prism/config.example.yamlliumbackendTest plan
pytestgate: 5 passed (test_lium_without_bundle_accepted_unattested, …)pytestdispatch + adjacent: 27 passedrequire_execution_backend("lium")OK.omo/evidence/attested-isolated-platform/T14-lium-dispatch/ORCH-VERIFY.txtMust not
liumexecution_backendremainsbase_gpuRefs
BASE_LIVE_PROVIDER_TESTS=1onlySummary by CodeRabbit
New Features
Documentation
Bug Fixes