feat(agent-challenge): CVM sizing, token handoff, and Phala eval deploy hardening

packages/challenges/agent-challenge/docker/review/phala_pre_launch.sh

@@ -59,7 +59,7 @@ echo "Starting login process..."
 # Check if Docker credentials exist
 if [[ -n "$DSTACK_DOCKER_USERNAME" && -n "$DSTACK_DOCKER_PASSWORD" ]]; then
  echo "Docker credentials found"
- DOCKER_REGISTRY_TARGET="${DSTACK_DOCKER_REGISTRY:-docker.io}"
+ DOCKER_REGISTRY_TARGET="${DSTACK_DOCKER_REGISTRY:-ghcr.io}"
  echo "Target Docker registry: $DOCKER_REGISTRY_TARGET"
 
  # Check if already logged in

packages/challenges/agent-challenge/docs/miner/self-deploy.md

@@ -29,7 +29,7 @@ There is no closed agent-model catalog; personal finetunes are banned. Concepts:
 [Attestation TEE — agent-driven order](attestation-tee.md#agent-driven-order-package-verify--tree-sha--tee--eval).
 
 The mission is **CPU Intel TDX only** (no GPU) with a hard **$20** spend cap and a
-preference for the smallest CPU shape that works (`tdx.small`/`tdx.medium`). GPU
+per-stage CPU sizing (review `tdx.small` / 20 GB disk; eval `tdx.xlarge` / 100 GB disk so four concurrent Terminal-Bench tasks fit). GPU
 targets, over-cap shapes, and missing Phala credentials are refused **before** any
 Phala call.
 
@@ -126,7 +126,9 @@ python -m agent_challenge.selfdeploy review deploy \
  --openrouter-key-env OPENROUTER_API_KEY \
  --phala-api https://cloud-api.phala.com/api/v1 \
  --review-instance-type tdx.small \
- --eval-instance-type tdx.small \
+ --eval-instance-type tdx.xlarge \
+ --review-disk-size-gb 20 \
+ --eval-disk-size-gb 100 \
  --review-runtime-hours 6 \
  --eval-runtime-hours 6 \
  --money-cap-usd 20
@@ -145,7 +147,9 @@ python -m agent_challenge.selfdeploy review deploy \
  --openrouter-key-env OPENROUTER_API_KEY \
  --phala-api https://cloud-api.phala.com/api/v1 \
  --review-instance-type tdx.small \
- --eval-instance-type tdx.small \
+ --eval-instance-type tdx.xlarge \
+ --review-disk-size-gb 20 \
+ --eval-disk-size-gb 100 \
  --review-runtime-hours 6 \
  --eval-runtime-hours 6 \
  --money-cap-usd 20
@@ -279,6 +283,11 @@ python -m agent_challenge.selfdeploy review retry \
 
 ### `review teardown`
 
+Deletes the review CVM via the Phala Cloud HTTP API (`DELETE /cvms/{id}`).
+No external `phala` CLI binary is required. Pass `--cvm-id` from the deploy
+output, or `--app-id` (the plan `app_identity`) to resolve a unique match from
+`GET /cvms`. Ambiguous matches are refused.
+
 Delete the review CVM after allow, reject, expiry, cancellation, provider
 failure, verification failure, or interruption. The CLI uses the exact
 `phala cvms delete <id> -f` operation. If deletion fails, the command exits
@@ -287,6 +296,8 @@ logs).
 
 ```bash
 python -m agent_challenge.selfdeploy review teardown --cvm-id review-cvm-1
+# or, when the deploy output was lost but app_identity is known:
+python -m agent_challenge.selfdeploy review teardown --app-id <app_identity>
 ```
 
 ### `eval prepare`
@@ -321,6 +332,20 @@ spend projection counts both the review and eval stage shapes against the shared
 money cap. Signing again accepts either `--auto-sign` or explicit `--signature`
 + `--nonce` (+ optional `--timestamp`).
 
+**Host posts the result — the guest only emits.** The measured guest orchestrator
+emits the attested envelope on stdout (a line prefixed `BASE_BENCHMARK_RESULT=`)
+and exits. It never calls `eval result`. The miner host must scrape that line
+and post the exact bytes with the one-time `EVAL_RUN_TOKEN`.
+
+**Live deploy refuses to run without a token handoff.** Pass at least one of:
+
+- `--token-output PATH` (primary) — writes the token with mode `0600`
+- `--emit-run-token` — adds `eval_run_token` to success stdout JSON
+
+Validate the handoff destination **before** prepare spends the single delivery.
+Success stdout always includes `eval_run_id`. The token never appears in
+`--output` plan JSON, redacted prepare/status output, logs, or exception text.
+
 ```bash
 python -m agent_challenge.selfdeploy eval deploy \
  --base-url https://<challenge-host> \
@@ -329,9 +354,25 @@ python -m agent_challenge.selfdeploy eval deploy \
  --auto-sign \
  --llm-cost-limit-env LLM_COST_LIMIT \
  --phala-api https://cloud-api.phala.com/api/v1 \
- --eval-instance-type tdx.small \
- --money-cap-usd 20
-```
+ --eval-instance-type tdx.xlarge \
+ --review-disk-size-gb 20 \
+ --eval-disk-size-gb 100 \
+ --money-cap-usd 20 \
+ --token-output ~/.cache/agent-challenge/eval-run.token
+```
+
+Capture `eval_run_id` from deploy stdout (and keep the `0600` token file for
+`eval result`). Optional: `--expected-measurement PATH` compares plan `rtmr0`
+to a local pin JSON before Phala create (mismatch prints truncated prefixes only).
+
+**Shape / measurement-pin footgun.** `--eval-instance-type` must match the
+validator-issued plan `vm_shape` / `instance_type`. A shape change also requires
+a matching `rtmr0` pin on the validator allowlist and a re-prepare. A stale pin
+surfaces only as a generic key-release denial deep in the TEE flow — the CLI
+now aborts before Phala create and names both shapes plus `vm_shape` /
+`instance_type` / `rtmr0`. Prepare already spent the one-shot token on that
+abort path; re-run `eval deploy` (cancel+retry recovery) or explicit
+`eval cancel` + `eval retry` to re-prepare.
 
 Use `--dry-run` to validate the signed plan and show only safe names, digests,
 measurement metadata, and cost. Without a validator eval allowlist the dry-run
@@ -349,6 +390,42 @@ python -m agent_challenge.selfdeploy eval deploy \
 A post-create failure deletes the attributable eval CVM before the command
 returns.
 
+### Host post path after eval CVM finishes
+
+1. Wait for the guest to finish. Scrape guest stdout for the line prefixed
+ `BASE_BENCHMARK_RESULT=` and write the **exact** payload bytes after the
+ prefix into a file (production envelopes are on the order of ~18KB — do not
+ pretty-print or re-encode).
+2. Source the one-time token from the `0600` file without echoing it (no
+ `set -x`):
+
+```bash
+export EVAL_RUN_TOKEN="$(cat ~/.cache/agent-challenge/eval-run.token)"
+# never: echo "$EVAL_RUN_TOKEN" or set -x while the token is in the environment
+```
+
+3. Post with the deploy `eval_run_id`:
+
+```bash
+python -m agent_challenge.selfdeploy eval result \
+ --base-url https://<challenge-host> \
+ --run-id <eval_run_id_from_deploy_stdout> \
+ --result ./eval-result.json \
+ --token-env EVAL_RUN_TOKEN
+```
+
+4. Tear down the eval CVM:
+
+```bash
+python -m agent_challenge.selfdeploy eval teardown --cvm-id <cvm-id>
+# or, when only app_identity is known:
+python -m agent_challenge.selfdeploy eval teardown --app-id <app_identity>
+```
+
+Invariant: `EVAL_RUN_TOKEN` never belongs in plan JSON, compose, ordinary logs,
+or committed files — only the `0600` handoff file and/or one-shot stdout when
+`--emit-run-token` is set.
+
 ### `eval result`
 
 Post the exact CVM-emitted result bytes directly to
@@ -538,7 +615,7 @@ python -m agent_challenge.selfdeploy verdict \
 ### `deploy`
 
 Deploy a CPU-only, miner-funded CVM. Absent `--instance-type`, the smallest CPU
-shape (`tdx.small`) is chosen. A GPU instance type or GPU OS image is refused, and
+shape is stage-specific (review `tdx.small`, eval `tdx.xlarge`). A GPU instance type or GPU OS image is refused, and
 a shape whose projected cost would breach the money cap is refused, both before
 any provisioning. Use `--dry-run` to print the full plan (compose, image digest,
 instance type, region, key-release endpoint, projected cost) and make zero
@@ -613,10 +690,11 @@ python -m agent_challenge.selfdeploy teardown --cvm-id cvm-1
 
 Every CVM you deploy is **miner-funded** and must be deleted when you are done.
 The total mission spend cap is **$20**; always use the smallest CPU shape that
-works (`tdx.small`/`tdx.medium`) and never deploy a GPU CVM. Review and eval
+is stage-sized (review `tdx.small`/20 GB; eval `tdx.xlarge`/100 GB) and never deploy a GPU CVM. Disk is billed separately (~$0.000139/GB/hour) and is included in the projected $20 cap. Review and eval
 spend are projected together before either create.
 
-The `teardown` subcommand runs `phala cvms delete <id> -f` for you, but you can
+The `teardown` subcommand issues `DELETE /cvms/{id}` through the Phala Cloud
+HTTP client (no `phala` binary required). For manual verification you can still
 also delete and confirm directly with the `phala` CLI:
 
 ```

packages/challenges/agent-challenge/docs/validator/self-deploy.md

@@ -274,12 +274,12 @@ emitter.
 
 Any review or eval CVM created for live verification is miner-funded and subject
 to the cumulative review+eval money cap of **$20**. Prefer the smallest CPU shape
-(`tdx.small` / `tdx.medium`) and never a GPU shape. Miners must delete every
+(review `tdx.small`/20 GB disk; eval `tdx.xlarge`/100 GB disk) and never a GPU shape. Disk billing (~$0.000139/GB/hour) is included in the shared $20 projection. Miners must delete every
 attributable CVM after success, reject, expiry, provider failure, quote failure,
 cancellation, interruption, or result failure. Confirm none remain:
 
 ```bash
-phala cvms delete <id> -f
+HTTP `DELETE /cvms/{id}` (or manual `phala cvms delete <id> -f`)
 phala cvms list
 ```
 

packages/challenges/agent-challenge/golden/live-registry-refs.json

@@ -1,25 +1,28 @@
 {
  "schema": "harbor-independence/live-registry-refs@1",
- "note": "SIDE manifest (NOT the frozen golden dataset digest). Maps a small deterministic subset of Terminal-Bench 2.1 task_ids to PULLABLE, digest-pinned registry refs published to the miner's public Docker Hub namespace so an in-CVM DooD orchestrator can `docker pull` them for a live smoke E2E. This file does NOT affect golden/dataset-digest.json, its per-task content_digest_sha256, the canonical_content_digest_sha256, or the canonical compose/measurement. Resolution is opt-in and fail-closed (see agent_challenge.canonical.live_registry).",
+ "note": "SIDE manifest (NOT the frozen golden dataset digest). Maps a small deterministic subset of Terminal-Bench 2.1 task_ids to PULLABLE, digest-pinned GHCR refs under ghcr.io/baseintelligence so an in-CVM DooD orchestrator can pull them for a live smoke E2E. D6: only GHCR shipping refs are accepted (see agent_challenge.canonical.live_registry). This file does NOT affect golden/dataset-digest.json, its per-task content_digest_sha256, the canonical_content_digest_sha256, or the canonical compose/measurement. Resolution is opt-in and fail-closed. OPS_REQUIRED: orchestrator_image digest is the T1 local buildx manifest (not yet pushed to GHCR); task image digests retain prior content hashes under the new GHCR path pattern and must be republished to GHCR before live pull succeeds.",
  "dataset": "terminal-bench/terminal-bench-2-1",
- "namespace": "docker.io/mathiiss",
- "orchestrator_image": "docker.io/mathiiss/agent-challenge-canonical@sha256:02331f0909f617e333f113be376d353770a673669946bcddaac3c53cbde7c9d8",
- "orchestrator_image_source": "services.yaml build-canonical (`uv run python -m agent_challenge.canonical.build --build`) built reproducibly (BuildKit SOURCE_DATE_EPOCH + rewrite-timestamp, provenance/sbom off) and pushed to the miner Docker Hub namespace",
+ "namespace": "ghcr.io/baseintelligence",
+ "orchestrator_image": "ghcr.io/baseintelligence/agent-challenge-canonical@sha256:ea399cee1b3c9015024918a6070901e6b7b4bee432c8afe1e7dd40a95547e0bc",
+ "orchestrator_image_source": "T1 local agent-recipe buildx (SOURCE_DATE_EPOCH + rewrite-timestamp, provenance/sbom off) → manifest sha256:ea399cee1b3c9015024918a6070901e6b7b4bee432c8afe1e7dd40a95547e0bc; tag was ghcr.io/baseintelligence/agent-challenge-canonical:t1-local. OPS_REQUIRED: replace with the first published GHCR digest from agent-recipe publish-eval-image.yml before T3/prod pin.",
  "tasks": {
  "adaptive-rejection-sampler": {
- "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-adaptive-rejection-sampler@sha256:7c8bd5835f19506222805de68d65f83d6cca5b502f7d71dc5e2d9d4dd447c0c8",
+ "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-adaptive-rejection-sampler@sha256:7c8bd5835f19506222805de68d65f83d6cca5b502f7d71dc5e2d9d4dd447c0c8",
  "source_ref": "alexgshaw/adaptive-rejection-sampler:20251031",
- "content_digest_sha256": "bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de"
+ "content_digest_sha256": "bcaa2399985cd57666018025846289ab25e193ae0dd8fb7f0ffab2410c24d4de",
+ "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/"
  },
  "bn-fit-modify": {
- "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-bn-fit-modify@sha256:c0371862a0861f282eb206471452ea9aa8d494e3687bdc8167c96ab39d73db50",
+ "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-bn-fit-modify@sha256:c0371862a0861f282eb206471452ea9aa8d494e3687bdc8167c96ab39d73db50",
  "source_ref": "alexgshaw/bn-fit-modify:20251031",
- "content_digest_sha256": "b5f9644970c17ad9ddb46b7266f7bcd87c761d77d7e6f55d7cfe7284d5ff66e9"
+ "content_digest_sha256": "b5f9644970c17ad9ddb46b7266f7bcd87c761d77d7e6f55d7cfe7284d5ff66e9",
+ "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/"
  },
  "break-filter-js-from-html": {
- "registry_ref": "docker.io/mathiiss/agent-challenge-tb21-break-filter-js-from-html@sha256:2a5bd51bab582993befc1a24252c03af673cb1db7cf7d0b2195d66a42a3872a7",
+ "registry_ref": "ghcr.io/baseintelligence/agent-challenge-tb21-break-filter-js-from-html@sha256:2a5bd51bab582993befc1a24252c03af673cb1db7cf7d0b2195d66a42a3872a7",
  "source_ref": "alexgshaw/break-filter-js-from-html:20251031",
- "content_digest_sha256": "678008d1a4fd1e6e1b9b3cc9a327219fe4b410a31eafc52e9099bbf947eea600"
+ "content_digest_sha256": "678008d1a4fd1e6e1b9b3cc9a327219fe4b410a31eafc52e9099bbf947eea600",
+ "ops_status": "BLOCKED_REPUBLISH — path retargeted to GHCR; digest is historical content pin until image is pushed under ghcr.io/baseintelligence/"
  }
  }
 }

packages/challenges/agent-challenge/src/agent_challenge/canonical/live_registry.py

@@ -4,12 +4,15 @@
 Terminal-Bench task by a *bare content digest* (``harbor_registry_ref =
 "sha256:<64hex>"``) with NO repository, so an in-CVM DooD orchestrator cannot
 ``docker pull`` it. For a live smoke E2E a small deterministic subset of task
-images is published to the miner's public Docker Hub namespace as pullable,
-digest-pinned refs and recorded in a SEPARATE side manifest
+images is published to the org GHCR namespace (``ghcr.io/baseintelligence/…``)
+as pullable, digest-pinned refs and recorded in a SEPARATE side manifest
 (``golden/live-registry-refs.json``). That side manifest never touches
 ``dataset-digest.json`` -- the frozen content digests and the canonical
 measurement stay byte-identical.
 
+D6: shipping live refs MUST be GHCR. ``docker.io`` and the retired
+``mathiiss`` namespace are rejected fail-closed at parse time.
+
 Resolution is **opt-in and fail-closed**: with no live manifest configured
 (no explicit path, no env var) callers get NO live refs and fall back to the
 existing per-task behavior, so flag-off / offline runs are byte-identical. When a
@@ -50,6 +53,9 @@
 # and an un-namespaced ``name@sha256`` so only a real registry ref is accepted.
 _PULLABLE_REF_RE = re.compile(r"^(?=[^@]*/)[A-Za-z0-9][\w.\-/:]*@sha256:[0-9a-f]{64}$")
 
+#: Shipping live-registry refs must live under this registry host prefix (D6).
+LIVE_REGISTRY_HOST_PREFIX = "ghcr.io/"
+
 
 class LiveRegistryError(ValueError):
  """The live-registry side manifest is missing, malformed, or not pullable."""
@@ -83,6 +89,32 @@ def assert_pullable_ref(ref: Any, *, what: str = "registry ref") -> str:
  return ref # type: ignore[return-value]
 
 
+def assert_live_registry_ref(ref: Any, *, what: str = "registry ref") -> str:
+ """Return ``ref`` if it is a GHCR digest-pinned shipping ref (D6), else raise.
+
+ Builds on :func:`assert_pullable_ref`, then rejects ``docker.io``, the
+ retired ``mathiiss`` namespace, and any non-GHCR host so live smoke never
+ pins a Hub image again.
+ """
+
+ pinned = assert_pullable_ref(ref, what=what)
+ lowered = pinned.lower()
+ if "mathiiss" in lowered:
+ raise LiveRegistryError(
+ f"{what} must not use the retired mathiiss namespace (D6), got {pinned!r}"
+ )
+ if lowered.startswith("docker.io/") or lowered.startswith("index.docker.io/"):
+ raise LiveRegistryError(
+ f"{what} must not use docker.io (D6 — GHCR only), got {pinned!r}"
+ )
+ if not lowered.startswith(LIVE_REGISTRY_HOST_PREFIX):
+ raise LiveRegistryError(
+ f"{what} must be a GHCR digest-pinned ref "
+ f"({LIVE_REGISTRY_HOST_PREFIX}…@sha256:<64hex>), got {pinned!r}"
+ )
+ return pinned
+
+
 @dataclass(frozen=True)
 class LiveRegistry:
  """Parsed live-registry side manifest.
@@ -111,23 +143,24 @@ def __bool__(self) -> bool:
 
 
 def _ref_from_entry(task_id: str, entry: Any) -> str:
- """Extract + validate the pullable ref from a manifest task entry."""
+ """Extract + validate the pullable GHCR ref from a manifest task entry."""
 
  if isinstance(entry, str):
  ref = entry
  elif isinstance(entry, Mapping):
  ref = entry.get("registry_ref")
  else:
  raise LiveRegistryError(f"live-registry task {task_id!r} is not a string or mapping")
- return assert_pullable_ref(ref, what=f"live-registry ref for task {task_id!r}")
+ return assert_live_registry_ref(ref, what=f"live-registry ref for task {task_id!r}")
 
 
 def parse_live_registry(data: Mapping[str, Any]) -> LiveRegistry:
  """Parse + validate a live-registry side-manifest document.
 
- Every task ref must be a pullable ``repo@sha256`` ref (a bare content digest
- or floating tag is rejected). ``orchestrator_image``, when present, must be
- pullable too. Task keys are normalized to their bare name.
+ Every task ref must be a GHCR pullable ``repo@sha256`` ref (a bare content
+ digest, floating tag, ``docker.io``, or retired ``mathiiss`` namespace is
+ rejected). ``orchestrator_image``, when present, must be GHCR-pullable too.
+ Task keys are normalized to their bare name.
  """
 
  if not isinstance(data, Mapping):
@@ -143,7 +176,7 @@ def parse_live_registry(data: Mapping[str, Any]) -> LiveRegistry:
 
  orchestrator = data.get("orchestrator_image")
  if orchestrator is not None:
- orchestrator = assert_pullable_ref(orchestrator, what="orchestrator_image")
+ orchestrator = assert_live_registry_ref(orchestrator, what="orchestrator_image")
 
  return LiveRegistry(orchestrator_image=orchestrator, task_refs=task_refs, raw=dict(data))
 
@@ -203,9 +236,11 @@ def resolve_live_registry_refs(
  "DEFAULT_LIVE_REGISTRY_PATH",
  "LIVE_REGISTRY_ENV",
  "LIVE_REGISTRY_FILENAME",
+ "LIVE_REGISTRY_HOST_PREFIX",
  "LIVE_REGISTRY_SCHEMA",
  "LiveRegistry",
  "LiveRegistryError",
+ "assert_live_registry_ref",
  "assert_pullable_ref",
  "is_pullable_ref",
  "load_live_registry",