Roll out one warm ACK research worker and prove cache-hit mission latency

[proerror77]

Parent

#240

What to build

Roll out the reviewed exact-main research image, authenticated snapshot cache seam, and mission dispatcher on the existing Aliyun research plane under one named runtime controller. Keep one bounded worker ready during the benchmark session, pre-pull the immutable image digest, stage one verified snapshot in the read-only cache, and publish cold, warm cache-miss, warm cache-hit, and concurrent mission timing evidence. Return unused burst capacity to the documented idle state after the proof.

This issue is the cloud rollout and end-to-end performance proof. It does not change code behavior, research semantics, collectors, live trading, or promotion policy.

Acceptance criteria

• Before mutation, record the live ACK cluster, namespace, worker pool, node runtime/OS, registry endpoint/edition, current capacity, active Jobs, and current controller ownership.
• Use one named controller for all worker, pre-pull, cache, Job, and cleanup commands; all other agents remain read-only.
• Deploy only reviewed exact-main artifacts from immutable image digests and record the source revision and image digest in the benchmark evidence.
• Keep exactly one bounded warm research worker for the active benchmark window and document the idle-cost and shutdown/scale-in action.
• Pre-pull the exact research image digest on the eligible warm worker without requiring an ACR Enterprise migration.
• Stage one authenticated snapshot cache entry keyed by its expected SHA-256 and prove the runner independently verifies the cached bytes.
• Run and publish machine-readable milestones for: cold worker/cache miss, warm worker/cache miss, warm worker/cache hit, and at least four isolated concurrent missions.
• A warm cache-hit mission targets evaluator start within 90 seconds; if it misses, publish the measured dominant stage and stop rather than silently adding infrastructure.
• The cache-hit proof shows no OSS snapshot transfer and no Cargo/Docker build while preserving mission SHA, snapshot SHA, image digest, evaluator policy, and immutable result binding.
• Concurrent runs retain distinct mission, attempt, checkpoint, writable workspace, and result identities and preserve market_id/common-time cohort split semantics.
• A failed/paused attempt publishes its immutable evidence bundle and one retry resumes only from the pinned bundle without identity drift.
• After the benchmark, delete short-lived URL secrets and completed test Jobs, scale unused burst workers down, and report any intentionally retained warm worker/cache with its owner and expiry condition.
• Do not enable ACR Enterprise P2P/on-demand loading, Fluid, ClickHouse, Kafka, Redis, external authoritative sandboxes, or a custom operator unless a separate reviewed issue is created from measured evidence.

Blocked by

• Publish ACR images from the smoke-tested exact-main release artifact #317
• Materialize authenticated ready cohorts into cached snapshots before Mission admission #320
• Add an explicit no-alpha pipeline_smoke run mode #323

#236, #241, and #242 are completed foundations; this rollout must use their current exact-main successors rather than reopening them.

[proerror77]

Runtime controller and pre-mutation baseline

Named runtime controller: this Codex thread /root only. No sub-agent or other task may mutate the ACK research plane, worker pool, image pre-pull, cache, benchmark Jobs, or cleanup until this comment is explicitly superseded.

Read-only baseline recorded on 2026-07-23:

• Cluster: monday-research-apne1 / c4d1db514b47a4c40995d2ec4bcc8c0ca, ACK ManagedKubernetes, Kubernetes 1.36.1-aliyun.1, region ap-northeast-1, VPC vpc-6wesy84ixw2esl6lb3ov5, state running.
• Namespace: monday-research, state Active.
• Current capacity: one Ready system node, ecs.u1-c1m2.large, Alibaba Cloud Linux 3, containerd 2.1.8. No research worker is currently present.
• Research pool: monday-research-spot / npa509e34c913c45a59a33b805a66267e5, ecs.u1-c1m4.xlarge (4C16G), SpotAsPriceGo, 40 GiB PL0 system disk plus 100 GiB PL1 ext4 mounted at /work, labels include workload=backtest, autoscaling enabled with min 0 and max 4, current desired/healthy/total nodes all 0.
• Runtime load: 117 historical Jobs, 0 active Jobs; 61 retained Pods, 0 Running and 0 Pending. No competing benchmark workload exists.
• Cache/storage baseline: no DaemonSet cache and no general research cache PVC. Four old static evidence PVCs are retained and out of scope.
• Registry: existing Tokyo Personal Edition VPC endpoint crpi-ygobwehhof7qs9m3-vpc.ap-northeast-1.personal.cr.aliyuncs.com; no Enterprise/P2P/on-demand-loading migration will be used.
• ACR pull secret monday-acr exists; no secret data was read.
• Exact source revision selected for the benchmark: current origin/main 05d19663ef90cdfe699b7f782310d1af55542179.
• Latest published research image predates this revision, so the next action is one targeted research-runner workflow dispatch for the exact main SHA. No Cargo or Docker build will occur in benchmark Jobs.

Mutation boundary: publish one exact-main image, scale the existing research pool to one bounded warm worker, pre-pull that immutable digest, stage one authenticated snapshot cache entry, run the four required benchmark modes, then remove test Jobs/secrets and return burst capacity to zero unless a retained warm worker is explicitly reported with owner and expiry.

[proerror77]

Pre-dispatch identity update: main advanced before the guarded workflow dispatch, so the old-SHA dispatch was intentionally not executed. No image or cluster mutation occurred. The benchmark exact-main revision is now 4d2c2291bb0fe823d790cc7ab9f2dc46ffdb5dbd (PR #253 merged); all other controller and scope boundaries remain unchanged.

[proerror77]

Controlled stop after exact-main image publication

• Exact-main source: 4d2c2291bb0fe823d790cc7ab9f2dc46ffdb5dbd.
• Published immutable research image: crpi-ygobwehhof7qs9m3-vpc.ap-northeast-1.personal.cr.aliyuncs.com/wildcard0923/research-runner@sha256:42037971f09cb24721407e40eadcee60e0b65868cd5641e34deddd56f017398c.
• ACR run: https://github.com/proerror77/monday/actions/runs/29984194849, success, 20m31s.
• Trace: Docker source-build step took 1140.2s; manifest push took 11.7s and cache export 24.5s. PR ci: package cached research binaries #244 intentionally excluded ACR publish/cross-workflow artifact reuse, so this is a separate measured release-pipeline gap.
• ACK remains unchanged: research worker pool is still 0; no warm holder, cache PVC, benchmark Job, or short-lived Secret was created.
• Benchmark stop reason: OSS contains only the older BTC 5m ResearchSnapshot fc2851a7...d79772f; exact-main compatibility evidence already rejects its upstream immutable evidence with unsupported immutable evidence manifest contract and decision reject_window_no_retry. Reusing it would bypass the established trust/stop rule. Issue BTC 5m cohort to sealed ResearchSnapshot #169 remains open and has not produced a current compatible snapshot.

The runtime controller claim is released while #243 is paused. Resume only after a compatible authenticated ResearchSnapshot exists, then re-establish one named controller before any ACK mutation.