PRD: make agent research loops start without rebuilds or repeated snapshot transfers

[proerror77]

Problem Statement

Prediction-market research currently pays release-pipeline and data-staging costs inside the hypothesis loop. A small research direction can wait for Rust compilation, image construction, node startup, and repeated immutable snapshot download before the evaluator begins. This prevents many agents from submitting independent bounded hypotheses, makes parallel research operationally expensive, and obscures whether time is spent scheduling, pulling an image, staging data, evaluating, or publishing evidence.

Solution

Provide one fast, auditable seam from an immutable ResearchMission submission to an immutable result bundle. Build the exact-main research image once per source revision, keep at least one research worker ready during an active research session, reuse content-addressed snapshot bytes through a read-only cache, and submit independent missions through the existing ACK Job boundary. Every run remains bound to its mission SHA, snapshot SHA, image digest, evaluator policy, budget, and isolated result path.

Agents may use fast external sandboxes to propose hypotheses or mission parameters, but authoritative evaluation stays beside Aliyun OSS in the research VPC. Research-only parameter changes do not open code PRs or rebuild Rust. Code, data-contract, evaluator, and deployment changes continue through their own review and rollout units.

The highest verification seam is:

Given an immutable mission, snapshot SHA, and exact-main image digest, the system starts independent research work without compiling source or transferring an already cached snapshot from OSS, and publishes a verifiable resumable result bundle.

User Stories

1. As a researcher, I want a research-only hypothesis to run without a code build, so that feedback is not delayed by release work.
2. As a researcher, I want repeated missions to reuse the same immutable snapshot, so that identical data is not downloaded for every hypothesis.
3. As a researcher, I want a warm mission to enter the evaluator within a measured latency target, so that interactive iteration is practical.
4. As a researcher, I want several independent missions to run concurrently, so that elapsed time is determined by the slowest batch rather than the sum of all trials.
5. As a research agent, I want to submit a bounded typed mission instead of editing shared source code, so that agents do not create overlapping branches for parameter research.
6. As a research agent, I want explicit hypothesis, symbol, horizon, cohort, budget, and stop rules, so that one mission cannot expand indefinitely.
7. As an evaluator owner, I want every mission pinned to an exact evaluator policy and image digest, so that cached execution cannot weaken reproducibility.
8. As a data owner, I want cache identity to be the authenticated snapshot SHA, so that cache hits cannot substitute a different dataset.
9. As a security reviewer, I want cached snapshot inputs to remain read-only and result directories isolated per mission, so that one worker cannot contaminate another result.
10. As an operator, I want corrupted or mismatched cache content to fail closed and refetch from the trusted source, so that speed never replaces integrity.
11. As an operator, I want timestamps for submission, scheduling, image readiness, snapshot readiness, evaluator start, and completion, so that optimization is based on the real bottleneck.
12. As an operator, I want the first version to use existing ACK Jobs and OSS rather than a new queue platform, so that the fast path has minimal operational surface.
13. As an operator, I want one named controller for worker-pool and cache rollout, so that concurrent agents cannot issue conflicting cloud mutations.
14. As a cost owner, I want one warm worker during active research and bounded scale-out for bursts, so that latency and idle cost are both visible.
15. As a cost owner, I want warm-worker shutdown to be an explicit operational action after the research session, so that unused compute does not remain running.
16. As a platform engineer, I want exact-main binaries compiled once and packaged without a second Rust build, so that CI and ACR do not duplicate compilation.
17. As a platform engineer, I want the exact image digest pre-pulled before missions use it, so that Pod startup is not dominated by image distribution.
18. As a platform engineer, I want snapshot cache rollout independent of evaluator behavior, so that data acceleration can be rolled back without changing research semantics.
19. As a platform engineer, I want dispatcher rollout independent of worker or cache rollout, so that submission automation cannot hide infrastructure failures.
20. As a reviewer, I want each infrastructure slice to remain one rollback unit with focused evidence, so that an optimization does not become another oversized PR.
21. As a Polymarket researcher, I want mission partitions to preserve symbol, horizon, market_id episode, token-side, and common-time cohort identities, so that parallel work does not introduce leakage.
22. As a Polymarket researcher, I want sampled settlement research and full-tick execution research to remain different evidence products, so that cache reuse does not blur their authority.
23. As a control-loop owner, I want paused work to resume only from a pinned immutable result bundle, so that rapid retries do not reset or forge search state.
24. As a control-loop owner, I want terminal budgets and repeated-failure stop rules, so that fast infrastructure does not create an unbounded research loop.
25. As a user, I want cold, warm, cache-hit, and concurrent benchmark results, so that the claimed speedup is demonstrated rather than assumed.

Implementation Decisions

• Keep Platform CI, image publication, snapshot preparation, research execution, and result publication as separate authority and rollback boundaries.
• Reuse the existing ResearchMission, prediction execute, immutable result bundle, ACK Job, and resume contracts. Do not introduce a second research runtime.
• Reuse the existing issue for cacheable binary compilation and prebuilt Docker packaging as the image-release slice.
• The first data cache stores immutable snapshot archive bytes keyed by authenticated SHA-256. Every consumer verifies the expected digest before use.
• Do not reuse writable extraction or result directories between missions. A cache hit provides immutable input only.
• Prefer the existing Aliyun internal OSS endpoint and ACK worker pool for authoritative evaluation. External sandboxes may prepare typed missions but do not produce promotion-grade evidence.
• Keep one bounded warm worker during an explicitly active research session. Scale additional workers through the existing ACK node-pool boundary.
• Pre-pull the exact immutable research image digest on eligible warm workers. A tag alone is not an execution identity.
• Use the Kubernetes API and existing Job contract as the first submission queue. Do not add Kafka, Redis, Argo, or a custom Kubernetes operator.
• The dispatcher accepts only mission identity, mission object reference and SHA, snapshot object reference and SHA, image digest, output object identity, resource profile, and optional pinned resume bundle.
• Mission output objects are immutable and unique per attempt. Parallel missions never share a writable DuckDB database or result path.
• Record six latency milestones: submitted, scheduled, image_ready, snapshot_ready, evaluator_started, and completed.
• Treat the initial latency numbers as measured service objectives, not correctness requirements: warm cache-hit work should target evaluator start within 90 seconds; cold and cache-miss results must be reported separately.
• Preserve event-disjoint and common-time cohort split rules. Infrastructure parallelism must not change model samples, labels, settlement authority, or execution accounting.
• Worker/cache deployment is performed by one named runtime controller only after code and configuration are reviewed.
• Verify current ACK, registry edition, node OS/runtime, and cache support before selecting Alibaba Cloud image-acceleration products. The first slice must not require an ACR Enterprise upgrade.

Testing Decisions

• Test at the highest seam: submit an immutable mission and prove the result bundle binds the same mission SHA, snapshot SHA, image digest, and evaluator policy.
• Compare one cold run, one warm image/cache-miss run, one warm cache-hit run, and one concurrent multi-mission run.
• Assert that a research-only mission submission performs no Cargo or Docker build.
• Assert that a cache hit performs no OSS snapshot transfer while still verifying SHA-256.
• Corrupt cached bytes and prove the mission fails closed or replaces them from the trusted immutable object before evaluator start.
• Submit two missions against the same snapshot and prove their result paths, checkpoint state, and writable files remain isolated.
• Exercise retry/resume from a pinned prior bundle and reject mission, snapshot, image, or policy identity drift.
• Verify the dispatcher rejects mutable image tags, missing digests, duplicate output identities, unsupported resource profiles, and incomplete resume pairs.
• Record benchmark milestones as machine-readable evidence rather than relying only on workflow duration.
• Keep cloud rollout evidence separate from unit and manifest validation; a rendered Job is not proof that the deployed worker/cache is active.

Out of Scope

• Replacing the Polymarket collector or changing collector deployment.
• Changing ResearchSnapshot, evaluator, MCTS, factor, label, settlement, execution, or promotion semantics.
• Creating a distributed MCTS implementation.
• Deploying ClickHouse, Kafka, Redis, Argo Workflows, or a custom Kubernetes operator.
• Moving authoritative research evidence to Modal, E2B, or another external sandbox.
• ACR Enterprise Edition migration, P2P image distribution, or on-demand image conversion in the first version.
• Live trading, order submission, OMS changes, execution credentials, or production promotion.
• Claiming profitability or strategy readiness from infrastructure latency improvements.

Further Notes

The first useful proof is not a new platform. It is the existing immutable research runner starting from a cached exact input on a ready worker, producing the same verified result contract with measured lower latency. Additional registry or distributed-data products should be added only when the milestone measurements show that the simpler path is insufficient.