Uh oh!
There was an error while loading. Please reload this page.
feat(prism): AutoModel recipe 2.0 + miner-funded Lium BYOK - #120
Merged
Conversation
Non-normative spike doc synthesizing research into a single scoring recommendation; no source code changed.
Validators re-entering the on-chain 100-block window (e.g. after a restart — the submit dedupe is in-memory) pool-submitted a doomed extrinsic every tick and then blocked ~4 blocks in dispatch confirmation, wedging the async runtime long enough to fail the docker healthcheck (staging validator smoke failed twice on this). submit_timelocked_weights/set_weights now read LastUpdate + the window up front and return RateLimited immediately — no pool spam, no confirm block; the post-submit confirm stays the source of truth for the mid-flight race. Also gives staging services the ordered endpoint failover list (test.finney <-> test.chain, each service keeps its primary).
Single-file harness becomes a 12-file embedded package (main.py + prismlib/ + eval/ registry). Miner code runs in an unshare --net subprocess with an FD-3 result channel and per-phase timeouts (loud fallback when netns unavailable). Seeded harness-owned train_stream with a real tokens_seen counter; G6 intermediate probes fire from prism_telemetry.report; additive METRICS_JSON v2 parsed by prism-lium. Recipe pin now hashes the full harness file set.
composite.rs implements research/12 steps 0-6: fixed-anchor normalization, two-level group means, lexicographic gates, mirror-gap penalty, weighted geometric composite, clustered bootstrap SE, LCB -> lattice. Versioned placeholder anchor set (anchors/v0.json). PRISM_SCORING_MODE=shadow keeps the v2 bpb path bit-identical; composite mode is fail-closed. SCORING_VERSION_V3 constant added (v2 remains live default).
Transformer++ (340.9M: RoPE/RMSNorm/SwiGLU/tied, 24x1024) and 3:1 gated delta-net + sliding-window-attention hybrid (341.3M), pure torch, chunked WY delta rule (chunk 64, fp32 core), arbitrary eval context (no position tables). train_stream primary with legacy dataset_path fallback; CPU verified (delta recurrence 1e-6, SWA 1.9e-9, 2-step train both paths).
Seeded procedural generators (recall, reasoning, long-context families) plus eight eval modules g1..g8 with the run(model, ctx) contract, public dev family under eval/public_dev, per-item cluster records for clustered bootstrap, and the v3 two-phase flow: train subprocess saves a state_dict checkpoint, hard kill, parent gate (survivors + JIT reset), then a fresh netns eval subprocess rebuilds the model and runs the battery with secret-seed Cantor derivation. v1 flow byte-identical by default.
Migration 0013 (prism_eval_run/group/metric, mirror pairs, anchor sets,
prereg, metric reports). New prism-zoneb crate (envelope schema, hash
chain, verdicts ok|flagged|quarantined) and prism-eval-store crate
(EvalStore memory+postgres impls, finalize_composite glue, zone b
ingest). API: eval detail on submissions/{id}, /metrics?zone=a|b,
/anchors, /preregistration. Orchestrator wiring left for integration.Source-tree submissions (zip <=2MiB, prism.toml entry, vendor.lock hash check, banned-binary/AST pattern scan shared via cheatguard_patterns.json, canonical tree hash idempotency; legacy two-script ids unchanged). Lium client streams harness output, stages private eval assets + file-delivered SECRET_SEED after PHASE_TRAIN_DONE via a second ssh channel, fail-closed when assets are configured but the tier is not private. prismlib cheatguard battery (AST audit, thread diff, seed Cantor combine, determinism/poison/ timing checks). Attribution module (2x2 kernel-swap run builder) in prism-recipe, re-exported through prism-challenge.
- register cheatguard.py + cheatguard_patterns.json in HARNESS_FILES
- embed v3 baselines (transformer_pp, hybrid_delta) via baselines.rs
- orchestrator: with_eval_store + finalize_for_submission composite path,
cap_exceeded terminal guard, scoring_mode config (legacy default)
- new prism-attribution crate: POST /v1/submissions/{id}/attribution
- harness two-phase: _await_eval_assets .ready wait, cap_exceeded payload
- RemoteExecResult: serde-flatten extra map for METRICS_JSON v2 payload
- docs: PRISM.md, PRISM_RECIPE.md, external-miner, COMPLETENESS, spikes
- tests: e2e_v3_wiring; all gates pass (clippy, fmt, 168 suites, xtask x5)- rollup.py: emit flat battery.metrics (org.* keys) + mirrors harness-side
- contract test: real harness METRICS_JSON v2 fixture through finalize path
- torch_seed: Cantor lattice reduced mod 2^63-1 for torch APIs (g7/g8)
- g4_reasoning: tuple+list TypeError fixed; battery groups all ok
- PRISM_TEST_TRAIN_ROWS/VAL_ROWS overrides; harness smokes pass with torch
- POST /v1/submissions/{id}/zone-b with full validation lattice
- CI: harness-smoke job (cpu torch) gates deploy-staging- harness upload embedded ~240KB base64 in one ssh argv entry, exceeding MAX_ARG_STRLEN (128KB): every real-pod run failed at spawn (os error 7) - now deterministic ustar streamed over stdin (stage_eval_assets pattern); argv holds only tiny extract + run commands - fix latent /dev/urandom infinite read in random_seed_hex (read_exact 16B) - regression tests: argv stays small, tar contents byte-identical, deterministic - drop base64 dep; prism-lium LOC 1499 -> 1498
- chunked delta rule saved ~5.8 GiB of fp32 pairwise-decay tensors per block for backward (x18 blocks = ~104 GiB) -> CUDA OOM at first forward - wrap each block in grad-enabled-only activation checkpointing; eval and probes bypass it, gradients bitwise-identical, params unchanged (341.3M) - measured CPU peak 79.3 -> 9.3 GiB (fwd+bwd); estimated pod peak ~14 GiB
- unknown cheat_codes coerce to CheatCode::Other (verdict preserved, audited) instead of Parse -> llm_infra -> full 6h pod retrain - missing_telemetry_hooks added to tool schema + parser (prompt/parser drift) - token budget 24k -> 120k for v3 source trees; budget death triggers one forced-verdict turn before NoVerdict - post-run retries resume from persisted measurement (no re-provision); reset_for_retry retains pod/receipt/metrics/bpb/telemetry - prism domain rules: try/except ImportError telemetry shim is no violation - verified live: OpenRouter review of hybrid_delta completes Clean in 9s
CPU Generator + CUDA device crashed the G7 TPOT loop (swallowed), and both reference baselines ignored prism_width_multiplier so G8 stubbed as width_knob_unsupported — leaving org.g7.tpot_ms_32k and org.g8.mup_lr_stability missing and composite Ineligible.
The tokenizer was a challenge rule (GPT2TokenizerFast in every phase entry) although it is an architecture decision a miner should own. It is now part of the submission: prismlib.tokenizer is the single resolution path (tokenizer/ files, a build_tokenizer(ctx) hook beside build_model, else the pinned fallback) shared by the v1 entry and both v3 children, injected as ctx["tokenizer"] with ctx["vocab_size"]. Fail-closed rather than silently scoring the wrong tokenizer: bounded vocab, probe ids inside that vocab, encode/decode roundtrip, and a probe fingerprint the train phase stores in the checkpoint and the eval phase must reproduce. A hook in training.py is rejected (eval imports the architecture module only), and intake refuses tokenizer/ files while the pod stages seam projections instead of whole trees. Comparability follows: bits_per_byte (bits over scored UTF-8 bytes) joins the per-token bpb in METRICS_JSON and beside every g1.bpb.* key, g7 sizes its probes from the resolved vocab, and eval.common.fit_to_tokens builds contexts to an exact token budget for the long-context rework.
Real long documents scored the only way a pretrained base LM honestly can: LongBench-v2 four-way MCQ by length-normalized logprob over the answer texts, and HELMET RAG through HELMET's own non-chat few-shot template with a bounded greedy decode and substring exact match. No chat template, no judge, no summarization. Operator-side `xtask natural-pack` fetches both sources at a pinned revision, verifies every artifact's SHA-256, and writes a seeded private pool plus a disjoint public_dev mirror so the contamination gap works the way it already does for G2/G4. Datasets stay out of git; the packs live in the operator eval-assets dir, which prism-lium already stages post-train. Packs carry raw text + choices + gold only. All token math -- length measurement and LongBench's middle-truncation -- runs on-pod against the tokenizer the miner submitted, funnelled through one wrapper so there is a single owner of it. Phase C wires the rollup, composite and anchors; this lands the module, the builder, the staging path and the fixtures.
Phase A's modular tokenizer landed the shared helpers this module was standing in for. Swap the local encode/length wrappers for common.tokenizer_of / encode / decode / token_len, and drop the decode-less fallback: the contract validates an encode/decode roundtrip before the battery runs, so a tokenizer without decode is a contract error to surface, not a case to degrade around. `_fit_middle` stays local and is now documented against its neighbours -- truncate_tokens keeps a prefix and would throw away the end of a LongBench document, and fit_to_tokens grows a synthetic context rather than shrinking a real one.
Official generators with private-seed regeneration and base-LM logprob scoring; lengths via the modular tokenizer fit_to_tokens contract.
Stage kernels, helpers, and tokenizer/ under submission/ via tar-over-SSH-stdin so miners are no longer limited to architecture.py + training.py seam projections.
Wire community protocols into the ranked path with weighted anchors, fail-soft orchestration, and L* length capability.
Spell out recipe 1.4.0 G5 keys/weights, L*, and no IFT/chat/judge.
… prism/merge-source-tree
Public BaseIntelligence/prism now carries the miner guide; mirror tokenizer / source-tree / CAP_EXCEEDED tips in troubleshoot.
Brings the branch up to date with 118 commits of v2 hardening while keeping the v3 composite / source-tree structure. 21 conflicts resolved so that both sides' behaviour survives rather than one side winning: - score.rs: branch's `combine_final(outcome, mode)` keeps main's nuanced cheap-similarity gating (`Copied` always zeroes, `Suspicious` only at score >= 0.9 with non-trope evidence) ahead of the scoring-mode lattice. - prompts / types / tools: keep main's non-causal label-leak screen and AST bands alongside the branch's telemetry-shim tolerance and `Other` fallback. - client.rs / ssh.rs: keep the branch's tar-over-stdin + streaming exec path (BUG-5) and re-port main's harness log-tail retention onto it — `truncate_tail` now feeds `exec_with_retries`, and a dropped session harvests the on-pod log instead of losing the fatal tail. - prism_store.rs: schema union of `miner_coldkey` (main) and `tree_blob` (branch), with INSERT placeholders renumbered to match the bind chain. - api.rs: intake resolves the owning coldkey *and* packs the source tree. - submit.rs: accept main's deletion; the orchestrator takes `GatewayClient` from `challenge_common`, which is what `EpochEmitter` expects. - agentic.rs: drop the branch's inline `corpus_from_rows` for main's coldkey-aware version in `prism_pipeline::precheck`. Migrations 0013/0014 collided (one per side); the branch's two are renumbered 0017/0018 so `sqlx::migrate!` cannot silently skip a version. Fixed the `PRISM.md` claim that source trees may be submitted as raw `application/zip` — `sources_from_zip` rejects them and points at JSON `zip_base64`. fmt, clippy, consensus-lint, spec-check, design-check and external-docs-check pass; the workspace test suite is green. loc-cap does not pass: five crates went over the 1500 non-test LOC cap purely by addition (prism-pipeline 1710, prism-challenge 1620, prism-lium 1562, prism-store 1548, challenge-agentic 1514), since both branches had each trimmed to sit just under it. Fixing that means relocating ~450 LOC across crate boundaries and is left to follow-ups.
prism-pipeline sat 210 LOC over the 1500 per-crate cap after merging main. zone_b.rs was already a leaf (std + prism-zoneb re-exports + sha2, no pipeline internals), so it belongs next to the contract types it validates rather than behind a re-export in the pipeline crate. prism-pipeline 1710 -> 1376; prism-zoneb 133 -> 471. Consumers (prism-eval-store, prism-attribution) now import prism_zoneb directly.
prism-challenge sat 123 LOC over the 1500 per-crate cap after merging main.
Three moves, each to the crate that already owns the concern:
- Submission JSON views (list/detail/eval + the sha digest) to
prism-eval-store, whose module doc already claims the API JSON views.
- The v3 read-only eval routes (metrics?zone=a|b, /v1/anchors,
/v1/preregistration) to prism-attribution, following the narrow-state
MethodRouter pattern the attribution and zone-b routes already use.
- The queued row builder to prism-pipeline, next to ephemeral_candidate;
api.rs keeps a thin wrapper that resolves the coldkey from the metagraph.
The intake front-end (body parse, arch materialization, membership, error
envelope) is shared verbatim by real intake and the advisory precheck, so it
moves to a new prism-intake crate with the precheck route; keeping the two
paths on one front-end is what makes the advisory verdict comparable to the
one intake returns. That also collapses three copies of the error envelope
in prism-attribution onto the crate-root helper.
prism-challenge 1623 -> 1317. Behaviour unchanged: same routes, same error
shape ({"error": message, "code": code}), same front-end ordering.prism-lium sat 61 LOC over the 1500 per-crate cap after merging main. Its error taxonomy, provider shapes, pod telemetry series, and signed EvalReceipt were already a pure leaf (std + serde + sha2 + the recipe train-hours cap), so they move to a types crate the way prism-zoneb holds the Zone B contract. prism-lium 1561 -> 1324. prism-lium re-exports every moved name, so prism_lium::… stays the single import path for callers.
prism-store sat 48 LOC over the 1500 per-crate cap after merging main. The submission row, stage lattice, patch, error taxonomy, and registry / epoch / top-model records are a pure leaf, so they move out the way prism-lium-types holds the Lium contract; the PrismStore trait, memory impl, and Postgres impl stay behind. Because the row's only Lium types are EvalReceipt and TelemetryPoint, both now in prism-lium-types, this also drops the heavy prism-lium dependency (reqwest, tokio) from the store crates entirely. prism-store 1548 -> 1409. prism-store re-exports every moved name, so prism_store::… stays the single import path for callers.
Unread local stderr pipes can fill and stall the SSH session mid-run even when the remote merges 2>&1 into stdout — matching a hung live-Lium eval that left an idle pod with a dead harness.
Live Lium e2e harvested a torch.save FP32 checkpoint that exceeded the old BF16×1.5 budget (exact n_params×3), so RECEIPT.json never landed and playground could not load the parked weights. Budget now matches the harness train dtype.
Live Lium harvest still refused under FP32×1.5: tied embed/lm_head state_dict keys can pickle as ~2× measured n_params. Budget now covers tying duplication plus pickle/tar overhead so RECEIPT.json can land.
Decouple tiny_caps stubs from PRISM_TEST_TRAIN_MINUTES so operators can run a complete (non-stub) battery on real Lium with a ~12m train cap. Forward eval sample/budget knobs into the pod harness env.
Operator-side script streams G1/G2/G5 mirrors (wikitext/finemath/codeparrot/fineweb 2025, OLMES G2, pg19+squad) without pulling FineWeb-Edu train pins.
Challenge runs as uid 65532 and defaulted to /var/lib/prism/artifacts without an image-owned directory, so create_dir_all failed with Permission denied after eval (termination RECEIPT only). Pre-create the park in the Dockerfile, probe writability before SSH harvest, and surface the configured path in mkdir errors.
Candidate walks that hit HTTP 429 left PENDING pods billing because cleanup_after_rent(None) was a no-op. Always reclaim by pod_name after failed rent/wait, terminate known ids, and keep the 429 backoff so retries never abandon rented instances.
A real µP sweep that diverged stubbed g8.mup without the org key, so G8 composite completeness dropped mup_lr_stability. Emit stability=0.0 when the sweep path runs but fails, keep 1/(1+|log2 ratio|) on success, and document the semantics for anchors/rollup.
Keep per-item prompts, choices, gold/selected, and choice logprobs
(plus generative text) in METRICS_JSON inference_traces with size caps,
expose them via GET /v1/submissions/{id}/inference, and journal
playground completions next to parked artifacts.Baseline architectures return a ModelOutput carrier; treating the whole object as a tensor broke admin playground complete on harvested ckpts.
Default staged eval_tier=public from public HF held-out assets (full G1 domains+fresh, G2, G5 natural); keep private only for optional secret mirrors. Raise pack cap to 256MiB and add overnight operator recipe.
Live GPU eval bills the miner's Lium account instead of the operator key. Intake requires the header on live; keys stay in-process only.
Keeps prism-lium under the 1500 LOC cap.
Freeze NeMo AutoModel v0.5.0 (content SHA + PRISM_AUTOMODEL_PIN_DIR), reject fixture pins on live unless PRISM_AUTOMODEL_FIXTURE=1, and keep miner-funded Lium BYOK required for real GPU eval.
Important Review skippedToo many files! This PR contains 247 files, which is 147 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (247)
You can disable this status message by setting the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
NeMo AutoModel pin trees exceed the 100-byte name field under submission/; use POSIX prefix so live recipe 2.0 pack/upload fail-closes only past 255 bytes.
Persist only .prism/ + patch delta under the 17MiB DB cap; expand against PRISM_AUTOMODEL_PIN_DIR when staging the Lium harness tar so live trains still see the full applied pin tree.
Live eval fell back to stock train_ft (mlflow) because slim tree_blob dropped miner prism.toml; keep knobs + recover entry on pod expand.
Preserve recipe 2.0 and miner-funded retries while integrating current weight sealing and deployment changes, then split support code to keep all workspace gates green.
5 tasks
Uh oh!
There was an error while loading. Please reload this page.
4 tasks
echobt added a commit
that referenced
this pull request
Aug 12, 2026
PR #120 landed a second 0017_* file that collided with tip-supersede already applied on staging/prod, crashing migrate with checksum mismatch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
automodel.base+automodel.patchsubmissions against the frozen NeMo AutoModel pin (automodel@v0.5.0, commitd02f49cb…, content SHAf8af64ef…), fail-closed pin application, persisted slim deltas, and a submission diff API.X-Lium-Api-Key; BYOK credentials remain in the in-process payer vault and are never persisted or logged.PRISM_AUTOMODEL_PIN_DIR, private eval assets are delivered after training, and the bounded real-Lium smoke remains an explicit operator validation.main, preserving its weight-sealing, deployment-pin, and retry changes while keeping AutoModel 2.0 and miner-paid Lium behavior intact.Validation
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspacecargo deny checkcargo run -p xtask -- loc-capcargo run -p xtask -- consensus-lintcargo run -p xtask -- spec-checkcargo run -p xtask -- design-checkcargo run -p xtask -- external-docs-checkcargo clippy -p validator-bin --features dcap --all-targets -- -D warningsbash deploy/scripts/assert-compose-matrix.shOperator follow-ups
deploy/scripts/stage-automodel-pin.shand mountPRISM_AUTOMODEL_PIN_DIRonprism-challenge.X-Lium-Api-Key; confirm missing keys fail closed and the miner account is billed.BaseIntelligence/prismrepository (not a CI blocker for this PR).