fix(raw-weights): rebuild expired pending and surface rejection reasons

[echobt]

Summary

• Real challenge raw-weight push path was stuck replaying expired pending ledger rows, so master returned 422 freshness forever for both prism and agent-challenge.
• Clients now drop non-retryable pending, rebuild fresh payloads, reject all-zero weight maps, and surface attributable rejection details.
• Ingress logs specific rejection reasons and returns distinct detail strings so operators are not blind.

Why

On-chain burn is currently the only thing keeping validator weights fresh. That masks a broken aggregation pipeline: challenge raw-weight pushes never land durable snapshots. Operators need visible rejection reasons, not a generic 422 loop.

Root cause (prod evidence)

• Prism ledger pending epoch 4959201, expires_at 2026-07-29T01:49:28Z, never acknowledged.
• Agent-challenge ledger pending epoch 4959347, expires_at 2026-07-29T22:52:48Z, never acknowledged.
• POST of that pending body returns 422 {"detail":"snapshot outside freshness window"}.
• Hotpatch succeeds with 200 because it builds fresh payloads each cycle and does not reuse the challenge pending ledger.
• Real path (prism_challenge.raw_weight_push loop) is the broken side; schema itself accepts valid fresh payloads.

Changes

• Client (prism + agent-challenge): expire/clear pending on freshness and other non-retryable 4xx; include response detail in result error and logs (no tokens/payloads).
• Client: skip all-zero weight maps locally (validator placeholder smell).
• Ingress: reject all-zero weight maps; map pydantic failures to field-level details; log rejection reason server-side.
• Tests for attributable details, zero maps, expired pending rebuild, and detail surfacing.

Test plan

• RED then GREEN on new tests (8 targeted)
• pytest tests/unit/test_raw_weight_ingress.py packages/challenges/prism/tests/test_raw_weight_push.py packages/challenges/agent-challenge/tests/test_ac_raw_weight_push.py → 39 passed
• Full agent-challenge suite A/B: main 24 failed / 2307 passed; branch 24 failed / 2309 passed (zero new failures)
• ruff check/format on touched paths

CI

• Waiting for required checks to go green on this head SHA.

Notes

• Does not touch burn-weights container, submit_on_chain_enabled, emission_shares, or ac-keyrelease-ratls.
• Hotpatch script is not the fix; real path should replace it after deploy + pending ledger clear/restart.

Summary by CodeRabbit

• Bug Fixes

  • Expired pending weight submissions are now rebuilt instead of retrying stale data.
  • Non-retryable rejections clear pending submissions to prevent repeated failures.
  • Rejection responses now provide clearer error details (without exposing full response bodies).

• Validation / New Behavior

  • Receiving now rejects all-zero weight maps.
  • Explicitly submitting only zero weights is skipped locally without making a request.

• Reliability

  • Raw-weight delivery state reuse is handled more safely across retries and restarts.
  • Rejection events now include structured diagnostic logging.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 7efd706c-4a04-4c0c-8099-7571f642343b

📥 Commits

Reviewing files that changed from the base of the PR and between 46c5bed and 1754651.

📒 Files selected for processing (1)

• tests/unit/test_raw_weight_ingress.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/unit/test_raw_weight_ingress.py

---

📝 Walkthrough

Walkthrough

Raw-weight push clients now rebuild expired pending snapshots, skip all-zero weight maps, and clear pending state for non-retryable rejections. The master ingress returns attributable rejection details, logs structured metadata, and rejects zero-only payloads.

Changes

Raw-weight push safety

Layer / File(s)	Summary
Master rejection details and logging src/base/master/raw_weight_ingress.py, tests/unit/test_raw_weight_ingress.py	Ingress validation returns specific rejection details, logs rejection metadata, and rejects all-zero weight maps with HTTP 422.
Durable client state and validation packages/challenges/agent-challenge/src/.../raw_weight_push.py, packages/challenges/prism/src/.../raw_weight_push.py	Clients clear expired or non-retryable pending snapshots, rebuild stale payloads, skip zero-only weights, and include safe response details in rejection errors.
Push behavior coverage packages/challenges/agent-challenge/tests/test_ac_raw_weight_push.py, packages/challenges/prism/tests/test_raw_weight_push.py	Tests cover expired snapshot rebuilding, local zero-weight skips, pending cleanup, and rejection detail propagation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
participant RawWeightPushClient
participant RawWeightPushStore
participant raw_weight_ingress
RawWeightPushClient->>RawWeightPushStore: read pending snapshot
RawWeightPushClient->>RawWeightPushClient: check pending freshness
RawWeightPushClient->>RawWeightPushStore: clear expired pending snapshot
RawWeightPushClient->>raw_weight_ingress: submit rebuilt payload
raw_weight_ingress-->>RawWeightPushClient: return status and rejection detail
RawWeightPushClient->>RawWeightPushStore: clear pending for non-retryable rejection

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main fixes: rebuilding expired pending raw-weight pushes and surfacing rejection reasons.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/raw-weight-pending-freshness-visibility

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}