Skip to content

fix(headless): upstream published-run failure semantics from experiment branch - #1758

Merged
Astro-Han merged 5 commits into
mainfrom
fix/headless-published-run-semantics
Aug 1, 2026
Merged

fix(headless): upstream published-run failure semantics from experiment branch#1758
Astro-Han merged 5 commits into
mainfrom
fix/headless-published-run-semantics

Conversation

@Astro-Han

Copy link
Copy Markdown
Contributor

Summary

Upstreams the remaining failure-handling semantics that produced the published DeepSeek Flash TB2.1 report (#1719) from the experiment branch. Audit of all 11 branch commits against main:

branch commitstatus on main
acd979a64 runtime profilealready merged (#1730)
23b6bef7c DeepSeek OpenCode proxy routingalready merged (#1753)
17746a319 command-output isolationalready merged (#1210)
699a81554 agent-deadline propagationalready merged (#1717)
84e0c6ffb pre-execution failure exclusionsuperseded by #1717's richer version
b31d6e469 execution-opportunity classificationalready merged (#1717)
99f1ee37e identity-attested prompt hashesthis PR
d783aa3a4 incomplete terminal provider streamsthis PR
b2d78c28b one adjudicated infra retrythis PR
5c2bbc6db budget-aware pass ratesthis PR
e88bcc2f0 resume infra retries from frozen manifestthis PR

The five cherry-picks (clean, only one auto-merge in a test file) complete the semantics the published numbers were computed under:

  • identity-attested prompt hashes — execution-identity check accepts the hash attested by the cell's identity record
  • incomplete terminal provider streams — a trial whose last provider request did not complete classifies as infra, not a graded failure
  • one adjudicated infra retry — explicit per-task one-shot retry of terminal infra failures, gated by the durable attempt WAL
  • budget-aware pass rates — report separates budget-exhausted arms from scored pass rates instead of diluting them
  • resume retries from frozen manifest — adjudicated retries resume against the frozen manifest fingerprint, never a re-resolved one

Without these, rerunning the published composition on main would score the same raw cells differently from the published report.

Verification

  • packages/headless suite: 1456 pass / 0 fail (each commit carries its own tests from the original TDD)
  • npm run format / npm run lint clean
  • Every other branch commit verified already on main or superseded (table above, checked by symbol against current source)

Review focus

Cherry-pick fidelity: the five commits are byte-identical to the branch versions except the auto-merged context in harness-ab-cli.test.ts. The superseded six were deliberately not picked — #1717's versions are strictly richer.

@Astro-Han
Astro-Han merged commit ac927ae into mainAug 1, 2026
2 of 3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@Astro-Han