Publish provisional Phase 1 light-tier benchmark results - #68
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0786201d95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| attempts_index = index_attempts(runs_root, label) | ||
| source_label = run.get('source_label') or label | ||
| cells = index_cells(runs_root, source_label) | ||
| attempts_index = index_attempts(runs_root, source_label) |
There was a problem hiding this comment.
When the shared base50-light run resumes and a previously unfinished interim cell later writes an outcome.json, this indexes that new outcome into base50-light-interim-378; build_row then counts it as a no-patch result even though it is absent from the frozen evaluator report, changing the published snapshot's denominator and score. Retrying or replacing an existing source cell can similarly remove its telemetry, so the interim snapshot needs immutable source artifacts or filtering to the frozen snapshot's membership.
Useful? React with 👍 / 👎.
| 'benchmarks/results/code/predictions/%s/%s.jsonl --label %s' | ||
| % (run_label, cell, run_label)) | ||
| % (run_label, cell, evaluator_label)) |
There was a problem hiding this comment.
For a run where source_label != run_label, such as the new interim snapshot, the reproduction command generates under the source label but tries to evaluate under the display label. Specifically, benchmarks/code/scripts/run-canary.sh:82 writes predictions beneath $RUN_ID, so --run-id base50-light creates predictions/base50-light/D.jsonl, while this line requests predictions/base50-light-interim-378/D.jsonl and fails before evaluation. Use source_label for the prediction path while retaining evaluator_label for --label.
Useful? React with 👍 / 👎.
Summary
Publishes the official-evaluator results available from the frozen
base50-light-interim-378snapshot while keeping them visibly separate from the preregistered finalbase50-lightrun. The observatory’s current export now links a displayed evaluator snapshot to preserved source-run telemetry and restricts overlapping snapshots to their declared conditions; archived leaderboard and proof-of-concept files remain byte-identical.The provisional B-vs-D contrast uses 49 shared tasks: both resolved 38, B-only 4, D-only 2, neither 5, one excluded, exact two-sided McNemar p = 0.6875. This is not the preregistered final 50-pair contrast and makes no Phase 2 decision.
Provenance and presentation
base50-lightand the nine new arms underbase50-light-interim-378, with no duplicate rows.current-results.*, then rebuilds the standalone observatory index.leaderboard.*andpoc.*editions unchanged.Verification
benchmarks/site/tests/test-site-build.sh— 30/30 assertions passed locally.node --test benchmarks/site/tests/test-observatory-data.cjs— 6/6 passed locally.bash benchmarks/code/code-bench.sh check— passed locally.current-results.json.build_tree_dirty: falseat source commit212de43.