Fix #180 #181: unjudgeable checkers for problems 11, 22 and 210 - #184
Merged
Conversation
- 11/chk.cc: the verdict was sprintf'd into a char[30] buffer although the formatted message is 41 characters, so every correct answer aborted with "buffer overflow detected" and only the three "-1" cases could score. Enlarge the buffer. Fixes FrontierCS#180. - 210/chk.cc: the per-base g/c/d/v read bounds were tighter than the shipped testdata (v up to 250 against readLong(0, 200)), so cases 4-9 FAILed on the input file before any contestant output was read. Loosen the bounds and document the observed ranges in the statement. Fixes FrontierCS#181. Re-judging the judge-recorded outputs of 32 archived agent runs (16 per problem) with the original checkers reproduces the official per-case scores exactly; with the fixed checkers, 52 previously crashing cases on 11 and 96 previously FAILing cases on 210 are judged, and no case that was judged before changes its score. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUL12r3HsmHBUTDGZxXFQr
algorithmic/problems/22/checker.cpp formats its verdict into char mes[30] with the same 41-character sprintf as problem 11, so every valid answer aborted with "buffer overflow detected". Enlarge the buffer. Reported by a reviewer on FrontierCS#184. A repository-wide search of every checker and interactor finds no other instance of this pattern. Re-judging the 40 judge-recorded outputs of 4 archived agent runs: the original checker aborts on all 40, the fixed one returns a verdict on all 40, and the build is clean under ASan/UBSan. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AUL12r3HsmHBUTDGZxXFQr
Contributor
|
LGTM! Thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three algorithmic checkers score every submission 0 (or cap it) for reasons unrelated to the
submission. Fixes #180 (problem 11, and the identical bug in problem 22 found in review) and
#181 (problem 210); details and reproduction are in the issues. Problem 145 (#182) is tracked
separately because its fix is a different kind of change.
11/chk.cc: enlarge the 30-byte message buffer (the formatted verdict is 41 characters, soevery correct answer aborted in the fortified
sprintf).22/checker.cpp: the same buffer and the samesprintf, same one-line fix. A search ofevery checker and interactor under
algorithmic/problemsfinds no other instance.210/chk.cc: loosen the per-baseg/c/d/vread bounds; the shipped4.in-9.incarry
vup to 250 and the checkerFAILed on its own input for 6/10 cases. The observedranges are added to the statement's "Constraints (from released datasets)".
Type of Change
Testing
Compiled with the judge's flags (
g++ -std=gnu++17 -O2 -pipe -I algorithmic/judge/include).Regression corpus: the exact per-case outputs the judge recorded for 32 archived agent runs
(16 per problem, two harness arms × 8), re-judged with the original and the patched checker.
Feeding the recorded outputs to the original checker reproduces the official per-case ratio
on every checker-judged case (11: 150/150, 210: 160/160), so the comparison is apples to apples.
buffer overflow detectednow score (all 1.000 — no archivedpath exceeded the length bound); the 45
-1cases and the 53 wrong-answer cases areunchanged. Run rewards move from a uniform 0.300 to 0.3–1.0 (10 of 15 runs change). The ten
shipped
.ansfiles also score 1.0 /ok, a non-palindrome scores 0, a path 30× over thebound scores 0.
.ansfiles are empty (the checker does not read them), so the corpus isthe 40 judge-recorded outputs of 4 archived agent runs from another campaign, all officially
*** buffer overflow detected ***: the original checker aborts on 40/40, the fixed onereturns a verdict on 40/40 (all 1.0; their K is N or ≈ 3N), clean under ASan/UBSan. Unrelated
to the crash, the score formula on line 120 of the same file has an operator-precedence issue
(
/ 2 * N) that removes the partial-credit gradient for 3N < K < 5N; it does not affect theseoutputs and, since it changes scoring, is left out of this PR.
FAILed on input (cases 4–9 × 16 runs) are now judged; the 64 othercases are unchanged. Cases 7 and 9 stay 0 for every run because those programs printed
nothing there. Run rewards move from 0.000–0.187 to 0.000–0.351.
All three patched checkers run clean under AddressSanitizer + UBSan over the whole corpus plus the
shipped
.ansfiles. For 11 the scoring formula was exercised on a synthetic 1×3 grid: lengthbelow / at / between 1× and 2× / above 2× the bound gives 1.0 / 1.0 / 0.870 / 0.0,
-1on asolvable instance gives 0, and an empty output on a 1×1 grid gives 1.0 as the statement
requires.
No testdata or
.anschanges. Thevalidate-algorithmicCI job will report "Missing referencesolution" as it did for #174; these problems ship no
reference.cpp.Checklist
🤖 Generated with Claude Code
https://claude.ai/code/session_01AUL12r3HsmHBUTDGZxXFQr