Skip to content

fix: problem 22 checker scoring - #187

Open
weiwch wants to merge 1 commit into
FrontierCS:mainfrom
weiwch:main
Open

fix: problem 22 checker scoring#187
weiwch wants to merge 1 commit into
FrontierCS:mainfrom
weiwch:main

Conversation

@weiwch

@weiwch weiwch commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Correct the score denominator to divide by 2 * N, and enlarge the verdict message buffer to prevent valid submissions from crashing the checker.

Summary

  • Fix operator precedence in Problem 22's bounded and unbounded score calculations.
  • Increase the verdict message buffer from 30 to 128 bytes.
  • Prevent valid submissions from being reported as checker failures.

Please read CONTRIBUTING.md before submitting.

Type of Change

  • New research problem
  • New algorithmic problem
  • New Frontier-CS 2.0 problem
  • Bug fix
  • Documentation update
  • Other:

Testing

Checklist

  • Code follows the project structure and conventions
  • Self-review completed
  • Documentation updated (if applicable)

Root cause

The checker calculated the score as:

(5 * N - K) / 2 * N

This divides by 2 and then multiplies by N, producing extremely large unbounded scores. Formatting those values overflowed the 30-byte mes buffer and caused the checker to terminate with:

*** buffer overflow detected *** (#184)

The intended formula is:

(5 * N - K) / (2 * N)

Even with the corrected formula, the full verdict message can exceed 30 bytes, so the buffer is also enlarged.

Validation

  • Replayed a historical submission that previously failed all 10 test cases due to checker crashes.
  • Confirmed that it passes all 10 cases with the fixed checker.

Correct the score denominator to divide by 2 * N, and enlarge the verdict message buffer to prevent valid submissions from crashing the checker.
@whuang369

Copy link
Copy Markdown
Contributor

Replayed this against the archived agent runs in the judge image (Ubuntu 22.04, g++ 11.4, glibc 2.35). Both changes are correct and the Codex run goes from 0.0 to 1.0 on all 10 cases as described. LGTM.

One correction to the record, since the root cause in the description mixes up the two fixes. For future readers, the accurate version is:

Root cause. Two independent bugs.

  1. Crash. The verdict is built with sprintf(mes, "Ratio: %lf, RatioUnbounded: %lf", ...) into char mes[30]. The fixed text is 33 characters and each %lf prints at least 8, so even the shortest possible message (Ratio: 1.000000, RatioUnbounded: 1.000000) is 41 characters plus the terminator. With _FORTIFY_SOURCE (on by default at -O2) glibc aborts with *** buffer overflow detected *** on every full-score answer, regardless of the values printed. Enlarging the buffer to 128 bytes fixes this on its own; fixing the formula alone does not (verified: formula-only variant still aborts 10/10).

  2. Scoring. 1.0 * (5 * N - K) / 2 * N divides by 2 and then multiplies by N, so the ratio is huge for any K < 5N and every valid answer scores 1.0, with a meaningless unbounded ratio (about 1.5e10 for the Codex run). With / (2 * N) the score is 1.0 for K ≤ 3N, 0.75 at 3.5N, 0.5 at 4N, and 0 at 5N, which is the partial-credit gradient the formula was written to express. This is a scoring change for submissions with 3N < K < 5N. It does not affect any archived result: the Codex run has K ≈ 2N (1.0 either way) and the Gemini CLI run is a genuine wrong answer (new tree not connected) under every checker variant.

Note that #184 carries the identical mes[128] hunk for this file and the two PRs conflict on overlapping hunks, so whichever lands second needs a rebase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants