Skip to content

Rename submission modes: benchmark→private, leaderboard→public - #413

Closed
msaroufim wants to merge 6 commits into
mainfrom
rename-benchmark-leaderboard-to-private-public
Closed

Rename submission modes: benchmark→private, leaderboard→public#413
msaroufim wants to merge 6 commits into
mainfrom
rename-benchmark-leaderboard-to-private-public

Conversation

@msaroufim

@msaroufimmsaroufim commented Feb 2, 2026•
edited
Loading

Copy link
Copy Markdown
Member

This renames the user-facing submission modes for clarity:

  • BENCHMARK → PRIVATE (run benchmarks without affecting leaderboard ranking)
  • LEADERBOARD → PUBLIC (official submission to the public leaderboard)

Also adds SECRET mode for internal secret validation runs.

Updates Discord commands: /benchmark → /private, /ranked → /public

Review with gpu-mode/popcorn-cli#33 and gpu-mode/reference-kernels#100

Mark Saroufim
Rename submission modes: benchmark→private, leaderboard→public
118b06a
This renames the user-facing submission modes for clarity:
- BENCHMARK → PRIVATE (run benchmarks without affecting leaderboard ranking)
- LEADERBOARD → PUBLIC (official submission to the public leaderboard)
Also adds SECRET mode for internal secret validation runs.
Updates Discord commands: /benchmark → /private, /ranked → /public
CopilotAI review requested due to automatic review settings February 2, 2026 01:59
CopilotAI reviewed Feb 2, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR renames user-facing submission modes from BENCHMARK/LEADERBOARD to PRIVATE/PUBLIC, introduces a new SECRET mode for internal validation, and propagates these changes through the backend, launch pipeline, reports, Discord commands, and tests.

Changes:

  • Update SubmissionMode enum and evaluation pipeline to use TEST, PRIVATE, PROFILE, PUBLIC, and SECRET, and adjust timeouts and run orchestration accordingly.
  • Adapt reporting, backend scoring/DB persistence, GitHub/Modal launchers, API validation, and Discord cogs to the new mode names and semantics (PRIVATE = non‑ranked benchmarks, PUBLIC/SECRET = ranked).
  • Update tests to use the new mode names and updated report strings, plus new run keys like "private" and "public" instead of "benchmark" and "leaderboard".

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file
FileDescription
tests/test_task.pySwaps SubmissionMode.BENCHMARK with SubmissionMode.PRIVATE in task-config tests to reflect new non-ranked benchmark mode.
tests/test_modal.pyUpdates Modal launcher tests to use SubmissionMode.PRIVATE/PUBLIC and to expect "private" run keys and “ranked submission” terminology.
tests/test_github.pyAdjusts GitHub launcher tests to use SubmissionMode.PUBLIC and to assert against "private" run keys in results.
tests/test_backend.pySwitches backend tests to PUBLIC/SECRET modes and new short-report wording; still expects "leaderboard" in configs/DB runs, which no longer matches the implementation.
src/libkernelbot/submission.pyExtends compute_score to accept a mode_key so scores can be computed from "public" or "secret" runs instead of hard-coded "leaderboard".
src/libkernelbot/run_eval.pyReworks evaluation modes so PRIVATE replaces bare benchmark and PUBLIC/SECRET replace leaderboard, updating how tests/benchmarks/ranked runs and timeouts are orchestrated.
src/libkernelbot/report.pyChanges reports to treat "private" as the benchmark run key and "public"/\"secret\" as ranked runs, and updates all user-facing messages from “Leaderboard” to “Ranked submission”.
src/libkernelbot/launchers/github.pyMaps GitHub timeouts to the new enum values, using benchmark timeout for PRIVATE and ranked timeout for PUBLIC.
src/libkernelbot/consts.pyRedefines SubmissionMode to TEST, PRIVATE, PROFILE, PUBLIC, SECRET and updates the docstring to document the new semantics.
src/libkernelbot/backend.pyUpdates submit_full, submit_leaderboard, and handle_submission to drive PUBLIC and SECRET runs correctly, compute scores from the appropriate run key, and adjust short reports and secrecy behavior; currently also (incorrectly) ranks PRIVATE runs.
src/kernelbot/cogs/verify_run_cog.pyAdjusts verification slash commands to use PRIVATE/PUBLIC and default to PUBLIC when no mode is specified.
src/kernelbot/cogs/leaderboard_cog.pyRenames Discord commands from /benchmark→/private and /ranked→/public and wires them to the new submission modes.
src/kernelbot/api/api_utils.pyRestricts API-allowed submission modes to TEST, PRIVATE, PROFILE, and PUBLIC, matching the new public interface.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadsrc/libkernelbot/backend.py Outdated
Comment threadtests/test_backend.py Outdated
Mark Saroufim added 4 commits February 1, 2026 18:12
Mark Saroufim
Fix test files to use new private/public mode naming
ee62fdc
Update test data keys and expected values:
- test_report.py: Change "benchmark"/"leaderboard" keys to "private"/"public"
- test_submission.py: Update compute_score test to use "public" key
- test_backend.py: Update mode values and mock data keys
Mark Saroufim
Fix test_submit_full mock and expected mode
6fcc19f
- Add 'secret' key to mock launcher runs so SECRET mode can find its result
- Fix second run's expected mode from 'public' to 'secret'
Mark Saroufim
Fix GitHub integration test to use PR branch
2e022ca
Set GITHUB_BRANCH env var to use the PR's source branch instead of
falling back to main. Uses github.head_ref for PRs, github.ref_name
for direct pushes.
Mark Saroufim
Fix test_submit_full mock to return mode-specific results
d47bad7
Use side_effect to return different FullResult for each call:
- First call (PUBLIC mode) returns {"public": eval_result}
- Second call (SECRET mode) returns {"secret": eval_result}
This prevents the backend from storing all keys from both calls.
@github-actions

github-actionsBot commented Feb 2, 2026•
edited
Loading

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)		Lines missing
src/libkernelbot
backend.py
consts.py
report.py347
submission.py
utils.py
Project Total

This report was generated by python-coverage-comment-action

@msaroufimmsaroufim mentioned this pull request Feb 2, 2026
Closed
@msaroufim
msaroufim requested review from S1ro1 and ngc92February 4, 2026 22:13
Mark Saroufim
Fix PRIVATE mode submissions incorrectly receiving scores
a002f08
PRIVATE mode runs return timing info but should not affect leaderboard
ranking. This change restricts score computation to only PUBLIC and
SECRET modes, ensuring PRIVATE submissions have score=None as intended.
@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude are you working

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude wake up

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@codex wake up

CopilotAI reviewed Feb 8, 2026
View reviewed changes

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment threadsrc/libkernelbot/launchers/github.py
SubmissionMode.BENCHMARK.value: config.get("benchmark_timeout"),
SubmissionMode.LEADERBOARD.value: config.get("ranked_timeout"),
SubmissionMode.PRIVATE.value: config.get("benchmark_timeout"),
SubmissionMode.PUBLIC.value: config.get("ranked_timeout"),

CopilotAIFeb 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_timeout() maps test/private/public but not the new secret mode. When config['mode'] == 'secret', this falls back to DEFAULT_GITHUB_TIMEOUT_MINUTES, which can cause secret validation runs to time out (or use an unintended timeout). Add an explicit mapping for SubmissionMode.SECRET (likely to ranked_timeout, same as PUBLIC).

Suggested change
SubmissionMode.PUBLIC.value: config.get("ranked_timeout"),
SubmissionMode.PUBLIC.value: config.get("ranked_timeout"),
SubmissionMode.SECRET.value: config.get("ranked_timeout"),

Copilot uses AI. Check for mistakes.
Comment threadsrc/libkernelbot/consts.py
Comment on lines +89 to +92
Public: Official submission to the leaderboard. This first runs public tests, then a
repeated invocation of a single benchmark. If all tests pass, the submission is evaluated
and ranked on the public leaderboard.
Secret: Internal mode for running the full evaluation flow with a secret seed. This is used

CopilotAIFeb 8, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SubmissionMode docstring says PUBLIC runs are a “repeated invocation of a single benchmark”, but the evaluation code selects either the last benchmark or all benchmarks depending on ranking_by (e.g., MEAN/GEOM use multiple benchmarks). Update the docstring to reflect that PUBLIC/SECRET may run multiple benchmarks depending on the ranking criterion, to avoid misleading users/operators.

Suggested change
Public: Officialsubmissiontotheleaderboard. Thisfirstrunspublictests, thena
repeatedinvocationofasinglebenchmark. Ifalltestspass, thesubmissionisevaluated
andrankedonthepublicleaderboard.
Secret: Internalmodeforrunningthefullevaluationflowwithasecretseed. Thisisused
Public: Officialsubmissiontotheleaderboard. Thisfirstrunspublictests, thenrunsthe
benchmarksuiteaccordingtotheselectedrankingcriterion (forexample, usingonlythe
lastbenchmarkoraggregatingoverallbenchmarks). Ifalltestspass, thesubmissionis
evaluatedandrankedonthepublicleaderboard.
Secret: Internalmodeforrunningthefullevaluationflowwithasecretseed. Thismirrors
thepublicevaluation (includingthebenchmarkselection/aggregationstrategy) andisused

Copilot uses AI. Check for mistakes.
@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude any feebdback

@claude

claudeBot commented Feb 8, 2026•
edited
Loading

Copy link
Copy Markdown

Claude encountered an error —— View job

I'll analyze this and get back to you.

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude can you please review habibi

@claude

claudeBot commented Feb 8, 2026•
edited
Loading

Copy link
Copy Markdown

Claude encountered an error —— View job

I'll analyze this and get back to you.

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude one more time

@claude

claudeBot commented Feb 8, 2026•
edited
Loading

Copy link
Copy Markdown

Claude encountered an error —— View job

I'll analyze this and get back to you.

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude you ready???

@claude

claudeBot commented Feb 8, 2026•
edited
Loading

Copy link
Copy Markdown

Claude encountered an error —— View job

I'll analyze this and get back to you.

@msaroufim

msaroufim commented Feb 8, 2026

Copy link
Copy Markdown
MemberAuthor

@claude hello

@claude

claudeBot commented Feb 8, 2026•
edited
Loading

Copy link
Copy Markdown

Claude encountered an error —— View job

I'll analyze this and get back to you.

@msaroufim

msaroufim commented Mar 13, 2026

Copy link
Copy Markdown
MemberAuthor

meh

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@ngc92ngc92Awaiting requested review from ngc92
@S1ro1S1ro1Awaiting requested review from S1ro1

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@msaroufim@S1ro1