Uh oh!
There was an error while loading. Please reload this page.
Add dev skill agent evaluation framework with bug fixes and refactor - #1166
Closed
ramakrishnap-nv wants to merge 3 commits into
Closed
Add dev skill agent evaluation framework with bug fixes and refactor#1166ramakrishnap-nv wants to merge 3 commits into
ramakrishnap-nv wants to merge 3 commits into
Conversation
Run developer-skill prompts through the Claude CLI with the cuOpt developer skill as context, then validate that responses follow the skill (required phrases present, forbidden phrases absent). Includes: - ci/utils/run_dev_skill_agent_tests.py: live + replay runner with pass@K, runtimes/cost JSON, CSV + Markdown report - ci/utils/dev_skill_agent_tests.json: 53 prompt/assertion tests - ci/utils/dev_skill_agent_tests_issue_style.json: SWE-bench-style set - ci/utils/validate_developer_skills.sh: static validator for cuopt-developer and cuopt-installation-developer SKILL.md content - ci/run_dev_skill_agent_tests.sh: wrapper invoked from repo root Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
- pass@K continue on attempt error: previously the runner cleared all collected attempts and aborted on the first subprocess failure, defeating pass@K's "pass if any attempt passes" semantics. Now records the error reason and continues to the next attempt. - --save writes the passing attempt instead of always responses[0], so saved replays match the sample that demonstrated correctness. - Merge default_assertions with per-test assertions instead of letting per-test entries shadow them. The issue-style suite's defaults were effectively dead code under the previous behavior. - Fix _NEGATION_PATTERN: alternatives ending with \s (no\s, not\s) plus the closing \b never matched because \s consumed the boundary. Use \bno\b / \bnot\b instead — "should not run sudo" now correctly exempts "run sudo". - Tighten _NEGATION_LOOKBACK from 100 to 40 chars and reset on sentence-end punctuation (.!?), so a negation in the previous sentence no longer exempts a forbidden phrase in the next. Newlines are excluded from the boundary so multi-line code-comment markers (// WRONG\\nnew int[100]) still exempt the next-line forbidden phrase. - Surface CLI errors from partial pass@K attempts in the report's failure_reasons. - Narrow over-broad forbidden phrases (bare "yes" tokens replaced with specific affirmative substrings) so legitimate skill-aligned responses do not trip on the word "yes". - validate_developer_skills.sh: heading-aware section check (matches Markdown # heading lines instead of inline prose). - validate_developer_skills.sh: missing SKILL.md is now a hard error, not a silent SKIP. - Both shell scripts: set -euo pipefail. Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
Refactor: - Extract median_meta() to module level (was a closure redefined per test) and make_report_row() so the per-test row shape is built in one place instead of inline at four call sites. - Add _validate_config_schema() invoked from load_config() so typos in a tests JSON (missing id/prompt, wrong type, duplicate id) surface before any Claude CLI call instead of mid-run. New options: - --filter REGEX: run only tests whose label matches the regex (case-insensitive). Useful for iterating on a subset. - --list: print test labels for the configured suites and exit; no Claude CLI call required. - --junit PATH: write a JUnit-style XML report so CI dashboards can ingest pass/fail/skip alongside median runtime per testcase. PATH may be a file (anything ending .xml) or directory; a directory uses the timestamped run dir and writes junit.xml inside it. The wrapper script and runner docstring document the new flags. Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
ramakrishnap-nv
commented
Apr 30, 2026
CollaboratorAuthor
closing for #1167 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Branch from latest
maincontaining the dev-skill agent evaluation framework from #953, plus bug fixes and a small refactor. Can supersede #953.Three commits:
1. Add dev skill agent evaluation framework — same content as #953.
2. Fix bugs:
pass@Kaborted on the first attempt error and cleared all collected attempts; now records the error reason and continues to the next attempt.--savealways wroteresponses[0]; now writes the passing attempt so saved replays match the sample that demonstrated correctness.default_assertionswas shadowed by per-test entries (so the issue-style suite's defaults were dead code); now merged.no\s/not\s+ closing\bnever matched because\sconsumed the boundary. Use\bno\b/\bnot\bso "should not run sudo" exempts "run sudo"._NEGATION_LOOKBACKfrom 100 to 40 chars and reset on sentence-end punctuation (.!?); newlines intentionally excluded so multi-line code-comment markers (// WRONG\nnew int[100]) still exempt the next line."yes"tokens replaced with specific affirmative substrings) so legitimate skill-aligned responses don't trip on the word "yes".validate_developer_skills.sh: heading-aware section match (matches Markdown#heading lines instead of inline prose); missing SKILL.md is now a hard error, not a silent SKIP.set -euo pipefailon both shell scripts.3. Refactor + new options:
median_meta()andmake_report_row()so per-test rows are built in one place instead of inline at four call sites._validate_config_schema()so typos in a tests JSON surface before any Claude CLI call.--filter REGEX— run only tests whose label matches (case-insensitive).--list— print test labels and exit; no Claude CLI required.--junit PATH— emit a JUnit XML report for CI ingestion.Test plan
python3 ci/utils/run_dev_skill_agent_tests.py --listprints all labels--list --filter cudanarrows to the CUDA-related testsbash ci/utils/validate_developer_skills.shpasses against current SKILL.md files--filter run_tests) producesreport.md,results.csv, andjunit.xml// WRONG-marker code blocks,should notrecognition