Skip to content

evals: add 3 formulation evals for cuopt-numerical-optimization-formulation skill - #1398

Merged
ramakrishnap-nv merged 6 commits into
mainfrom
trigger-numerical-opt-formulation-eval
Jun 22, 2026
Merged

evals: add 3 formulation evals for cuopt-numerical-optimization-formulation skill#1398
ramakrishnap-nv merged 6 commits into
mainfrom
trigger-numerical-opt-formulation-eval

Conversation

@ramakrishnap-nv

@ramakrishnap-nvramakrishnap-nv commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

Adds 2 more evals to skills/cuopt-numerical-optimization-formulation/evals/evals.json (was 1, now 3), all drawn from skill-specific formulation patterns in SKILL.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ramakrishnap-nv
ramakrishnap-nv requested a review from a team as a code ownerJune 5, 2026 15:37
@ramakrishnap-nvramakrishnap-nv self-assigned this Jun 5, 2026
@ramakrishnap-nvramakrishnap-nv added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Jun 5, 2026
@coderabbitai

coderabbitaiBot commented Jun 5, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: cb462a8d-e3db-4ccd-bb05-eff525c9b6bf

📥 Commits

Reviewing files that changed from the base of the PR and between dd41260 and dc78a11.

📒 Files selected for processing (3)
  • skills/cuopt-numerical-optimization-formulation/BENCHMARK.md
  • skills/cuopt-numerical-optimization-formulation/skill-card.md
  • skills/cuopt-numerical-optimization-formulation/skill.oms.sig
✅ Files skipped from review due to trivial changes (2)
  • skills/cuopt-numerical-optimization-formulation/BENCHMARK.md
  • skills/cuopt-numerical-optimization-formulation/skill-card.md
🚧 Files skipped from review as they are similar to previous changes (1)
  • skills/cuopt-numerical-optimization-formulation/skill.oms.sig

📝 Walkthrough

Walkthrough

Adjusts SKILL.md frontmatter spacing and expands evals.json with two new evaluation cases (cutting-stock objective and goal programming with integer decisions). Updates BENCHMARK.md and skill-card.md with new evaluation run metadata, results tables with revised per-dimension scores, validation summaries, and refreshes the skill-card description and references. Updates signature bundle DSSE payload and signature.

Changes

Documentation Formatting and Metadata

Layer / File(s)Summary
Frontmatter spacing adjustment
skills/cuopt-numerical-optimization-formulation/SKILL.md
Line spacing between the YAML frontmatter closing delimiter and the # Numerical Optimization Formulation heading is adjusted.
Benchmark and skill-card evaluation results update
skills/cuopt-numerical-optimization-formulation/BENCHMARK.md, skills/cuopt-numerical-optimization-formulation/skill-card.md
Updates BENCHMARK.md evaluation metadata (date, environment, dataset size, attempt counts) and Tier 1/2 validation summaries. Updates skill-card description to focus on developers formulating LP/MILP/QP optimization problems, adjusts reference capitalization, revises evaluation tasks statement, and replaces evaluation results table with new per-dimension scores (Security, Correctness, Discoverability, Effectiveness, Efficiency) and revised agent performance percentages for both claude-code and codex.

Evaluation Cases Expansion

Layer / File(s)Summary
Add and refine evaluation cases in evals.json
skills/cuopt-numerical-optimization-formulation/evals/evals.json
Refines numopt-form-eval-001 ground_truth/expected_behavior into detailed multiline explanations (explicitly: implicit objective = maximize profit). Adds numopt-form-eval-002 (cutting-stock objective should minimize total material consumption, not ignore over-production waste) and numopt-form-eval-003 (goal programming: deviation variables remain continuous, but discrete count decision variables such as worker counts must be INTEGER). All remain "concepts only" with no code/API call sequences.

Signature Update

Layer / File(s)Summary
Update DSSE payload and signature
skills/cuopt-numerical-optimization-formulation/skill.oms.sig
Replaces the DSSE envelope payload (subject/predicate/resource digests) and the DSSE signature value to match updated content; verification material and tlogEntries unchanged.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Possibly related PRs

  • NVIDIA/cuopt#1433: Both PRs adjust the SKILL.md YAML frontmatter-to-content formatting (blank line after the closing ---) to enable/recompute NVSkills signing, and the main PR also updates the corresponding skill.oms.sig bundle.

Suggested reviewers

  • tmckayus
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check nameStatusExplanation
Title check✅ PassedThe title accurately describes the main change: adding 3 evaluation cases to the cuopt-numerical-optimization-formulation skill's evals.json file.
Description check✅ PassedThe description is related to the changeset, explaining that 2 more evals were added to bring the total from 1 to 3, sourced from SKILL.md formulation patterns.
Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch trigger-numerical-opt-formulation-eval

Comment @coderabbitai help to get the list of available commands.

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

1 similar comment
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

Adds cutting-stock objective gotcha and goal-programming integer-decision
gotcha — both drawn from skill-specific patterns in SKILL.md that agents
without the skill typically get wrong.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@ramakrishnap-nvramakrishnap-nv changed the title ci: nudge numerical-optimization-formulation skill to trigger evalevals: add 3 formulation evals for cuopt-numerical-optimization-formulation skillJun 8, 2026
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

2 similar comments
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

1 similar comment
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/ok to test 688fffc

@github-actions

Copy link
Copy Markdown

🔔 Hi @anandhkb @rgsl888prabhu, this pull request has had no activity for 7 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.

@copy-pr-bot

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/ok to test dd41260

@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@ramakrishnap-nv
ramakrishnap-nv merged commit f1d1b6b into mainJun 22, 2026
4 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvementImproves an existing functionalitynon-breakingIntroduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ramakrishnap-nv@Iroy30@svc-nvskills-signing