Skip to content

skills/cuopt-server-common: Add eval - #1325

Merged
ramakrishnap-nv merged 2 commits into
mainfrom
skills-cuopt-server-common
May 28, 2026
Merged

skills/cuopt-server-common: Add eval#1325
ramakrishnap-nv merged 2 commits into
mainfrom
skills-cuopt-server-common

Conversation

@ramakrishnap-nv

@ramakrishnap-nvramakrishnap-nv commented May 28, 2026

Copy link
Copy Markdown
Collaborator

skills/cuopt-server-common: Add eval

Adds an evals/evals.json with a QP-not-supported-over-REST trap case so
NV-BASE Tier 3 agent evaluation can demonstrate skill lift on the
server-common skill.
Last CI run: composite PASS with overall lift +0.16 on claude-code
(best-performing agent). Codex showed -0.05 variance on this run; the
per-agent gate may flag this on some CI runs due to LLM-judge variance
on saturated baselines.
Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
@copy-pr-bot

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@ramakrishnap-nv
ramakrishnap-nv marked this pull request as ready for review May 28, 2026 23:11
@ramakrishnap-nv
ramakrishnap-nv requested a review from a team as a code ownerMay 28, 2026 23:11
@ramakrishnap-nvramakrishnap-nv self-assigned this May 28, 2026
@ramakrishnap-nvramakrishnap-nv added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels May 28, 2026
@ramakrishnap-nvramakrishnap-nv changed the title skills/cuopt-server-common: Tier 3 evalskills/cuopt-server-common: Add evalMay 28, 2026
@ramakrishnap-nv

Copy link
Copy Markdown
CollaboratorAuthor

/nvskills-ci

@coderabbitai

Copy link
Copy Markdown

Actionable comments posted: 0

@coderabbitai

coderabbitaiBot commented May 28, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

Pull request was closed or merged during review

📝 Walkthrough

Walkthrough

Adds an evaluation case asserting QP is not supported over the cuOpt REST server, augments the skill card with evaluation metadata, adds a BENCHMARK report, and updates the Sigstore DSSE bundle to include the new evaluation file.

Changes

REST API Scope Clarification & Evaluation

Layer / File(s)Summary
Evaluation case and skill documentation
skills/cuopt-server-common/evals/evals.json, skills/cuopt-server-common/skill-card.md, skills/cuopt-server-common/BENCHMARK.md
Adds srv-common-eval-001-qp-not-supported-over-rest in evals.json; updates skill-card.md with evaluation metadata and adjusted use-case/license formatting; adds BENCHMARK.md evaluation report.
Sigstore DSSE bundle update
skills/cuopt-server-common/skill.oms.sig
Replaces the DSSE payload/signature to reflect added evals/evals.json and updated resource digests.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

non-breaking, improvement

Suggested reviewers

  • Iroy30
  • tmckayus
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check nameStatusExplanation
Title check✅ PassedThe title 'skills/cuopt-server-common: Add eval' clearly and concisely describes the main change—adding an evaluation to the cuOpt server common skill.
Description check✅ PassedThe description is identical to the title and relates to the changeset by referring to the addition of evaluations in the cuopt-server-common skill.
Docstring Coverage✅ PassedNo functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check✅ PassedCheck skipped because no linked issues were found for this pull request.
Out of Scope Changes check✅ PassedCheck skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch skills-cuopt-server-common

Comment @coderabbitai help to get the list of available commands and usage tips.

Signed-off-by: nvskills-svc-account <svc-nvskills-signing@nvidia.com>
@copy-pr-bot

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@ramakrishnap-nv
ramakrishnap-nv merged commit 0eebc4c into mainMay 28, 2026
3 of 4 checks passed
@ramakrishnap-nvramakrishnap-nv mentioned this pull request Jul 8, 2026
3 tasks
@ramakrishnap-nv
ramakrishnap-nv deleted the skills-cuopt-server-common branch July 9, 2026 18:34
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvementImproves an existing functionalitynon-breakingIntroduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@ramakrishnap-nv@tmckayus@svc-nvskills-signing