Add skill evaluation dataset for cuopt-installation-developer by ramakrishnap-nv · Pull Request #1169 · NVIDIA/cuopt

ramakrishnap-nv · 2026-04-30T21:59:18Z

Summary

Skill evaluation dataset for cuopt-installation-developer at skills/cuopt-installation-developer/evals/evals.json. 10 entries focused on the build-from-source path. Companion to #1167 (cuopt-developer evals).

Coverage

Theme	Count
First-time build / required questions	3
CUDA driver / env selection	2
Major-version mismatch diagnosis & clean build	2
User vs developer install boundary	1
Hand-off to cuopt-developer	1
Refusal (sudo conda)	1

Same schema as #1167 (id, question, expected_skill, expected_script, ground_truth, expected_behavior). Schema check and pre-commit hooks pass locally.

Companion to NVIDIA#1167. 10 entries at skills/cuopt-installation-developer/evals/evals.json covering the build-from-source path: - inst-001-first-time-build: end-to-end first-time setup - inst-002-cuda-driver-check: how to pick a conda env via nvidia-smi - inst-003-cuda-major-mismatch-diagnosis: cudaMallocAsync runtime error - inst-004-required-questions: pre-install clarifications - inst-005-build-prereqs: high-level dependencies - inst-006-clean-build-cuda-switch: ./build.sh clean across major switch - inst-007-user-vs-dev-install: redirect non-contributors to user skill - inst-008-after-build-works: hand-off to cuopt-developer - inst-009-list-env-files: ls conda/environments/all_cuda-*_arch-*.yaml - inst-010-no-sudo-for-install: refuse sudo conda env create Same schema as cuopt-developer evals (id, question, expected_skill, expected_script, ground_truth, expected_behavior). Schema check and pre-commit hooks pass. Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>

copy-pr-bot · 2026-04-30T21:59:21Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

coderabbitai · 2026-05-01T14:58:45Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b9b29517-f7c3-4741-a08c-e55d47ccb363

📥 Commits

Reviewing files that changed from the base of the PR and between f277565 and 68affad.

📒 Files selected for processing (1)

skills/cuopt-installation-developer/evals/evals.json

🚧 Files skipped from review as they are similar to previous changes (1)

skills/cuopt-installation-developer/evals/evals.json

📝 Walkthrough

Walkthrough

Adds a new evaluation dataset for the cuopt-installation-developer skill with 10 scenarios covering developer-focused from-source CUDA setup, conda env selection, CUDA version diagnostics, clean rebuilds, prerequisites, contribution workflow, and handling of conda permission issues.

Changes

Cohort / File(s)	Summary
Evaluation dataset `skills/cuopt-installation-developer/evals/evals.json`	Adds 10 evaluation scenarios for developer-focused, from-source CUDA/conda installation: env file selection, CUDA major/minor compatibility checks, runtime diagnostic guidance (e.g., `cudaMallocAsync` mismatch), clean rebuild requirement, prerequisite checklist, contributor workflow notes, env enumeration, and refusal of `sudo conda env create`.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Suggested reviewers

tmckayus

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely summarizes the main change: adding an evaluation dataset for the cuopt-installation-developer skill.
Description check	✅ Passed	The description is directly related to the changeset, providing clear details about the evaluation dataset being added, its coverage, schema, and verification status.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Review rate limit: 9/10 reviews remaining, refill in 6 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@skills/cuopt-installation-developer/evals/evals.json`:
- Around line 135-141: Update the ground_truth and expected_behavior entries to
remove ambiguity about sudo: change the "ground_truth" sentence that currently
says the agent declines sudo "without explicit user approval" to state the agent
always refuses to run sudo on the user's behalf, and modify the first item in
"expected_behavior" from "Refuses to run sudo on the user's behalf" to "Always
refuses to run sudo on the user's behalf"; ensure this aligns with the inst-010
evaluation so there is no conditional allowance of sudo even with approval.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6e25030c-bf66-4360-ad0e-220c5deac2d8

📥 Commits

Reviewing files that changed from the base of the PR and between 746f46b and f277565.

📒 Files selected for processing (1)

skills/cuopt-installation-developer/evals/evals.json

mlubin · 2026-05-01T15:01:15Z

Nice! Does this actually test that agents successfully build or is it just checking their textual outputs?

ramakrishnap-nv · 2026-05-01T15:05:57Z

Nice! Does this actually test that agents successfully build or is it just checking their textual outputs?

This would check their textual output or steps that it would take, and there is a LLM judge which will check whether it took all appropriate steps or not.

CodeRabbit pointed out that 'declines to run sudo without explicit user approval' implies sudo would be acceptable with approval, which contradicts the unconditional 'Refuses to run sudo' bullet. For this scenario the refusal does not depend on the consent rule — the conda env permission error is almost always a non-writable prefix, which sudo does not actually fix. Rewording the ground_truth and expected_behavior to ground the refusal in the diagnosis ('sudo is the wrong fix') so the eval is unambiguous. Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>

mlubin · 2026-05-01T16:12:13Z

This would check their textual output or steps that it would take, and there is a LLM judge which will check whether it took all appropriate steps or not.

Seems like a potential loophole; we don't know if the steps it would take are correct if it doesn't actually build successfully.

ramakrishnap-nv · 2026-05-04T21:58:58Z

/merge

ramakrishnap-nv self-assigned this Apr 30, 2026

ramakrishnap-nv added non-breaking Introduces a non-breaking change improvement Improves an existing functionality labels Apr 30, 2026

ramakrishnap-nv marked this pull request as ready for review May 1, 2026 14:55

ramakrishnap-nv requested a review from a team as a code owner May 1, 2026 14:55

ramakrishnap-nv requested a review from tmckayus May 1, 2026 14:55

coderabbitaiBot reviewed May 1, 2026
View reviewed changes

Comment threadskills/cuopt-installation-developer/evals/evals.json Outdated

Merge branch 'main' into add-installation-developer-evals
ff19fca

tmckayus approved these changes May 4, 2026
View reviewed changes

rapids-botBot merged commit 336dd4f into NVIDIA:mainMay 4, 2026
43 checks passed

Navigation Menu

Uh oh!

Add skill evaluation dataset for cuopt-installation-developer - #1169

Add skill evaluation dataset for cuopt-installation-developer#1169
rapids-bot[bot] merged 3 commits into
NVIDIA:mainfrom
ramakrishnap-nv:add-installation-developer-evals

ramakrishnap-nv commented Apr 30, 2026

Uh oh!

copy-pr-botBot commented Apr 30, 2026

Uh oh!

coderabbitaiBot commented May 1, 2026•
edited
Loading

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

Uh oh!

coderabbitaiBot left a comment

Uh oh!

Uh oh!

mlubin commented May 1, 2026

Uh oh!

ramakrishnap-nv commented May 1, 2026•
edited
Loading

Uh oh!

mlubin commented May 1, 2026

Uh oh!

ramakrishnap-nv commented May 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

ramakrishnap-nv commented Apr 30, 2026

Summary

Coverage

Uh oh!

copy-pr-botBot commented Apr 30, 2026

Uh oh!

coderabbitaiBot commented May 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

Uh oh!

coderabbitaiBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlubin commented May 1, 2026

Uh oh!

ramakrishnap-nv commented May 1, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mlubin commented May 1, 2026

Uh oh!

ramakrishnap-nv commented May 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitaiBot commented May 1, 2026•
edited
Loading

ramakrishnap-nv commented May 1, 2026•
edited
Loading