Test code quality rubric with static datasets (Meta-Evals)

tool/dart_skills_lint/.agents/skills/run-evals/SKILL.md

@@ -8,10 +8,10 @@ metadata:
 # Run Skill Evals
 
 1. **Read Framework**: Read `tool/dart_skills_lint/evals/README.md` for understanding the difference between per-skill evals and cross-skill evals.
-2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`.
-3. **Determine Agent Configuration**: Check the `agent_config` field in the target `evals.json` to determine the environment/harness to spawn. If `agent_config` is `"bare-agent"`, spawn a subagent with the `bare-agent` profile. If it is a specific contributor profile (e.g. `"reidbaker-agent"`), use that profile to provide the necessary contributor context.
+2. **Locate Targets**: Find target `evals/evals.json` files inside `.agents/skills/` and/or `skills/`. For cross-skill evaluations, look for `*_evals.json` files directly in `tool/dart_skills_lint/evals/`.
+3. **Determine Agent Configuration**: Check the `agent_config` field in the target target JSON file to determine the environment/harness to spawn. If `agent_config` is `"bare-agent"`, spawn a subagent with the `bare-agent` profile. If it is a specific contributor profile (e.g. `"reidbaker-agent"`), use that profile to provide the necessary contributor context.
 4. **Orchestrate**: By default, run an Integration Test by spawning a single **With-Skill** subagent using `Workspace: branch` and the identified `agent_config`.
- - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in `<path-to-skill>`, you MUST use a relative path from the repository root, not an absolute path, to prevent the subagent from accidentally anchoring to and modifying the parent workspace.
+ - Provide the task prompt. See `resources/with_skill_execution_prompt.md` for the template. When filling in `<path-to-skill>`, you MUST use a relative path from the repository root, not an absolute path. If you are running a cross-skill evaluation, fill in `<path-to-skill>` with `"none (cross-skill meta-eval)"`.
  - **Only if the user explicitly requests a comparison or benchmark**, also spawn a **Baseline** subagent. See `resources/baseline_execution_prompt.md` for the template.
  Instruct the subagent(s) to return their `git diff` and verification outputs (`dart format`, `dart analyze`, `dart test`) without committing.
  **CRITICAL**: You must explicitly warn the subagent(s) to confine all file edits strictly to their current working directory and avoid using absolute paths to modify the parent workspace.

tool/dart_skills_lint/.agents/skills/run-evals/resources/baseline_execution_prompt.md

@@ -5,3 +5,4 @@ Execute this task:
 WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace.
 
 Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`).
+NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results.

tool/dart_skills_lint/.agents/skills/run-evals/resources/with_skill_execution_prompt.md

@@ -6,3 +6,4 @@ Execute this task:
 WARNING: You are executing in an isolated branch workspace. Confine all file modifications strictly to your current working directory. Do NOT use absolute paths to modify files in the parent workspace.
 
 Once you are done, do not commit. Just send me a message with the `git diff` of your changes, and the output of running verification commands (e.g., `dart format`, `dart analyze`, `dart test`).
+NOTE: If your task is strictly to grade, review, or evaluate code, do NOT fix the issues you find. Leave the code exactly as it is, even if verification commands fail. Your job is only to report the evaluation results.

tool/dart_skills_lint/evals/README.md

@@ -44,3 +44,8 @@ dart test test/skills_evals_test.dart
 
 ### 2. Running Evals via Agent Orchestration
 You should use the `/run-evals` skill to run evaluations. The bulk of the execution logic and prompts are located within the `run-evals` skill itself (`.agents/skills/run-evals/SKILL.md`). The environment, model, and harness are determined by the `agent_config` specified in the corresponding `evals.json` file.
+
+### 3. Testing Meta-Evals (Testing the Rubrics)
+To ensure our universal rubrics correctly catch anti-patterns (and permit clean code), we use meta-evaluations. Standalone cross-skill evaluations are defined as `evals/*_evals.json` files (e.g., `evals/code_quality_rubric_evals.json`). These files contain evals strictly intended to grade static fixtures located in `evals/test_data/`.
+
+To run the meta-evals and verify the rubrics, invoke the `/run-evals` skill and ask the agent to run the standalone `code_quality_rubric_evals.json` file.

tool/dart_skills_lint/evals/code_quality_rubric_evals.json

@@ -0,0 +1,31 @@
+{
+ "repo_criteria": [
+ "evals/code_quality_rubric.json"
+ ],
+ "evals": [
+ {
+ "id": 1,
+ "prompt": "Grade the code in 'evals/test_data/bad_code/tmp/bad_script.dart' using the code_quality_rubric.json.",
+ "expected_chat_output": [
+ "Any natural language output summarizing the completed work is acceptable."
+ ],
+ "expected_repo_state": [
+ "The evaluation artifact explicitly flags a failure for directory placement hygiene.",
+ "The evaluation artifact explicitly flags a failure for effective Dart idioms.",
+ "The evaluation artifact explicitly flags a failure for cross platform compatibility."
+ ],
+ "agent_config": "reidbaker-agent"
+ },
+ {
+ "id": 2,
+ "prompt": "Grade the code in 'evals/test_data/ok_code/bin/good_script.dart' using the code_quality_rubric.json.",
+ "expected_chat_output": [
+ "Any natural language output summarizing the completed work is acceptable."
+ ],
+ "expected_repo_state": [
+ "The evaluation artifact explicitly flags that all code quality criteria passed successfully without any failures."
+ ],
+ "agent_config": "reidbaker-agent"
+ }
+ ]
+}

tool/dart_skills_lint/evals/test_data/bad_code/tmp/bad_script.dart

@@ -0,0 +1,19 @@
+// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+// ignore_for_file: unused_local_variable, prefer_final_locals, avoid_print, use_raw_strings
+
+// Violates placement hygiene (in tmp/ instead of bin/ or lib/)
+// Violates cross-platform compatibility (hardcoded Windows path)
+// Violates effective Dart idioms (raw strings instead of interpolation)
+
+void main() {
+ var user = 'Test';
+
+ // Anti-pattern: Raw string concatenation instead of interpolation
+ print(r'Hello ' + user);
+
+ // Anti-pattern: Hardcoded platform specific path
+ var path = 'C:\\my\\path\\data.txt';
+}

tool/dart_skills_lint/evals/test_data/ok_code/bin/good_script.dart

@@ -0,0 +1,19 @@
+// Copyright (c) 2026, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+import 'package:path/path.dart' as p;
+
+void main() {
+ const user = 'Test';
+
+ // Interpolation (clean Dart idiom)
+ const message = 'Hello $user';
+
+ // Cross platform path
+ final String path = p.join('my', 'path', 'data.txt');
+
+ if (message.isEmpty || path.isEmpty) {
+ throw Exception('Validation failed');
+ }
+}

tool/dart_skills_lint/test/skills_evals_test.dart

@@ -4,6 +4,7 @@
 
 import 'dart:convert';
 import 'dart:io';
+import 'dart:isolate';
 
 import 'package:path/path.dart' as p;
 import 'package:test/test.dart';
@@ -12,10 +13,16 @@ void main() {
  group('Evals structure consistency', () {
  // Ensures all evals.json files dynamically share the exact same JSON schema.
  // Keys are not hardcoded to ensure enforcement remains schema-agnostic and flexible.
- test('all evals.json files across skills share consistent structure and keys', () {
+ test('all evals.json files across skills share consistent structure and keys', () async {
+ final Uri? packageUri = await Isolate.resolvePackageUri(
+ Uri.parse('package:dart_skills_lint/'),
+ );
+ final String packageRoot = packageUri!.resolve('..').toFilePath();
+
  final List<File> evalsFiles = [
- ..._findEvalsFiles(Directory(p.join(Directory.current.path, 'skills'))),
- ..._findEvalsFiles(Directory(p.join(Directory.current.path, '.agents', 'skills'))),
+ ..._findEvalsFiles(Directory(p.join(packageRoot, 'skills'))),
+ ..._findEvalsFiles(Directory(p.join(packageRoot, '.agents', 'skills'))),
+ ..._findEvalsFiles(Directory(p.join(packageRoot, 'evals'))),
  ]..sort((a, b) => a.path.compareTo(b.path));
 
  expect(
@@ -29,8 +36,13 @@ void main() {
 
  // Note: We intentionally only require an evals.json file for published skills.
  // Contributor skills in .agents/skills/ are not currently required to have one.
- test('all published skills have an evals.json file', () {
- final skillsDir = Directory(p.join(Directory.current.path, 'skills'));
+ test('all published skills have an evals.json file', () async {
+ final Uri? packageUri = await Isolate.resolvePackageUri(
+ Uri.parse('package:dart_skills_lint/'),
+ );
+ final String packageRoot = packageUri!.resolve('..').toFilePath();
+
+ final skillsDir = Directory(p.join(packageRoot, 'skills'));
  if (!skillsDir.existsSync()) {
  return;
  }
@@ -48,8 +60,13 @@ void main() {
  }
  });
 
- test('all rubric JSON files in evals/ share consistent structure and keys', () {
- final rubricsDir = Directory(p.join(Directory.current.path, 'evals'));
+ test('all rubric JSON files in evals/ share consistent structure and keys', () async {
+ final Uri? packageUri = await Isolate.resolvePackageUri(
+ Uri.parse('package:dart_skills_lint/'),
+ );
+ final String packageRoot = packageUri!.resolve('..').toFilePath();
+
+ final rubricsDir = Directory(p.join(packageRoot, 'evals'));
  if (!rubricsDir.existsSync()) {
  return;
  }
@@ -58,7 +75,7 @@ void main() {
  rubricsDir
  .listSync()
  .whereType<File>()
- .where((File f) => f.path.endsWith('.json'))
+ .where((File f) => f.path.endsWith('.json') && !f.path.endsWith('_evals.json'))
  .toList()
  ..sort((a, b) => a.path.compareTo(b.path));
 
@@ -128,9 +145,8 @@ List<File> _findEvalsFiles(Directory baseDir) {
  if (!baseDir.existsSync()) {
  return [];
  }
- return baseDir
- .listSync(recursive: true)
- .whereType<File>()
- .where((File f) => p.basename(f.path) == 'evals.json')
- .toList();
+ return baseDir.listSync(recursive: true).whereType<File>().where((File f) {
+ final String name = p.basename(f.path);
+ return name == 'evals.json' || name.endsWith('_evals.json');
+ }).toList();
 }