Skip to content

Fix skipped test: llm-coverage-evaluator "backup judge" (stale vs current DEFAULT_JUDGES) #48

Description

@nojibe

Problem

The "backup judge" test in src/cli/evaluators/__tests__/llm-coverage-evaluator.test.ts is broken and currently .skipped. Its expectations are stale relative to the current DEFAULT_JUDGES defined in src/cli/evaluators/llm-coverage-evaluator.ts, so it fails.

This test was failing on main before the Vitest migration too — it simply never ran in CI, so the breakage went unnoticed. During the Jest→Vitest migration (#45 / PR #46) it was marked .skip with an explanatory note rather than deleted, so no coverage decision is hidden. Now that unit suites run in CI (#33 / PR #47), this should be fixed and re-enabled.

Task

  • Review the current DEFAULT_JUDGES in src/cli/evaluators/llm-coverage-evaluator.ts and the backup-judge selection logic.
  • Update the test's expectations to match current behavior (or fix the source if the test is asserting the correct intended behavior and the source regressed — determine which).
  • Remove the .skip and confirm it passes under pnpm test:cli.

Acceptance criteria

  • The "backup judge" test is un-skipped and passing
  • Test reflects the actual intended backup-judge behavior (not just made to pass)
  • pnpm test:cli green

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions