Dimensional verdict scoring for multi-model judge (Meta-Ralph pattern)

[AlexMikhalev]

Summary

Replace binary pass/fail verdicts in the multi-model judge system with 6-dimensional continuous scoring. Each judge tier produces a score vector instead of a single verdict, enabling nuanced escalation decisions and feedback loops.

Motivation

Inspired by vibeship-spark-intelligence Meta-Ralph quality gating pattern. Current judge system uses 3-tier binary verdicts (pass/fail). Meta-Ralph scores across 6 dimensions with configurable thresholds.

Current State

Tier 1 (quick): pass/fail
 |
 v (if fail)
Tier 2 (deep): pass/fail
 |
 v (if disagree)
Tier 3 (tiebreaker): pass/fail

Proposed State

Tier 1 (quick): [reliability: 0.7, context_fit: 0.9, freshness: 0.8, ...]
 |
 v (if any dimension below threshold)
Tier 2 (deep): [reliability: 0.85, context_fit: 0.6, freshness: 0.9, ...]
 |
 v (if dimensions conflict across tiers)
Tier 3 (tiebreaker): [reliability: 0.9, context_fit: 0.8, freshness: 0.95, ...]
 |
 v
Final verdict: weighted aggregate across all tiers that responded

Six Scoring Dimensions

1. Reliability -- How often has this type of finding been validated in past verdicts?
2. Context fit -- How relevant is the finding to the specific code being reviewed?
3. Freshness -- Is the finding based on current patterns or stale heuristics?
4. Authority -- How credible is the source model for this type of finding?
5. Signal strength -- Confidence in the signal (strong evidence vs. weak heuristic)
6. Utility -- How actionable is the finding? (specific fix vs. vague warning)

Escalation Logic

Replace binary escalation with dimensional:

• Escalate when any dimension drops below tier-specific threshold (not just overall pass/fail)
• Skip escalation when all dimensions are above threshold even if one tier "failed" on a specific finding
• Aggregate final verdict as weighted average across tiers, with later tiers weighted higher on dimensions where earlier tiers scored low

Feedback Loop (Future)

Track whether findings were acted on (follow-rate):

• Finding surfaced -> developer fixed issue -> positive signal
• Finding surfaced -> developer dismissed -> negative signal
• Over time, adjust per-dimension weights based on follow-rate

This requires #597 (event sourcing) for tracking.

Schema Extension

Extend automation/judge/verdict-schema.json:

{
 "dimensions": {
 "reliability": { "score": 0.85, "threshold": 0.7 },
 "context_fit": { "score": 0.9, "threshold": 0.6 },
 "freshness": { "score": 0.8, "threshold": 0.5 },
 "authority": { "score": 0.7, "threshold": 0.6 },
 "signal_strength": { "score": 0.6, "threshold": 0.5 },
 "utility": { "score": 0.9, "threshold": 0.7 }
 },
 "escalated": false,
 "escalation_reason": null
}

Affected Components

• automation/judge/verdict-schema.json (extend schema)
• automation/judge/pre-push-judge.sh (update prompt to request dimensional scores)
• Judge prompt templates (instruct models to score per dimension)
• terraphim-skills judge plan (plans/terraphim-skills-judge.md)

Dependencies

• Extend session store with real-time event capture and health invariants #597 Event sourcing (for feedback loop tracking)
• Budget-aware multi-model routing with per-task cost tracking #598 Budget-aware routing (dimensional scoring needs cost-aware model selection)

Estimated Effort

~4 hours for schema + prompt changes. Feedback loop integration is a follow-up after #597.