Structured Evaluation

A reusable evaluation framework for LLM-as-Judge and multi-agent workflows.

Overview

structured-evaluation provides standardized types for evaluation reports, enabling:

• ⚖️ LLM-as-Judge assessments with categorical and 1-5 integer scoring
• 🔧 Automated repair via reason codes with repair prompts
• 📊 Coverage tracking for spec completeness metrics
• 📈 Confidence & routing for human review of low-confidence evaluations
• ✅ GO/NO-GO summary reports for deterministic checks (CI, tests, validation)
• 🔗 Multi-agent coordination with DAG-based report aggregation
• 📋 Claims validation for factual claim extraction and source verification
• 🟦 TypeScript / Zod bindings generated from the same Go structs, never hand-maintained

Architecture

┌───────────────────────────────────────────────────────────┐
│ SummaryReport (GO/NO-GO) │
│ ┌──────────────────────┐ ┌──────────────────────┐ │
│ │ Embedded Reports │ │ Team Sections │ │
│ │ (Full-Fidelity) │ │ (Task Results) │ │
│ └──────────────────────┘ └──────────────────────┘ │
└───────────────────────────────────────────────────────────┘
▲
┌───────────────┴───────────────┐
│ │
┌─────────────┴─────────────┐ ┌─────────────┴─────────────┐
│ Rubric (rubric/) │ │ ClaimsReport (claims/) │
│ ┌─────────────────────┐ │ │ ┌─────────────────────┐ │
│ │ Category Results │ │ │ │ Claims + Validation │ │
│ │ (pass/partial/fail) │ │ │ │ (verified/rejected) │ │
│ ├─────────────────────┤ │ │ ├─────────────────────┤ │
│ │ Findings │ │ │ │ Sources │ │
│ │ (severity-based) │ │ │ │ (external/internal) │ │
│ └─────────────────────┘ │ │ └─────────────────────┘ │
│ LLM-as-Judge scoring │ │ Fact verification │
└───────────────────────────┘ └───────────────────────────┘

Three complementary report types:

Package	Purpose	Evaluation Type
rubric/	Categorical scoring with findings	Subjective (LLM-as-Judge)
claims/	Fact verification with sources	Objective (source-backed)
summary/	GO/NO-GO aggregation	Deterministic

Installation

go get github.com/plexusone/structured-evaluation

Packages

Package	Description
rubric	Rubric, CategoryResult, Finding, Severity types for LLM-as-Judge
claims	ClaimsReport, Claim, Validation, Verdict for source verification
summary	SummaryReport, TeamSection, TaskResult for GO/NO-GO checks
combine	DAG-based report aggregation using Kahn's algorithm
render/box	ASCII box renderer for deterministic TUI output
render/detailed	Detailed terminal renderer for rubric reports
render/terminal	ANSI-colored terminal renderer with UTF8 icons
render/markdown	Markdown report renderer
render/html	Self-contained HTML renderer for claims reports, grouped by verdict
schema	JSON Schema generation and embedding

Report Types

Rubric (LLM-as-Judge)

For subjective quality assessments with detailed findings:

import"github.com/plexusone/structured-evaluation/rubric"report:=rubric.NewRubric("prd", "document.md")
// Add category with 1-5 integer score and confidenceresult:=rubric.NewCategoryResultWithIntScore(
"problem_definition",
rubric.ScoreGood, // 4/50.9, // High confidence"Clear problem statement with measurable goals",
)
report.AddCategoryResult(*result)
// Add finding with reason code for automated repairfinding:=rubric.NewFindingWithCode(
"f1", "metrics",
rubric.CodeMETRICNoBaseline,
"Missing baseline metrics",
"No baseline measurements defined for success metrics",
)
finding.SetRecommendation("Add current baseline measurements")
report.AddFinding(*finding)
// Set minimum score thresholdreport.PassCriteria.MinIntScore=rubric.ScoreGood// Require 4+report.Finalize(nil, "sevaluation check document.md")
// report.IntScore, report.Confidence, report.Blocking are computed

Summary Report (GO/NO-GO)

For deterministic checks with pass/fail status:

import"github.com/plexusone/structured-evaluation/summary"report:=summary.NewSummaryReport("my-service", "v1.0.0", "Release Validation")
report.AddTeam(summary.TeamSection{
ID: "qa",
Name: "Quality Assurance",
Tasks: []summary.TaskResult{
{ID: "unit-tests", Status: summary.StatusGo, Detail: "Coverage: 92%"},
{ID: "e2e-tests", Status: summary.StatusWarn, Detail: "2 flaky tests"},
},
})

Claims Report (v0.6.0)

For factual claim extraction and source validation:

import"github.com/plexusone/structured-evaluation/claims"report:=claims.NewClaimsReport("security-advisory.md")
// External source: CVE from NVDclaim:=claims.NewClaim("cvss", "CVSS 8.8 High", claims.ClaimRiskAssessment,
claims.Location{Section: "severity"})
claim.SetValidation(claims.NewExternalValidation(
"https://nvd.nist.gov/vuln/detail/CVE-2026-25253",
claims.ExternalNVD,
))
report.AddClaim(*claim)
// Internal validation: exploit confirmed via codeexploit:=claims.NewClaim("exploit", "RCE confirmed", claims.ClaimTechnicalFinding,
claims.Location{Section: "impact"})
exploit.SetValidation(claims.NewInternalValidation(
claims.MethodCodeExecution, "poc.py", true,
))
report.AddClaim(*exploit)
report.Finalize()
// report.Decision.Passed, report.Summary.Counts

Severity Levels

Following InfoSec conventions:

Severity	Icon	Blocking	Description
Critical	🔴	Yes	Must fix before approval
High	🔴	Yes	Must fix before approval
Medium	🟡	No	Should fix, tracked
Low	🟢	No	Nice to fix
Info	⚪	No	Informational only

CategoryResult.Severity (v0.11.0) is the worst severity among a category's findings, computed automatically — never set independently by the judge, so it can't drift from Findings. Useful for sorting or highlighting which categories to fix first, distinct from Score/IntScore (which measure quality, not urgency).

Pass Criteria

Default criteria (zero blocking findings, all categories passing):

criteria:=rubric.DefaultPassCriteria()
// MaxCritical: 0, MaxHigh: 0, MaxMedium: -1 (unlimited), RequireAllPass: falsecriteria:=rubric.StrictPassCriteria()
// MaxCritical: 0, MaxHigh: 0, MaxMedium: 3, RequireAllPass: true

Report Validation (v0.7.0)

Validate evaluation reports for correctness:

result:=rubric.ValidateReport(&report)
if!result.Valid {
fmt.Printf("Invalid: %d errors, %d warnings\n", result.ErrorCount, result.WarningCount)
for_, issue:=rangeresult.Issues {
fmt.Printf("[%s] %s: %s\n", issue.Severity, issue.Path, issue.Message)
}
}
// Get valid enum values for toolingscores:=rubric.ValidScoreValues() // ["pass", "partial", "fail"]severities:=rubric.ValidSeverityValues() // ["critical", "high", "medium", "low", "info"]

CLI Tool

# Install
go install github.com/plexusone/structured-evaluation/cmd/sevaluation@latest
# Render reports
sevaluation render report.json --format=detailed
sevaluation render report.json --format=terminal # ANSI colors + UTF8 icons
sevaluation render report.json --format=markdown # Markdown output
sevaluation render report.json --format=box
sevaluation render report.json --format=json
# Lint reports for correctness (v0.7.0; claims-report checks added v0.13.0)
sevaluation lint report.json # Basic validation
sevaluation lint report.json --strict # Warnings are errors
sevaluation lint report.json --format=json
# Claims reports: render, lint, check all auto-detect (top-level "claims" key)
sevaluation render claims.json --format=html > report.html
sevaluation lint claims.json --strict
# Check pass/fail (exit code 0/1)
sevaluation check report.json
# Validate structure
sevaluation validate report.json
# Generate JSON Schema
sevaluation schema generate -o ./schema/

DAG-Based Aggregation

For multi-agent workflows with dependencies:

import"github.com/plexusone/structured-evaluation/combine"results:= []combine.AgentResult{
{TeamID: "qa", Tasks: qaTasks},
{TeamID: "security", Tasks: secTasks, DependsOn: []string{"qa"}},
{TeamID: "release", Tasks: relTasks, DependsOn: []string{"qa", "security"}},
}
report:=combine.AggregateResults(results, "my-project", "v1.0.0", "Release")
// Teams are topologically sorted: qa → security → release

JSON Schema

Schemas are embedded for runtime validation:

import"github.com/plexusone/structured-evaluation/schema"rubricSchema:=schema.RubricSchemaJSONclaimsSchema:=schema.ClaimsSchemaJSONsummarySchema:=schema.SummarySchemaJSON

TypeScript / Zod (v0.11.0)

For consumers that read reports in TypeScript, @plexusone/structured-evaluation provides Zod schemas and TS types (Rubric, RubricSet, ClaimsReport, SummaryReport) generated from the same JSON Schema above — downstream of the Go structs, never hand-maintained:

npm install @plexusone/structured-evaluation

import{RubricSchema,typeRubric}from'@plexusone/structured-evaluation'constreport: Rubric=RubricSchema.parse(JSON.parse(rawJson))console.log(report.intScore)// the 1-5 score, correctly typedconsole.log(report.categories[0].severity)// "critical" | "high" | "medium" | "low" | "info" | undefined

Every schema is .strict() — an unrecognized key (a stale consumer still expecting a field that was renamed upstream) fails parsing loudly instead of silently reading undefined. See ts/README.md for regeneration instructions and known limitations.

RubricSet (v0.4.0)

Define explicit criteria for consistent categorical evaluations:

cat:=rubric.NewCategory("quality", "Output Quality", "Overall quality assessment").
WithPassPartialFail(
[]string{"Meets all requirements, no significant issues"},
[]string{"Meets most requirements, minor issues"},
[]string{"Missing key requirements or major issues"},
)
rubricSet:=rubric.NewRubricSet("output-review", "Output Review", "1.0").
AddCategory(*cat)

Rich Weighted Criteria (v0.10.0)

Categories can decompose into weighted sub-criteria, each with pass/partial/fail bands carrying a description and concrete indicators, scored by numeric scoreThresholds. Rubric definitions also carry yaml tags, so they can be authored as YAML and parsed directly into a RubricSet. See docs/features/rubrics.md.

Layered Classification (v0.14.0)

Categories and criteria can be classified so a rubric separates advisory, principle-based judgment from gating implementation checks instead of collapsing everything into one composite score:

cat:=rubric.NewCategory("traceability", "Requirement Traceability",
"Every requirement maps to a stable ID").
WithPassPartialFail(pass, partial, fail)
cat.Class=rubric.ClassDeterministicIntegrity// vs. ClassLeadershipPrinciple, etc.cat.Evaluation=rubric.EvalMethodDeterministic// vs. EvalMethodSemantic / EvalMethodHumancat.Blocking=true// a hard gate, distinct from RequiredrubricSet.JudgeInstructions= []string{ // cross-category evidence-discipline rules"Cite the relevant section and requirement IDs for every score",
"Do not reward length; reward completeness and precision",
}

Class, Blocking, and Evaluation live on both Category and Criterion; all are omitempty, so a v0.13.0-shaped rubric parses unchanged. RubricSet.Validate() enforces INV-3: a leadership_principle class must never be Blocking — advisory judgment cannot gate implementation. See docs/features/rubrics.md.

Judge Metadata (v0.2.0)

Track LLM judge configuration for reproducibility:

judge:=rubric.NewJudgeMetadata("claude-3-opus").
WithProvider("anthropic").
WithPrompt("prd-eval-v1", "1.0").
WithTemperature(0.0).
WithTokenUsage(1500, 800)
report.SetJudge(judge)

Pairwise Comparison (v0.2.0)

Compare two outputs instead of absolute scoring:

comparison:=rubric.NewPairwiseComparison(input, outputA, outputB)
comparison.SetWinner(rubric.WinnerA, "A is more accurate", 0.9)
// Aggregate multiple comparisonsresult:=rubric.ComputePairwiseResult(comparisons)
// result.WinRateA, result.OverallWinner

Multi-Judge Aggregation (v0.4.0)

Combine evaluations from multiple judges:

result:=rubric.AggregateEvaluations(evaluations, rubric.AggregationMajority)
// Methods: AggregationMajority, AggregationConservative, AggregationOptimistic// result.Agreement - inter-judge agreement (0-1)// result.Disagreements - categories with significant disagreement// result.ConsolidatedDecision - final aggregated decision

Likert Scales (v0.5.0)

Use 1-5 numeric scales for human comparison studies:

// Create a Likert-scale categorycat:=rubric.NewCategory("quality", "Content Quality", "Overall quality").
WithLikert5(rubric.StandardLikert5Anchors())
// Record a Likert score (automatically maps to categorical)result:=rubric.NewCategoryResultFromLikert("quality", 4, config, "Good quality")
// result.Score = ScorePass, result.NumericScore = 4.0// Or record both categorical and numericresult:=rubric.NewCategoryResultWithNumeric("quality", rubric.ScorePass, 4.5, "reasoning")

Inter-Rater Reliability (v0.5.0)

Compare LLM evaluations with human ground truth:

// Compute IRR metricsmetrics:=rubric.ComputeIRRFromResults(humanResults, llmResults)
fmt.Printf("Exact Agreement: %.1f%%\n", metrics.ExactAgreement*100)
fmt.Printf("Adjacent Agreement: %.1f%%\n", metrics.AdjacentAgreement*100)
fmt.Printf("Pearson r: %.3f\n", metrics.PearsonCorrelation)
// Categorical agreement with confusion matrixagreement:=rubric.ComputeCategoricalAgreement(humanResults, llmResults)

Claims Validation (v0.6.0)

Validate factual claims have proper source backing:

import"github.com/plexusone/structured-evaluation/claims"report:=claims.NewClaimsReport("article.md")
// Source types: external (URL), internal (code/lab), derived, subjective// External sources also classify aggregator sites (stats-roundup blogs// with no original reporting) as low-reliability, auto-reject — distinct// from an unrecognized-but-real community source, which just needs review// Reliability tiers: authoritative, high, medium, low// Verdicts: verified, unverified, needs-review, rejected// ClaimStatistical claims can carry a structured value/unit/precision/as-of// date via claim.SetStatistical(...), independent of the rendered Text// Configure pass criteriareport.SetCriteria(claims.ClaimsCriteria{
RequireAllVerified: true,
AllowSubjectiveWithDisclaimer: false,
MinReliabilityTier: claims.ReliabilityHigh,
})
report.Finalize()
ifreport.IsPassing() {
fmt.Println("Ready for publication")
}

Evidence-Integrity Linting (v0.13.0)

A verdict can be hand-authored directly on a Claim, bypassing DetermineVerdict entirely. claims.Lint re-checks that every claim stated as verified actually earns the label — this is the check that would have caught a "$3B ARR" claim sourced from a secondary-analysis estimate that a primary-source check later put closer to $500M:

findings:=claims.Lint(&report)
ifclaims.HasErrors(findings) {
// a verified claim is missing a URL/quote/evidence, or an opted-in// criteria check (corroboration, staleness) failed
}

• SourceRole (primary / secondary-relay / secondary-analysis / self-reported) on ExternalValidation.Role — orthogonal to ExternalSourceType's authority tier, capturing how directly a source speaks for the claim. secondary-analysis and self-reported require a corroborating RelatedClaimIDs entry.
• ClaimsCriteria.MinCorroboratingSources — a configurable "N independent sources" threshold, applied regardless of role.
• ClaimsCriteria.MaxClaimAge — flags a verified statistic whose Statistical.AsOfDate is older than the threshold.

Both new criteria are opt-in (disabled by default) and, when set, feed into EvaluateClaims's report-level decision the same way needs-review claims do. See Evidence-Integrity Linting for the full rule reference.

Embedded Reports (v0.6.0)

Archive full-fidelity reports within SummaryReport:

report:=summary.NewSummaryReport("project", "v1.0.0", "RELEASE")
// Embed detailed reportsreport.EmbedRubricReport("quality-review", rubricReport)
report.EmbedClaimsReport("source-validation", claimsReport)
// Retrieve latervarr rubric.Rubricreport.GetEmbeddedRubricReport("quality-review", &r)

OmniObserve Integration

Export evaluations to Opik, Phoenix, or Langfuse:

import"github.com/plexusone/omniobserve/integrations/sevaluation"// Export to observability platformerr:=sevaluation.Export(ctx, provider, traceID, report)

Integration

Designed to work with:

• github.com/plexusone/omniobserve - LLM observability (Opik, Phoenix, Langfuse)
• github.com/grokify/structured-requirements - PRD evaluation templates
• github.com/plexusone/multi-agent-spec - Agent coordination
• github.com/grokify/structured-changelog - Release validation

License

MIT License - see LICENSE for details.