EIDOS episodic reasoning: prediction-outcome tracking for agent evolution

[AlexMikhalev]

Summary

Implement episodic reasoning in terraphim_agent_evolution: every advisory/decision carries a predicted outcome. When the actual outcome is observed, prediction accuracy feeds back into future confidence scoring.

Motivation

Inspired by vibeship-spark-intelligence EIDOS (Episode-based Distillation) loop. terraphim_agent_evolution exists as a crate but is incomplete -- it's supposed to "track agent performance metrics, evolve agent capabilities over time." EIDOS provides the concrete mechanism.

EIDOS Loop

1. PREDICT: "This advisory/action will produce outcome X"
 |
2. ACT: Agent or developer takes action
 |
3. OBSERVE: Actual outcome Y recorded
 |
4. EVALUATE: Compare prediction X vs outcome Y, compute accuracy
 |
5. DISTILL: Update confidence for this class of advisory/action
 |
 (loop)

Concrete Application in Terraphim

Learning Capture Predictions

When terraphim-agent learn surfaces a past learning as advisory:

• Predict: "Following this learning will prevent error type Z"
• Observe: Did the developer follow the advice? Did the error recur?
• Evaluate: If followed and error didn't recur -> prediction validated
• Distill: Increase learning's reliability score

Judge Verdict Predictions

When the judge system issues a verdict:

• Predict: "This code change will cause issue type Z if merged"
• Observe: Was the verdict followed? Did the predicted issue manifest?
• Evaluate: If ignored and issue manifested -> judge was right but ignored
• Distill: Increase judge finding weight for this pattern

Hook Replacement Predictions

When terraphim_hooks replaces text (e.g., npm -> bun):

• Predict: "This replacement will succeed without side effects"
• Observe: Did the subsequent command succeed?
• Evaluate: If replacement caused a failure -> hook rule needs refinement
• Distill: Update hook confidence or add exception rule

Implementation Plan

1. Prediction type in terraphim_types

pub struct Prediction {
 pub id: Ulid,
 pub source: PredictionSource, // Learning, Judge, Hook, Agent
 pub predicted_outcome: String,
 pub confidence: f64, // 0.0 - 1.0
 pub created_at: jiff::Timestamp,
 pub observed_outcome: Option<ObservedOutcome>,
 pub accuracy: Option<f64>,
}

2. Prediction tracking in terraphim_agent_evolution

• Store predictions in event store (Extend session store with real-time event capture and health invariants #597)
• Match outcomes to predictions via correlation IDs
• Compute rolling accuracy per prediction source
• Expose accuracy metrics via MCP resource

3. Confidence feedback

• When prediction accuracy is high (>0.8 over 10+ predictions): increase source confidence
• When prediction accuracy is low (<0.5 over 10+ predictions): decrease source confidence or flag for review
• Confidence scores feed back into advisory ranking and judge dimensional scoring (Dimensional verdict scoring for multi-model judge (Meta-Ralph pattern) #600)

Affected Crates

• terraphim_agent_evolution (primary -- implement EIDOS loop)
• terraphim_types (add Prediction, ObservedOutcome types)
• terraphim_hooks (emit predictions for text replacements)
• terraphim_mcp_server (expose prediction accuracy metrics)

Dependencies

• Extend session store with real-time event capture and health invariants #597 Event sourcing (predictions stored as events)
• Enhanced learning capture: multi-hook pipeline with importance scoring #599 Enhanced learning capture (learning advisories generate predictions)
• Dimensional verdict scoring for multi-model judge (Meta-Ralph pattern) #600 Dimensional verdict scoring (judge verdicts generate predictions)

Estimated Effort

~1 day for prediction types + basic tracking. Feedback loop refinement is ongoing.

Key Insight

Agent evolution is not about self-modifying code (Ouroboros pattern). It's about self-improving knowledge quality. The evolution happens in the advisory confidence scores, not in the source code. This is the right approach for terraphim's deterministic-first philosophy -- the Aho-Corasick engine stays constant, the confidence weights evolve.