feat(#814): believability scores memory carry — facts reused across conversations

[0frankie]

What this is

A sixth rubric dimension for backend/eval/believability.py: memory_carry — does a later conversation reuse a specific detail that first entered this agent's stream in an earlier conversation, ideally with a different partner? That is the behavior #760's R1 demonstrated — the single clearest evidence the cognition stack works — and the one #781 shipped only proxies for (per-decision memory_use, the social_grounding novelty term).

It is a pure offline heuristic like the dimensions around it — no LLM calls, deterministic, always available under --no-llm. The LLM judge inherits the dimension through DIMENSIONS (tool schema + rubric bullet), exactly as world_grounding (#780) was registered.

How it scores

Per conversation after the agent's first, walk the agent's own lines for a detail that:

• was heard, not owned — it first appeared in a partner's line of an earlier window (or is an earlier partner: meeting someone is how their name enters your stream, which is what lets "My friend Diego mentioned it might be interesting" score), and was not already in the agent's persona, schedule, or plan memories;
• does not name the current interlocutor — direct address ("Hey Chris!") is not recall;
• is specific (below).

Full credit when the source conversation shares no partner with the reusing one (the strong, R1-shaped form), 0.5 when the agent retells its source. The dimension is the mean credit over those later conversations; fewer than two conversations is n/a, excluded from the means like every other n/a.

"Specific detail": what the issue's suggested proxy measured out to

The issue records three proxies as already rejected (kept here so nobody re-derives them):

proxy	why it fails
retrieval reach (step - created_turn)	backwards: R2's loop reaches further back than healthy R1 (median 185 vs 111 steps); a static agent re-retrieving one old plan memory maximises reach
distinct-memories-used	no dynamic range: 2–12 across every agent in the batch
bare content-word overlap	can't tell carry from vocabulary collapse: the word aiden alone was 4,775 of Chris Donnelly's overlaps

The issue's suggested cheapest-thing — rare-word weighting over the run's corpus — was validated against the reference runs as instructed, and it fails in both available corpus readings:

rarity corpus	measured failure
transcript lines only	a small run makes everything look rare: plain filler slips through as "carry" ("doing", "anyway", "hoping", "nice" — the R2 loop pair scored 3.6–4.9 from filler alone)
whole corpus (memories, acts, schedules, personas)	inflates exactly the true positives: a fact that was carried gets re-memorized and re-discussed (design df 57, diego df 74 in R1), so the carried details get filtered out

So the shipped definition is structural first, frequency second: a detail is a met agent's name, a mid-sentence proper noun, a number, or a lowercase content word that is rare by both counts (CARRY_MAX_TRANSCRIPT_LINES = 4, CARRY_MAX_CORPUS_DOCS = 5 — filler saturates memory streams while a real detail like R1's "warm wood paneling" appears only in its few echoes). Names skip the frequency gates entirely: the run-wide line count of chris is direct-address noise the current-interlocutor guard already removes, and naming an absent person you met earlier is the R1 evidence no matter how often the pair greeted each other in between.

Acceptance: measured on the batch-1 reference runs

memory_carry, heuristic judge, both runs from the batch-1 artifacts (#783):

agent	memory_carry	note
R1 Sofia Ramirez (the Diego→Sofia→Tanaka carry)	5.5	'diego' — first heard from Diego Torres at steps 276-370 — resurfaces with Professor Tanaka (steps 810+, ~530 steps later, a different partner) — the exact event the issue cites
R2 Aiden Park (the #778 loop)	1.0	0/7 later conversations reuse a detail
R2 Chris Donnelly (the #778 loop)	1.0	0/7 later conversations reuse a detail

R1's carry scores 4.5 points above R2's loop. Corroboration across the rest of the batch:

• R3's Hannah↔Ravi loop (the other Co-located pair locks into a groundhog-day conversation loop: talk_to starves perform, schedule never advances, 63% of budget burned #778 pair) also floors: Hannah 1.0, Ravi 1.8 (one marginal same-partner digit).
• R2 Nina Alvarez scores 10.0, and it is genuine cross-partner carry: she tells Jamal what Grace is working on, one conversation later ('grace' — first heard from Grace Kim at steps 252-976 — resurfaces with Jamal Reed).
• R5 (zero conversations) is n/a across the board.
• Diego Torres (R1) scores 1.0: his known carry ("warm wood paneling") shows up in his retrieved memories, but he paraphrases rather than reuses a token in a transcript — a recall ceiling of a transcript-level heuristic, noted here so it reads as known.

Run-level means with Penn meta.locations injected (the #817 procedure — these bakes predate #780, see #813): dimension mean R1 3.25 / R2 4.00 — R2's edge is Nina's genuine carry, the same healthy-majority effect the #817 spec documents for the overall mean ("run-mean ordering is not the bar"). The loop pair lands where it should within its run: Aiden and Chris stay the two lowest agents in R2 (6.87, 7.00). Overall means move R1 7.97, R2 8.45, R3 7.25, R4 7.90, R5 7.90 — same ordering as #817.

This validation is a reported experiment, not a committed test, because the run artifacts live in godot-generative-agents/runs/, not in a fixture; the committed unit tests pin the mechanics on small synthetic replays (a carried rare detail scores, a looping pair's converged vocabulary doesn't, cross-partner beats same-partner, numbers carry, innate vocabulary and direct address don't, n/a below two conversations).

Files

• backend/eval/believability.py — the dimension, its helpers (_detail_words, _proper_case_words, _carry_frequencies, _is_specific_detail), the CARRY_* constants, registration in DIMENSIONS/score_agent/tool schema
• backend/prompt_templates/believability_rubric.prompty — the sixth rubric bullet ("all six dimensions"), with the pinned render test updated
• backend/prompt_templates/README.md — the usage-table row
• tests/test_believability_eval.py — 8 new tests plus the fixture-high / ledger-ceiling / GRADE updates for a dimension that can be n/a

Full sweep: 2476 passed, 1 failed, 2 skipped — the one failure is tests/test_godot_smoke.py's known fresh-worktree timeout, unrelated.

Closes #814

🤖 Generated with Claude Code