Skip to content

Strip UUIDs from full lever context string in enrich step - #457

Merged
neoneye merged 1 commit into
mainfrom
fix/enrich-strip-uuid-from-lever-context
Mar 30, 2026
Merged

Strip UUIDs from full lever context string in enrich step#457
neoneye merged 1 commit into
mainfrom
fix/enrich-strip-uuid-from-lever-context

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • Remove lever_id UUIDs from the full-list context string provided to the LLM in each batch
  • Models were copying UUIDs into synergy_text and conflict_text (llama3.1 89%, gemini 34%, gpt-5-nano 8-char truncated)
  • The per-batch prompt still includes lever_id in lever_details_for_prompt so the LLM can return structured characterizations by ID
  • One-line change: f"- {lever.lever_id}: {lever.name}"f"- {lever.name}"

Documented in OPTIMIZE_INSTRUCTIONS since analysis 54. Highest priority content quality fix from analysis 59 backlog.

Test plan

  • Verify llama3.1 UUID contamination drops from 89% toward <5%
  • Verify gemini UUID contamination drops from 34% toward 0%
  • Verify lever enrichment still succeeds (lever_id still in per-batch prompt)
  • Verify no increase in phantom lever names
  • Check assessment verdict before merging

🤖 Generated with Claude Code

Remove lever_id UUIDs from the full-list context provided to the LLM
in each batch. Models were copying UUIDs into synergy_text and
conflict_text (llama3.1 89%, gemini 34%). The per-batch prompt still
includes lever_id so the LLM can return structured characterizations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 60

Verdict: CONDITIONAL

Wins:

  • UUID contamination: 105 → 15 occurrences (–86%)
  • gemini: 30 → 0 UUIDs (fully cleaned)
  • gpt-oss-20b: 42 → 0 UUIDs (fully cleaned for completed plans)
  • llama3.1 phantom lever IDs: fully resolved (3 → 0 errors, 35/35 levers enriched)
  • Content quality improved: clean lever-name references in gemini synergy/conflict text

Residual:

  • llama3.1: 15 UUID occurrences remain (same-batch, from lever_details_for_prompt)
  • haiku: 7 new unknown_lever_id errors (fabricated extra entries, all real levers correctly enriched — benign noise)
  • gpt-oss-20b: 2/5 success (network noise, not PR-related)

Recommended follow-up: Add UUID prohibition to system prompt + exact-count instruction to user prompt

@neoneye
neoneye merged commit 8891aa3 into mainMar 30, 2026
3 checks passed
neoneye added a commit that referenced this pull request Mar 30, 2026
…user prompt
Add "Do NOT include any Lever ID, UUID, or identifier string" to
ENRICH_LEVERS_SYSTEM_PROMPT to suppress llama3.1's same-batch UUID
copying (15 residual occurrences from lever_details_for_prompt).
Add "Return exactly N characterizations — one per lever, no more, no
fewer" to the per-batch user prompt to suppress haiku's fabricated
extra LeverCharacterization objects (7 errors in analysis 60).
Update OPTIMIZE_INSTRUCTIONS to reflect post-PR #457 state: document
the per-batch UUID vector and haiku's extra-characterization behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
Replace full UUIDs with integer indices (1, 2, 3...) in the per-batch
prompt to prevent UUID contamination in free-text fields. Map indices
back to full UUIDs when processing the response. Integer indices work
universally for both text-completion and function-calling models,
unlike 6-char hex prefixes which degraded haiku (PR #460) or negative
prohibitions which backfired on llama3.1 (PR #458).
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-460.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
Move the lever UUID from prominent first line (Lever ID: {uuid}) to an
inconspicuous position at the end: (internal reference: {uuid}). Models
tend to copy identifiers from the beginning of a block; placing it last
reduces UUID leakage into synergy/conflict text while keeping the full
UUID for reliable structured-output matching.
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-462.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
Replace "Lever ID: {uuid}" with "<lever>{uuid}</lever>" in the
per-batch prompt. XML tags signal structural markup rather than text
content, so models are less likely to copy UUIDs into synergy/conflict
fields. The full UUID stays first in each lever block for reliable
positional matching.
Add positive framing ("refer to levers by their name") and exact-count
instruction. Update OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-464.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye
neoneye deleted the fix/enrich-strip-uuid-from-lever-context branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye