Skip to content

Use integer lever indices to eliminate UUID contamination in enrich step - #462

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-integer-lever-id
Closed

Use integer lever indices to eliminate UUID contamination in enrich step#462
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-integer-lever-id

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • Integer indices: Replace Lever ID: {uuid} with Lever 1, Lever 2, etc. in per-batch prompt. Map indices back to full UUIDs when processing the response. Works universally for both text-completion and function-calling models.
  • Positive framing: "refer to levers by their name" in system prompt
  • Exact-count instruction: "Return exactly N characterizations — one per lever, no more, no fewer"
  • OPTIMIZE_INSTRUCTIONS: Document lessons from PRs Strip UUIDs from full lever context string in enrich step #457-460 (negative prohibitions backfire, hex prefixes degrade function-calling models, integer indices work universally)

Supersedes #458 (negative prohibition backfired on llama3.1) and #460 (6-char prefix degraded haiku).

Test plan

  • Verify llama3.1 UUID contamination at 0 (was 32 before PR Strip UUIDs from full lever context string in enrich step #457)
  • Verify haiku unknown_lever_id errors near 0 (was 43 with 6-char prefix, 2 with full UUIDs)
  • Verify all levers correctly enriched (integer→UUID mapping works)
  • Verify no content regressions
  • Check assessment verdict before merging

🤖 Generated with Claude Code

Replace full UUIDs with integer indices (1, 2, 3...) in the per-batch
prompt to prevent UUID contamination in free-text fields. Map indices
back to full UUIDs when processing the response. Integer indices work
universally for both text-completion and function-calling models,
unlike 6-char hex prefixes which degraded haiku (PR #460) or negative
prohibitions which backfired on llama3.1 (PR #458).
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-460.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 63

Verdict: NO

UUID contamination: eliminated (0 across all models) — the integer index approach works for preventing UUID leakage.

Bug: prompt/mapping key mismatch causes catastrophic failures. Prompt shows Lever 1 but mapping key is "1". Models that echo the full label ("Lever 1") fail the lookup. Results:

  • llama3.1: 0/35 levers enriched (was 35/35)
  • gpt-oss-20b: 23/35 (was 35/35)
  • gpt-5-nano: 22/35 (was 35/35)
  • 152 total errors (was 7)

The design is sound but the implementation has a fixable bug. Need to normalize the returned lever_id (strip "Lever " prefix) before lookup.

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — prompt/mapping key mismatch (Lever 1 vs 1) causes 3 models to fail. Will fix normalization in next PR.

@neoneyeneoneye closed this Mar 31, 2026
@neoneye
neoneye deleted the fix/enrich-integer-lever-id branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye