Skip to content

Move UUID to end of lever details, add positive framing and exact-count - #464

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-move-uuid-to-end
Closed

Move UUID to end of lever details, add positive framing and exact-count#464
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-move-uuid-to-end

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • Move UUID to end: Change Lever ID: {uuid} (first line) to (internal reference: {uuid}) (last line). Models tend to copy identifiers from the beginning of text blocks. Full UUID kept for reliable matching — no mapping code changes needed.
  • Positive framing: "refer to levers by their name — for example, write 'Policy Advocacy Strategy', not an identifier"
  • Exact-count instruction: "Return exactly N characterizations — one per lever, no more, no fewer"
  • OPTIMIZE_INSTRUCTIONS: Document lessons from PRs Strip UUIDs from full lever context string in enrich step #457-462 (negative prohibitions backfire, short prefixes/integers break matching, full UUIDs at end of block is the sweet spot)

Test plan

  • Verify llama3.1 UUID contamination drops from 15 (analysis 60 baseline)
  • Verify all levers still correctly enriched (UUID matching unchanged)
  • Verify haiku errors drop from 7 (analysis 60 baseline)
  • Verify no content regressions
  • Check assessment verdict before merging

🤖 Generated with Claude Code

Move the lever UUID from prominent first line (Lever ID: {uuid}) to an
inconspicuous position at the end: (internal reference: {uuid}). Models
tend to copy identifiers from the beginning of a block; placing it last
reduces UUID leakage into synergy/conflict text while keeping the full
UUID for reliable structured-output matching.
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-462.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 64

Verdict: NO

Moving UUID to end of lever block broke structured-output matching for small models. They use positional heuristics — the first item in a block becomes the lever_id value.

  • llama3.1: 35/35 → 0/35 levers (returned lever names as lever_id)
  • gpt-5-nano: 35/35 → 10/35 levers
  • Total: 245 → 181 characterized levers (–26%)
  • 152 errors (was 7)

UUID position matters for matching. Must stay as the first item in each lever block.

@neoneyeneoneye closed this Mar 31, 2026
@neoneye
neoneye deleted the fix/enrich-move-uuid-to-end branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye