Skip to content

Use 6-char lever ID prefix, positive framing, and exact-count instruction - #460

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-short-lever-prefix-and-exact-count
Closed

Use 6-char lever ID prefix, positive framing, and exact-count instruction#460
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-short-lever-prefix-and-exact-count

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • 6-char prefix: Replace full UUIDs with 6-char prefixes in lever_details_for_prompt (Lever ID: {uuid}Lever {uuid[:6]}). Map prefixes back to full UUIDs when processing the response. Eliminates the source of UUID contamination without losing lever matching.
  • Positive framing: "refer to levers by their name — for example, write 'Policy Advocacy Strategy', not an identifier". Replaces the negative prohibition from Add UUID prohibition and exact-count instruction to enrich prompts #458 which backfired on llama3.1.
  • Exact-count instruction: "Return exactly N characterizations — one per lever, no more, no fewer". Reduced haiku's fabricated entries 71% in Add UUID prohibition and exact-count instruction to enrich prompts #458.
  • OPTIMIZE_INSTRUCTIONS: Document negative-priming lesson and current fix state.

Supersedes #458 (negative prohibition backfired).

Test plan

  • Verify llama3.1 UUID contamination drops from 29 toward 0
  • Verify haiku unknown_lever_id errors drop from 7 toward 0
  • Verify all levers still correctly enriched (prefix→UUID mapping works)
  • Verify no content regressions for other models
  • Check assessment verdict before merging

🤖 Generated with Claude Code

…tion
Replace full UUIDs with 6-char prefixes in lever_details_for_prompt to
prevent models from copying long UUIDs into synergy/conflict text. Map
prefixes back to full UUIDs when processing the response.
Add positive framing ("refer to levers by their name") instead of
negative prohibition which caused regression in llama3.1 (PR #458).
Add exact-count instruction to suppress haiku's fabricated extra entries.
Update OPTIMIZE_INSTRUCTIONS to document the negative-priming lesson.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 62

Verdict: CONDITIONAL

UUID contamination: fully eliminated (32 → 0)

  • llama3.1: 32 → 0 UUID occurrences in synergy/conflict text. Clean name-only references.
  • All other models: remain at 0.
  • 100% success rate (35/35), including gpt-oss-20b 5/5.

Haiku regression: 7 → 43 fabricated-ID errors (+514%)

  • The 6-char prefix is too short/vague for haiku's function-calling JSON generation — it fabricates sequential non-hex strings (d9e2f1, e8f4g2, ...) instead of reading the actual prefix from the prompt.
  • All 35 real levers correctly enriched — the errors are noise, not data loss.
  • Root cause: LeverCharacterization.lever_id field description says "The 6-character identifier" with no anchoring. Haiku pattern-completes it.

Recommended fix: Use full UUIDs for function-calling models (is_function_calling_model=True), keep 6-char prefix for text-completion models. PR #458 demonstrated 2 haiku errors with full UUIDs.

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — 6-char prefix causes haiku regression (7→43 fabricated-ID errors). Will implement model-type-aware identifier: full UUIDs for function-calling models, 6-char prefix for text-completion models.

@neoneyeneoneye closed this Mar 31, 2026
neoneye added a commit that referenced this pull request Mar 31, 2026
Replace full UUIDs with integer indices (1, 2, 3...) in the per-batch
prompt to prevent UUID contamination in free-text fields. Map indices
back to full UUIDs when processing the response. Integer indices work
universally for both text-completion and function-calling models,
unlike 6-char hex prefixes which degraded haiku (PR #460) or negative
prohibitions which backfired on llama3.1 (PR #458).
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-460.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
…ences field
Replace "Do NOT include 'Controls ... vs.', 'Weakness:'" with "Focus
on cause-effect relationships and factual outcomes; save critical
assessments for the review_lever field." in both Lever and LeverCleaned
Pydantic models.
The negative prohibition names exact banned phrases that small models
(llama3.1) copy as templates — the same anti-pattern that caused
regressions in the enrich step (PRs #458, #460). Positive framing
guides the model toward the desired behavior without priming the
unwanted pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Apr 1, 2026
Document proven failure modes and fixes from 10 self_improve iterations:
- Field-description template lock: expanded with evidence from PR #484
(the fix) and PR #479 (replacement phrase also locks). Strip to
content-type + word count, let examples teach.
- Negative prohibitions activate banned patterns: confirmed across
PRs #458, #460, #475. Use positive framing instead.
- Consequence parroting in later calls: llama3.1 copies consequences
into review_lever. Anti-parrot needed in subsequent-call prompt.
- Field descriptions vs system prompt consistency: haiku follows
system prompt, text-completion models follow field descriptions.
- Stripping field descriptions too far: weak models need "one sentence"
and section pointer as structural anchors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye
neoneye deleted the fix/enrich-short-lever-prefix-and-exact-count branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye