Skip to content

Add UUID prohibition and exact-count instruction to enrich prompts - #458

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-uuid-prohibition-and-exact-count
Closed

Add UUID prohibition and exact-count instruction to enrich prompts#458
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-uuid-prohibition-and-exact-count

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • UUID prohibition: Add "Do NOT include any Lever ID, UUID, or identifier string" to system prompt. Targets llama3.1's 15 residual same-batch UUID occurrences (from lever_details_for_prompt)
  • Exact-count instruction: Add "Return exactly N characterizations — one per lever, no more, no fewer" to user prompt. Targets haiku's 7 fabricated extra entries from analysis 60
  • OPTIMIZE_INSTRUCTIONS: Update stale UUID entry to reflect post-PR Strip UUIDs from full lever context string in enrich step #457 state, document haiku's extra-characterization behavior

Test plan

  • Verify llama3.1 UUID contamination drops from 15 toward 0
  • Verify haiku unknown_lever_id errors drop from 7 toward 0
  • Verify all models still enrich all levers (no drops from exact-count constraint)
  • Check assessment verdict before merging

🤖 Generated with Claude Code

…user prompt
Add "Do NOT include any Lever ID, UUID, or identifier string" to
ENRICH_LEVERS_SYSTEM_PROMPT to suppress llama3.1's same-batch UUID
copying (15 residual occurrences from lever_details_for_prompt).
Add "Return exactly N characterizations — one per lever, no more, no
fewer" to the per-batch user prompt to suppress haiku's fabricated
extra LeverCharacterization objects (7 errors in analysis 60).
Update OPTIMIZE_INSTRUCTIONS to reflect post-PR #457 state: document
the per-batch UUID vector and haiku's extra-characterization behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 61

Verdict: CONDITIONAL

Haiku exact-count instruction: improved (–71%)

  • 7 → 2 unknown_lever_id errors. gta_game fully cleaned (5→0). All real levers correctly enriched.

UUID prohibition: backfired

  • The "Do NOT include any Lever ID, UUID" instruction triggered negative-priming in llama3.1 — sovereign_identity went from 0 to 20 UUID occurrences. Net llama3.1 UUIDs: 29 → 29 (zero improvement).
  • This violates the documented best practice in identify_potential_levers.py:80-82: "Do NOT add explicit prohibitions naming banned phrases — small models treat the prohibition text as a template."

Recommendation: Close this PR. Replace the negative prohibition with a post-process regex strip (model-agnostic, guaranteed) + positive framing ("refer to levers by NAME only — for example, write 'Policy Advocacy Strategy'").

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — UUID prohibition backfired (negative-instruction priming). Will use short 6-char lever ID prefixes instead of full UUIDs in batch prompt.

@neoneyeneoneye closed this Mar 31, 2026
neoneye added a commit that referenced this pull request Mar 31, 2026
…tion
Replace full UUIDs with 6-char prefixes in lever_details_for_prompt to
prevent models from copying long UUIDs into synergy/conflict text. Map
prefixes back to full UUIDs when processing the response.
Add positive framing ("refer to levers by their name") instead of
negative prohibition which caused regression in llama3.1 (PR #458).
Add exact-count instruction to suppress haiku's fabricated extra entries.
Update OPTIMIZE_INSTRUCTIONS to document the negative-priming lesson.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
Replace full UUIDs with integer indices (1, 2, 3...) in the per-batch
prompt to prevent UUID contamination in free-text fields. Map indices
back to full UUIDs when processing the response. Integer indices work
universally for both text-completion and function-calling models,
unlike 6-char hex prefixes which degraded haiku (PR #460) or negative
prohibitions which backfired on llama3.1 (PR #458).
Add positive framing ("refer to levers by their name") and exact-count
instruction ("Return exactly N characterizations"). Update
OPTIMIZE_INSTRUCTIONS with lessons from PRs #457-460.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Mar 31, 2026
…ences field
Replace "Do NOT include 'Controls ... vs.', 'Weakness:'" with "Focus
on cause-effect relationships and factual outcomes; save critical
assessments for the review_lever field." in both Lever and LeverCleaned
Pydantic models.
The negative prohibition names exact banned phrases that small models
(llama3.1) copy as templates — the same anti-pattern that caused
regressions in the enrich step (PRs #458, #460). Positive framing
guides the model toward the desired behavior without priming the
unwanted pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Apr 1, 2026
Document proven failure modes and fixes from 10 self_improve iterations:
- Field-description template lock: expanded with evidence from PR #484
(the fix) and PR #479 (replacement phrase also locks). Strip to
content-type + word count, let examples teach.
- Negative prohibitions activate banned patterns: confirmed across
PRs #458, #460, #475. Use positive framing instead.
- Consequence parroting in later calls: llama3.1 copies consequences
into review_lever. Anti-parrot needed in subsequent-call prompt.
- Field descriptions vs system prompt consistency: haiku follows
system prompt, text-completion models follow field descriptions.
- Stripping field descriptions too far: weak models need "one sentence"
and section pointer as structural anchors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye
neoneye deleted the fix/enrich-uuid-prohibition-and-exact-count branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye