Skip to content

Comprehensive prompt refinement: verbatim numbers, template-lock fix, tighter targets - #479

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/identify-comprehensive-prompt-refinement
Closed

Comprehensive prompt refinement: verbatim numbers, template-lock fix, tighter targets#479
neoneye wants to merge 1 commit into
mainfrom
fix/identify-comprehensive-prompt-refinement

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

Bundles all proven improvements from analyses 69-73 into one definitive PR:

  1. Verbatim-numbers on consequences AND options: "Use numbers only when the project context provides them directly — do not calculate, derive, or estimate figures." Closes the arithmetic-derivation loophole.
  2. Template-lock fix (top issue from analysis 40): Replace "the three options leave unaddressed" with "the proposed options collectively do not resolve" in review_lever. Applied to field description, section 4, AND section 6.
  3. Positive framing: Replace "Do NOT include 'Controls ... vs.'"
  4. Consistent targets: 2-3 sentences, one-sentence options/review, aligned across all field descriptions and system prompt sections.
  5. Section 5: "NO calculated, derived, or estimated figures — use only numbers that appear verbatim in the project context"

Supersedes #471, #473, #475, #477, #478.

Test plan

  • Verify gpt-oss-20b completes 5/5 plans
  • Verify haiku review template lock drops from 70-94%
  • Verify fabricated numbers ≤ baseline (23)
  • Verify real plan numbers still cited when appropriate
  • Assessment by GPT5.4 confirms convincingly better than analysis 40

🤖 Generated with Claude Code

… tighter targets
Bundles all proven improvements from analyses 69-73 into one PR:
1. Verbatim-numbers constraint on BOTH consequences AND options:
"Use numbers only when the project context provides them directly —
do not calculate, derive, or estimate figures."
Closes the arithmetic-derivation loophole (HK$470M→HK$141M).
2. Fix review_lever template lock (#1 issue from analysis 40):
Replace "state the specific gap the three options leave unaddressed"
with "identify the key risk or constraint that the proposed options
collectively do not resolve." Removes "the three options" as
grammatical anchor that caused 70-94% haiku template lock.
3. Positive framing: replace "Do NOT include 'Controls ... vs.'"
with "Save critical assessments for the review_lever field."
4. Consistent targets everywhere: consequences 2-3 sentences,
options one sentence, review_lever one sentence (20-40 words).
All field descriptions and system prompt sections aligned.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 74

Verdict: CONDITIONAL

Win:

  • gpt-4o-mini template lock: 5/16 → 0/17 (genuine improvement)
  • 100% call success rate (up from 97.1%)

Regressions:

  • qwen3-30b new template lock: 0/15 → 7/20 ("collectively do not resolve" is itself a copyable phrase)
  • gpt-oss-20b fabricated numbers: 0 → 5 (Section 5 abstract prohibition activated number-generation)
  • gpt-oss-20b template lock shifted and increased: ~10/17 → ~15/17

The fundamental problem: replacing one copyable phrase with another. Any sentence-structural phrase in the field description becomes a template for small models. The assessment cites OPTIMIZE_INSTRUCTIONS lines 86-92 — the PR violated its own documented anti-pattern.

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — replaced one template lock with another. See iteration_insights.md in analysis 74.

@neoneyeneoneye closed this Apr 1, 2026
neoneye added a commit that referenced this pull request Apr 1, 2026
Same as PR #479 (verbatim numbers, positive framing, consistent
targets) but with one tweak: strip review_lever field description
to bare minimum "Critical review of this lever (one sentence, 20-40
words)" with no structural phrase that models can copy as a template.
PR #479's "the proposed options collectively do not resolve" caused
qwen3-30b to lock (0→7/20) and worsened gpt-oss-20b (10/17→15/17).
This version removes ALL copyable sentence structures from review_lever
and lets the section 4 examples teach the desired style instead.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
neoneye added a commit that referenced this pull request Apr 1, 2026
Document proven failure modes and fixes from 10 self_improve iterations:
- Field-description template lock: expanded with evidence from PR #484
(the fix) and PR #479 (replacement phrase also locks). Strip to
content-type + word count, let examples teach.
- Negative prohibitions activate banned patterns: confirmed across
PRs #458, #460, #475. Use positive framing instead.
- Consequence parroting in later calls: llama3.1 copies consequences
into review_lever. Anti-parrot needed in subsequent-call prompt.
- Field descriptions vs system prompt consistency: haiku follows
system prompt, text-completion models follow field descriptions.
- Stripping field descriptions too far: weak models need "one sentence"
and section pointer as structural anchors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye
neoneye deleted the fix/identify-comprehensive-prompt-refinement branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye