Skip to content

Tighten number constraint in consequences to verbatim-only - #481

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/identify-verbatim-numbers-consequences-only
Closed

Tighten number constraint in consequences to verbatim-only#481
neoneye wants to merge 1 commit into
mainfrom
fix/identify-verbatim-numbers-consequences-only

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

One change, three locations: replace "only cite numbers if the project context provides evidence for them" with "use numbers only when the project context provides them directly — do not calculate, derive, or estimate figures" in the consequences field description (Lever + LeverCleaned) and system prompt section 2.

This was the single highest-impact change across analyses 69-74. In #478, this same wording eliminated fabricated % claims from consequences (haiku 20→0). Everything else is left unchanged from the baseline (PR #358) to isolate this one variable.

Test plan

  • Verify fabricated % claims in consequences reduced
  • Verify real plan numbers still cited when appropriate
  • Verify no regressions in any other metric
  • Assessment by GPT5.4 confirms better than analysis 40

🤖 Generated with Claude Code

Replace vague "only cite numbers if the project context provides
evidence for them" with precise "use numbers only when the project
context provides them directly — do not calculate, derive, or estimate
figures" in consequences field description and system prompt.
This single change eliminated fabricated % claims from consequences
in analysis 73 (#478). No other changes — isolating this variable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 75

Verdict: CONDITIONAL

  • Verbatim-numbers constraint works for consequences (1 residual via "typically" loophole)
  • But consequences wasn't the main fabrication site — options has 5+ fabricated figures per plan (unprotected)
  • Template lock unchanged at ~95% in haiku reviews
  • 98.1% call success rate (+1pp)
  • No regressions, but marginal improvement — the change tightens the right mechanism but on the wrong field

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — marginal improvement, consequences wasn't the main fabrication site. May build on this tomorrow.

@neoneyeneoneye closed this Apr 1, 2026
@neoneye
neoneye deleted the fix/identify-verbatim-numbers-consequences-only branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye