Skip to content

Allow verbatim plan numbers only, positive framing, and tighter targets - #478

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/identify-allow-verbatim-numbers-only
Closed

Allow verbatim plan numbers only, positive framing, and tighter targets#478
neoneye wants to merge 1 commit into
mainfrom
fix/identify-allow-verbatim-numbers-only

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • Verbatim numbers only: "Use numbers only when the project context provides them directly — do not calculate, derive, or estimate figures." Closes the arithmetic-derivation loophole from Positive framing, consistent word counts, and stronger number-evidence constraint #475 (HK$470M→HK$141M) while still allowing real plan numbers.
  • Positive framing: Replace "Do NOT include 'Controls ... vs.'" with "Save critical assessments for the review_lever field."
  • Tighter targets: consequences 2-3 sentences, options one sentence, review_lever one sentence (20-40 words). All consistent across field descriptions and system prompt.
  • Section 5 prohibition: "NO calculated, derived, or estimated figures — use only numbers that appear verbatim in the project context"

Supersedes #471, #473, #475, #477.

Test plan

🤖 Generated with Claude Code

Replace number constraints with: "Use numbers only when the project
context provides them directly — do not calculate, derive, or estimate
figures." This closes the arithmetic-derivation loophole (HK$470M →
HK$141M) from PR #475 while still allowing models to cite real plan
numbers verbatim.
Replace negative prohibition ("Do NOT include 'Controls ... vs.'")
with positive framing. Tighten targets: consequences 2-3 sentences,
options one sentence, review_lever one sentence (20-40 words). All
sources (field descriptions and system prompt) consistent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 73

Verdict: CONDITIONAL

Wins:

  • Verbatim-numbers constraint works for consequences: 0 fabricated % claims
  • Positive framing in place, no "Controls X vs Y"
  • 97.1% call success rate (same as baseline)
  • Haiku review template lock improved slightly (85%→70%)

Gaps:

  • Numbers still fabricated in options fields (constraint wasn't added to options field description)
  • Template lock still at 70-94% in haiku reviews (root cause: "the three options leave unaddressed" in field description)
  • Haiku consequences still over-long (2.2x baseline)

The verbatim-numbers approach is the right one — just needs to be applied to options too, and the review_lever field description needs the template-lock fix.

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — verbatim-numbers works for consequences but missing from options field. Will bundle with template-lock fix in next PR.

@neoneyeneoneye closed this Apr 1, 2026
neoneye added a commit that referenced this pull request Apr 1, 2026
Replace vague "only cite numbers if the project context provides
evidence for them" with precise "use numbers only when the project
context provides them directly — do not calculate, derive, or estimate
figures" in consequences field description and system prompt.
This single change eliminated fabricated % claims from consequences
in analysis 73 (#478). No other changes — isolating this variable.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye
neoneye deleted the fix/identify-allow-verbatim-numbers-only branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye