Skip to content

Include consequences and review in enrich batch prompt - #451

Merged
neoneye merged 1 commit into
mainfrom
fix/enrich-batch-prompt-context
Mar 29, 2026
Merged

Include consequences and review in enrich batch prompt#451
neoneye merged 1 commit into
mainfrom
fix/enrich-batch-prompt-context

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • The enrich step's batch prompt only provided lever_id, name, and options to the LLM, but omitted consequences and review — the two fields that document the lever's effects and primary trade-off.
  • This forced the model to infer descriptions and conflict analysis from the name alone, producing vague or ungrounded enrichment.
  • Adds both fields to the per-lever details in the batch prompt so the LLM can write grounded description, synergy_text, and conflict_text.

Identified by analysis/53_enrich_potential_levers synthesis as the #1 ranked improvement direction (B3 + I4).

Test plan

  • Run enrich_potential_levers experiment with this PR across all models
  • Compare assessment against analysis 53 baseline — expect more specific descriptions grounded in lever consequences/trade-offs
  • Verify no increase in validation failures (change is input-only, output schema unchanged)

🤖 Generated with Claude Code

The enrich step asks the LLM to generate description and conflict_text
for each lever, but the batch prompt only provided lever_id, name, and
options. The consequences (what happens when the lever is pulled) and
review (the documented primary trade-off) fields were loaded but never
forwarded to the prompt, forcing the model to infer them from the name
alone.
This adds both fields to the per-lever details in the batch prompt so
the LLM can produce grounded descriptions and conflict analysis.
Identified by analysis/53_enrich_potential_levers synthesis (direction #1).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-Improve Iteration Result: YES (keep)

Experiment ran as analysis/54_enrich_potential_levers (runs 85–91, 7 models × 5 plans).

Improvements

  • gpt-5-nano template artifact eliminated — "Purpose: / Objectives: / Key success metrics:" sub-header structure gone from all 5 plans
  • Conflict texts now grounded — multiple models echo review language in conflict_text instead of fabricating from lever name alone
  • qwen3-30b synergy/conflict improved — from 0.58–0.69× to 0.74–0.76× baseline

Regressions (minor)

  • llama3.1 description brevity — dropped from 0.80× to 0.65× baseline (echoes consequences verbatim instead of elaborating)
  • qwen3-30b UUID format nondeterministic — pre-existing B2 bug surfaced more visibly
  • Single CJK character leak in qwen3-30b (one occurrence, not systematic)

Unchanged

  • Success rate: 30/35 (gpt-oss-20b still 0/5 — separate B1 fix needed)
  • Input token cost: +18–20% (expected trade-off)

Full assessment: analysis/54_enrich_potential_levers/assessment.md

@neoneye
neoneye merged commit 2afb498 into mainMar 29, 2026
3 checks passed
@neoneye
neoneye deleted the fix/enrich-batch-prompt-context branch March 29, 2026 22:59
neoneye added a commit that referenced this pull request Mar 29, 2026
B1: The enrich step previously aborted the entire plan on any batch
failure. Now on failure it splits the batch in half and retries each
sub-batch. If a single-lever batch still fails, it logs and skips
that lever rather than discarding all successfully enriched levers.
This should recover gpt-oss-20b plans that fail on large batches
hitting the output token limit.
B4: runner.py now reports the actual batches_succeeded count from
EnrichPotentialLevers instead of hardcoding 1.
D5: Added two new known problems to OPTIMIZE_INSTRUCTIONS:
- Consequence echoing without elaboration (llama3.1 post-PR #451)
- UUID cross-reference format inconsistency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye