Skip to content

Add per-batch retry for enrich step, update OPTIMIZE_INSTRUCTIONS - #452

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-batch-retry-and-optimize-instructions
Closed

Add per-batch retry for enrich step, update OPTIMIZE_INSTRUCTIONS#452
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-batch-retry-and-optimize-instructions

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • B1: Enrich step now retries failed batches by splitting in half instead of aborting the entire plan. Single-lever failures are logged and skipped. Should recover gpt-oss-20b's 0/5 failure caused by output token limits on large batches.
  • B4: runner.py reports actual batches_succeeded count instead of hardcoding 1.
  • D5: Added two new known problems to OPTIMIZE_INSTRUCTIONS: consequence echoing without elaboration (llama3.1 post-PR Include consequences and review in enrich batch prompt #451) and UUID cross-reference format inconsistency.

Identified by analysis/54_enrich_potential_levers assessment backlog.

Test plan

  • Run enrich_potential_levers experiment with this PR across all models
  • Verify gpt-oss-20b recovers plans via batch splitting (was 0/5 in analysis 53 and 54)
  • Verify no regressions for models that already succeed (6/6 models at 100%)
  • Check calls_succeeded in output metadata reflects actual batch count

🤖 Generated with Claude Code

B1: The enrich step previously aborted the entire plan on any batch
failure. Now on failure it splits the batch in half and retries each
sub-batch. If a single-lever batch still fails, it logs and skips
that lever rather than discarding all successfully enriched levers.
This should recover gpt-oss-20b plans that fail on large batches
hitting the output token limit.
B4: runner.py now reports the actual batches_succeeded count from
EnrichPotentialLevers instead of hardcoding 1.
D5: Added two new known problems to OPTIMIZE_INSTRUCTIONS:
- Consequence echoing without elaboration (llama3.1 post-PR #451)
- UUID cross-reference format inconsistency
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-Improve Iteration Result: CONDITIONAL

Experiment ran as analysis/55_enrich_potential_levers (runs 92–98, 7 models × 5 plans).

Improvements

  • gpt-oss-20b recovered 2/5 plans (was 0/5) — batch splitting works, confirmed via usage_metrics (8192 token overflow triggered split into two sub-batches)
  • calls_succeeded now accurate — reports 3–4 per plan instead of hardcoded 1
  • No regressions for the 6 working models (all still 100%)

Issues

  • 3 gpt-oss-20b plans timeout at 600s instead of failing fast (~30s before) — retry cascade with no depth limit wastes ~1800s per run
  • Root cause unaddressed: BATCH_SIZE=5 is too large for gpt-oss-20b's context_window=3900

Required follow-ups

  1. Adaptive BATCH_SIZE based on context_window (would likely recover all 5 plans)
  2. Retry depth limit (max 1 split) + remaining-time guard
  3. UUID fix at line 168 (full_lever_context_str still exposes UUIDs)

Full assessment: analysis/55_enrich_potential_levers/assessment.md

@neoneye

Copy link
Copy Markdown
MemberAuthor

not good enough, I'm closing it.

@neoneyeneoneye closed this Mar 30, 2026
@neoneye
neoneye deleted the fix/enrich-batch-retry-and-optimize-instructions branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye