Skip to content

Adaptive batch size, guarded retry, and max_tokens bump for enrich step - #454

Closed
neoneye wants to merge 2 commits into
mainfrom
fix/enrich-adaptive-batch-and-retry-guards-v2
Closed

Adaptive batch size, guarded retry, and max_tokens bump for enrich step#454
neoneye wants to merge 2 commits into
mainfrom
fix/enrich-adaptive-batch-and-retry-guards-v2

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • max_tokens bump: gpt-oss-20b from 8192 → 128000 in baseline.json — the old limit truncated structured JSON output
  • Adaptive batch size: Probes model context_window metadata (not num_output which just reflects max_tokens config); uses batch_size=2 when context_window < 6000. gpt-oss-20b has context_window=3900 so it gets batch_size=2
  • Guarded retry: On batch failure, split once (depth limit=1) within a 300s time budget, then skip. Prevents unbounded retry cascades
  • Accurate batch counting: Report actual batches_succeeded instead of hardcoded 1 in runner.py
  • OPTIMIZE_INSTRUCTIONS: Document consequence-echoing and UUID format inconsistency as known problems

Supersedes #453 which had a bug where num_output (inflated by the max_tokens bump) disabled the adaptive batch size for gpt-oss-20b.

Test plan

  • Run self_improve experiment across 7 models
  • Verify gpt-oss-20b uses batch_size=2 (check logs for "Adaptive batch_size=2")
  • Verify gpt-oss-20b success rate improves (was 0/5 at baseline)
  • Verify all other models remain at 100% success rate
  • Check assessment verdict before merging

🤖 Generated with Claude Code

neoneyeand others added 2 commits March 30, 2026 14:25
Bump gpt-oss-20b max_tokens from 8192 to 128000 to match its 131K context
window. Add adaptive batch sizing based on context_window (not num_output,
which just reflects max_tokens config) — batch_size=2 when context_window
< 6000. Replace raise-on-error batch loop with guarded retry: split once
(MAX_RETRY_DEPTH=1) within a 300s budget, then skip. Report actual
batches_succeeded instead of hardcoded 1. Document consequence-echoing and
UUID format inconsistency in OPTIMIZE_INSTRUCTIONS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collect batch retries, skipped batches, unknown lever IDs, validation
errors, and incomplete levers into an errors list persisted alongside
metadata and characterized_levers in the raw output file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 57

Verdict: CONDITIONAL

Wins:

  • Adaptive batch size works correctly: gpt-oss-20b gets batch_size=2 (context_window=3900 < 6000)
  • Guarded retry works: failing plans abort in <2s (300× faster than pre-PR 600s timeouts)
  • Accurate batch counting: calls_succeeded now reports actual counts
  • Error tracking: errors array persisted in raw JSON for offline diagnosis

Regression — max_tokens too high:
max_tokens=128000 leaves only 131072 − 128000 = 3072 tokens for input. Each lever needs ~3900 input tokens → every batch fails with BadRequestError. 3/5 gpt-oss-20b plans silently produce zero levers while reporting status: "ok".

gpt-oss-20b: 2/5 functional success (same as before, but failure mode changed from timeout → silent empty output)

Fix needed: Reduce max_tokens to ≤65536 and add "context_window": 131072 to gpt-oss-20b config

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — max_tokens=128000 leaves only 3072 input tokens, causing BadRequestError for 3/5 gpt-oss-20b plans. Will rework with max_tokens=65536 and explicit context_window=131072.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye