Skip to content

Adaptive batch size, guarded retry, and gpt-oss-20b config fix for enrich step - #455

Closed
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-adaptive-batch-and-retry-guards-v3
Closed

Adaptive batch size, guarded retry, and gpt-oss-20b config fix for enrich step#455
neoneye wants to merge 1 commit into
mainfrom
fix/enrich-adaptive-batch-and-retry-guards-v3

Conversation

@neoneye

Copy link
Copy Markdown
Member

Summary

  • gpt-oss-20b config fix: max_tokens 8192→65536 (balanced: 65536 input headroom), add explicit context_window=131072. Previous attempts: 128000 left only 3072 input tokens causing BadRequestError
  • Adaptive batch size: batch_size=2 when context_window < 6000 (probes model metadata, no API call)
  • Guarded retry: On batch failure, split once (depth=1) within 300s budget, then skip
  • Error tracking: Persist errors in raw JSON output for offline diagnosis
  • Accurate batch counting: batches_succeeded instead of hardcoded 1
  • OPTIMIZE_INSTRUCTIONS: Document consequence-echoing, UUID format inconsistency, and max_tokens overflow

Supersedes #453 (num_output bug) and #454 (max_tokens=128000 too high).

Test plan

  • Verify gpt-oss-20b uses batch_size=5 (context_window=131072 > 6000 threshold)
  • Verify gpt-oss-20b achieves 5/5 plan success (was 0/5 at baseline)
  • Verify no BadRequestError in error arrays
  • Verify all other models remain at 100% success rate
  • Check assessment verdict before merging

🤖 Generated with Claude Code

…rich step
Fix gpt-oss-20b config: max_tokens 8192→65536, add context_window=131072.
The old 8192 truncated JSON output; previous attempts at 128000 left only
3072 input tokens causing BadRequestError. 65536 gives balanced headroom.
Add adaptive batch sizing based on context_window (batch_size=2 when
context_window < 6000). Add guarded retry: split once (MAX_RETRY_DEPTH=1)
within 300s budget, then skip. Persist errors in raw JSON output for
offline diagnosis. Report actual batches_succeeded instead of hardcoded 1.
Document consequence-echoing, UUID format inconsistency, and max_tokens
overflow in OPTIMIZE_INSTRUCTIONS.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@neoneye

Copy link
Copy Markdown
MemberAuthor

Self-improve iteration — analysis 58

Verdict: CONDITIONAL

Wins:

  • gpt-oss-20b config fix works: all 5 plans produce valid enriched lever output (was 0/5 before)
  • Error tracking works: llama3.1 UUID mutations now visible (4 errors in run 13)
  • OPTIMIZE_INSTRUCTIONS: 3 new entries documented

Regression — adaptive batch_size=2 fires for wrong models:
LlamaIndex's OpenRouter class reports context_window=3900 universally for ALL OpenRouter models. Since 3900 < SMALL_CONTEXT_THRESHOLD(6000), qwen3-30b, gpt-4o-mini, and gemini-flash all get batch_size=2 (3-4 API calls/plan instead of 1-2). The adaptive feature now protects nothing and penalizes 3 unrelated models.

gpt-oss-20b: 5/5 valid output, but 3/5 plan timeout (600s)
The enrichment completes and writes correct output, but LLMExecutor's provider failover (6-9 attempts) exhausts the 600s plan budget. Different failure mode than before — the data is there, just the status is wrong.

Fix needed: Lower SMALL_CONTEXT_THRESHOLD from 6000 to 3000 (OpenRouter's 3900 falls above threshold)

@neoneye

Copy link
Copy Markdown
MemberAuthor

Closing — adaptive batch_size=2 incorrectly triggers for all OpenRouter models (context_window=3900 fallback). Will rework with SMALL_CONTEXT_THRESHOLD=3000 and explicit context_window for OpenRouter models.

@neoneyeneoneye closed this Mar 30, 2026
@neoneye
neoneye deleted the fix/enrich-adaptive-batch-and-retry-guards-v3 branch April 1, 2026 23:06
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@neoneye