Uh oh!
There was an error while loading. Please reload this page.
perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) - #3088
Conversation
…indow default 0) Production producer-rate data (24h of client flush spans): most agents average 1.03-1.21 chunks per flush with 87-98% single-chunk flushes and >70% of chunks arriving more than 10ms after the previous request had already settled — a fixed 10ms leading window batches almost nothing for them while adding ~20% to isolated-chunk publish latency (~50ms median RTT). The one bursty producer (avg ~4-8 chunks/flush) gets its batching from in-flight accumulation, which does not depend on the window at all. The leading chunk of an idle sink now dispatches immediately by default (window 0): first chunk goes out at once, chunks arriving during its request coalesce into the next group, and each settle dispatches the accumulated group immediately — path-independent batching with no fixed tax on slow producers. A positive WORKFLOW_STREAM_FLUSH_INTERVAL_MS (or world.streamFlushIntervalMs, applying from the second group) opts into a windowed leading edge for slow-but-steady producers that prefer larger groups over first-chunk latency. Early-ack, the durability drain barrier, wire caps, and backpressure bounds are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 329dedd The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Uh oh!
There was an error while loading. Please reload this page.
| * Resolution note: this World option is read lazily on the first | ||
| * dispatch, so it takes effect from the second idle group onward; the | ||
| * `WORKFLOW_STREAM_FLUSH_INTERVAL_MS` environment variable governs from | ||
| * the very first chunk. |
There was a problem hiding this comment.
This is a confusing semantic. Can we change this so that WORKFLOW_STREAM_FLUSH_INTERVAL_MS generally overwrites streamFlushIntervalMs, and otherwise streamFlushIntervalMs is used for both leading edge and successive groups?
There was a problem hiding this comment.
Done in 329dedd. WORKFLOW_STREAM_FLUSH_INTERVAL_MS, when set, now overrides streamFlushIntervalMs; otherwise the World option governs both the leading edge and successive groups — including the very first chunk. The old second-group quirk existed because the option was read lazily on first dispatch; the window decision now waits for the world when needed, which adds no latency since sendGroup awaits the same promise before any request can leave. Added tests for world-option-delays-the-leading-chunk and env-overrides-world, and updated the docs to match.
Co-authored-by: Peter Wielander <mittgfu@gmail.com> Signed-off-by: Karthik Kalyan <105607645+karthikscale3@users.noreply.github.com>
…overns the leading edge too WORKFLOW_STREAM_FLUSH_INTERVAL_MS, when set, now takes precedence over world.streamFlushIntervalMs; otherwise the world option applies from the very first chunk (no more second-group lazy quirk). Deciding waits for the world when needed, which adds no latency: sendGroup awaits the same promise before any request can leave. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
No backport to This change explicitly builds on the group-commit stream batching rework from #3078 ( To override, re-run the Backport to stable workflow manually via |
…ry-2 * origin/main: (292 commits) feat(core): seal forwarded stream writes to the owner's public key (#3098) feat(core): seal hook payloads to the target run's public key (#3096) [e2e] Rebuild the event-log corruption repro around step-count divergence (#3147) feat: decrypt sealed payloads in the dashboard and CLI (#3146) Prewarm only appended replay payloads (#3131) feat: publish each run's X25519 public key on the run entity (#3095) feat(core): route sealed envelopes through the serialization layer (#3094) docs: redirect retired migration-guides URLs to comparisons (#3127) feat(core): add `encp` sealed-box encryption primitive (#3093) chore(core): clarify runtime comments (#3111) Remove obsolete world factory aliases (#3112) feat(core): deterministic sandbox hardening (#3045) Remove retired v1 step route plumbing (#3061) [core] Don't count racing invocations' duplicate step_started events toward the maxRetries ceiling (#3069) [world-testing] Isolate each spawned test server's data directory (#3055) fix: upgrade postcss to >=8.5.18 to address GHSA-r28c-9q8g-f849 (#3102) [next] Respect .gitignore in dev watcher to avoid EMFILE on large monorepos (#3085) [ci] Backport only stability fixes to `stable`, default to claude-opus-5 (#3092) perf(core): immediate leading-edge dispatch for idle streams (flush window default 0) (#3088) Optimize `processImportSpecifier` by computing `shouldFollowImportsFromFile` once per file (#3052) ... # Conflicts: # docs/components/geistdocs/desktop-menu.tsx # docs/components/geistdocs/mobile-menu.tsx # docs/content/docs/v5/cookbook/advanced/child-workflows.mdx # docs/content/docs/v5/cookbook/advanced/upgrading-workflows.mdx # docs/content/docs/v5/cookbook/agent-patterns/agent-cancellation.mdx # docs/content/docs/v5/cookbook/agent-patterns/durable-agent.mdx # docs/content/docs/v5/cookbook/agent-patterns/human-in-the-loop.mdx # docs/content/docs/v5/cookbook/common-patterns/batching.mdx # docs/content/docs/v5/cookbook/common-patterns/idempotency.mdx # docs/content/docs/v5/cookbook/common-patterns/rate-limiting.mdx # docs/content/docs/v5/cookbook/common-patterns/saga.mdx # docs/content/docs/v5/cookbook/common-patterns/scheduling.mdx # docs/content/docs/v5/cookbook/common-patterns/sequential-and-parallel.mdx # docs/content/docs/v5/cookbook/common-patterns/timeouts.mdx # docs/content/docs/v5/cookbook/common-patterns/webhooks.mdx # docs/content/docs/v5/cookbook/common-patterns/workflow-composition.mdx # docs/content/docs/v5/cookbook/index.mdx # docs/content/docs/v5/cookbook/integrations/ai-sdk.mdx # docs/content/docs/v5/cookbook/integrations/chat-sdk.mdx # docs/content/docs/v5/cookbook/integrations/sandbox.mdx # docs/next.config.ts # docs/proxy.ts # docs/scripts/lint.ts # pnpm-lock.yaml # pnpm-workspace.yaml
What & why
The leading chunk of an idle stream now dispatches immediately (flush window default
0, was 10ms). Production data shows most producers emit ~1.2 chunks/flush with >70% of chunks arriving more than 10ms after the previous request settled — the fixed window batched almost nothing for them while adding ~20% to isolated-chunk publish latency (~50ms median RTT). Fast producers lose nothing: their batching comes from in-flight accumulation (chunks arriving during a request form the next group), which is window-independent. A positiveWORKFLOW_STREAM_FLUSH_INTERVAL_MS(orworld.streamFlushIntervalMs, effective from the second idle group) opts back into a windowed leading edge. Builds on #3078 (merged); early-ack, drain durability, wire caps, and backpressure unchanged.Benchmark (real local server, same instrumented traffic per variant)
1+29) / 54ms / 221msSparse (the dominant production shape) improves
35%; bursts trade one extra request (+20ms total) for ~2× faster first chunk; steady throughput identical. All streams verified byte-exact server-side.🤖 Generated with Claude Code