Uh oh!
There was an error while loading. Please reload this page.
[core] Thread queue namespace through o11y run actions and bound healthCheck stream reads - #2874
Conversation
…thCheck stream reads Deployments that use a queue namespace (e.g. eve's __eve_wkf_workflow_* topics) could not be targeted by cross-context callers: start(), recreateRunFromExisting(), reenqueueRun(), and wakeUpRun() always built queue names from the caller's WORKFLOW_QUEUE_NAMESPACE env, so dashboard replays published to topics the target deployment has no consumer for. Additionally, healthCheck()'s poll loop only checked its timeout between iterations while world.streams.get() itself was unbounded — against workflow-server, which holds unwritten streams open for ~2 minutes, a 2s capability probe hung until the caller's function timed out (the observed 30s 504s on dashboard replay). - Add a `namespace` option to StartOptionsBase, RecreateRunOptions, StopSleepOptions, and new ReenqueueRunOptions; thread it into getWorkflowQueueName() and the cross-deployment capability probe. - Fold the inline `namespace` param into HealthCheckOptions. - Race streams.get() against the remaining health-check budget. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 28c0a0a The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
|
📊 Workflow Benchmarkscommit Backend:
Avg deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body execution · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (time outside step bodies, client start → last step body exit) · SL: stream latency (first chunk write → visible to the reader) Scenarios — stream: one step that streams chunks back to the client; no hooks, so the run stays in turbo mode · hook + stream: registers a hook before the same streaming step, which exits turbo mode · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges 🟢/🔴 mark percentiles within/above target. Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 TTFS/WO compare client vs deployment clocks and SL compares the step runner’s clock vs the client’s (NTP-synced in CI). WO ends at the last step body exit, the closest observable proxy for the final step-completion request. |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why
Replaying an eve agent run from the dashboard hangs ~30s and then fails with a 504. Two compounding issues in
@workflow/core:__eve_wkf_workflow_*topics (they setWORKFLOW_QUEUE_NAMESPACE=eveat build time), butstart()/recreateRunFromExisting()/reenqueueRun()/wakeUpRun()build queue names from the calling process's env. The dashboard has no namespace set, so its health-check probe and run enqueue publish to unprefixed topics that have no consumer — the message is never dispatched.healthCheck()can hang past its timeout. The poll loop only checks the deadline between iterations;world.streams.get()itself is unbounded. workflow-server holds unwritten streams open for ~2 minutes, so the 2s cross-deployment capability probe instart()hung until the dashboard function's 30s limit killed the request.What
namespaceoption toStartOptionsBase,RecreateRunOptions,StopSleepOptions, and newReenqueueRunOptions; thread it intogetWorkflowQueueName()and the cross-deployment capability probe. Env fallback behavior is unchanged when the option is omitted.namespaceparam intoHealthCheckOptions.streams.get()against the remaining health-check budget so probes fail at the configured deadline instead of hanging.A follow-up in
frontwill pass the target deployment's namespace from the observability replay / wake-up / deployment-health routes. Longer term, persisting the namespace on the run record would remove the need for callers to supply it.Validation
start/wakeUpRun/reenqueueRun, cross-deployment probe topic, and a hang-regression test (never-resolvingstreams.getnow returns unhealthy at the deadline; previously it hung indefinitely).packages/corefull unit suite: 1439 tests pass.🤖 Generated with Claude Code