Uh oh!
There was an error while loading. Please reload this page.
feat(supervisor): add prometheus metric for outbound http requests - #4350
Conversation
|
WalkthroughThe supervisor client now exposes HTTP request completion metrics and routes its HTTP methods through a shared reporting helper. The supervisor application registers labeled Prometheus counter and duration metrics, wires request completion callbacks to update them, and records warm-start HTTP, response-validation, success, and network outcomes. A server-change entry documents the added outbound request observability. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
@trigger.dev/buildtrigger.dev@trigger.dev/core@trigger.dev/python@trigger.dev/react-hooks@trigger.dev/redis-worker@trigger.dev/rsc@trigger.dev/schema-to-json@trigger.dev/sdkcommit: |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
## Summary 2 new features, 9 improvements, 3 bug fixes. ## Highlights - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](#4387)) ## Improvements - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](#4348)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Favorite any dashboard page to a new Favorites section in the side menu, and customize the sidebar by renaming favorites, hiding items, and reordering items and sections. ([#4375](#4375)) - List API endpoints now clamp the page size to a maximum of 100. Requests asking for a larger page size return up to 100 items and keep paginating, rather than pulling an unbounded page. ([#4360](#4360)) - Organizations without billing alerts now get default spend alert thresholds, so you're notified before usage grows unexpectedly. The billing limit page no longer pre-selects an option before you've set a limit and prompts you to configure one. Alert previews now update immediately after you change your billing limit. ([#4328](#4328)) - When you create a Personal Access Token, the generated token now shows its first and last few characters instead of being fully hidden, so you can confirm you copied the right value. ([#4363](#4363)) - Add metrics to the realtime backend that measure how often a single changed run is served to multiple subscriptions in one batch. ([#4341](#4341)) - Realtime run subscriptions can now be configured to read run data straight from the primary database, so a run's latest state is never served from a lagging replica. Off by default; replica reads are unchanged unless you turn it on. ([#4378](#4378)) - SSO and Directory Sync are no longer restricted to Enterprise plans — get in touch and we can turn them on for your organization whatever plan you're on. ([#4393](#4393)) - Improved supervisor observability: it now reports metrics for its outbound requests, making failed calls to upstream services easier to monitor. ([#4350](#4350)) - The runs list on a task's page now updates live — run statuses change and newly triggered runs appear without a manual refresh, matching the main Runs page. ([#4377](#4377)) - Speed up the Batches list page for environments with a large number of batches, which could previously time out while loading. ([#4361](#4361)) - Container startup no longer prints database and ClickHouse connection strings (with credentials) to the logs. ([#4346](#4346)) - The tasks page no longer runs two queries whose results were never displayed, cutting wasted work on every page load and removing a source of hidden server errors ([#4380](#4380)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## trigger.dev@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` - `@trigger.dev/schema-to-json@4.5.8` ## @trigger.dev/core@4.5.8 ### Patch Changes - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](#4387)) ## @trigger.dev/python@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.8` - `@trigger.dev/core@4.5.8` - `@trigger.dev/build@4.5.8` ## @trigger.dev/react-hooks@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/redis-worker@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/rsc@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/schema-to-json@4.5.8 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.8` ## @trigger.dev/sdk@4.5.8 ### Patch Changes - Preserve the partial assistant message when a chat turn's model stream fails mid-response. `chat.agent` now passes the recovered partial to `onTurnComplete`, and `chat.createSession`'s `turn.complete()` keeps it before rethrowing, instead of dropping the streamed-so-far output. ([#4348](#4348)) - Allow additional environment API keys to create scoped public access tokens through the Trigger.dev API. Use server-issued public access tokens for batch operations so environment-scoped API keys can read batch results. ([#4387](#4387)) - Updated dependencies: - `@trigger.dev/core@4.5.8` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Adds Prometheus metrics so the supervisor's outbound HTTP calls are observable - including client-side failures that previously only surfaced as a log line.
supervisor_outbound_request_total{name, method, status, outcome}- counts every outbound request.outcomeseparates a transport failure (network_error), an HTTP error response (http_error), a response that failed schema validation (invalid_response), and success (ok).supervisor_outbound_request_duration_seconds{name, outcome}- latency histogram. Leaner labels than the counter (nostatus) to avoid bucket×label cardinality; buckets match the existing dequeue-latency histogram since these calls share the same retrying HTTP client and long-poll envelope.Coverage:
fetch) - instrumented inline; the response status code is now also included in the failure log (it was previously dropped).SupervisorHttpClient: dequeue, run attempt start/complete, heartbeats, snapshots, continue, suspend, debug-log, connect) - routed through a single instrumentedrequest()helper that reports via an optionalonHttpRequestCompletecallback on the client, which the supervisor wires into the counter + histogram.Low cardinality by design:
nameis a static per-endpoint label (e.g.dequeue,start_run_attempt), never the interpolated URL - so no run/snapshot IDs land in labels, mirroring the templatedroutelabels on the inbound HTTP server.Registered on the existing metrics registry, exposed on
/metricswith no new wiring. Internal-only change (no package release needed), so the changelog note is a single.server-changesentry.