Uh oh!
There was an error while loading. Please reload this page.
feat: two-phase status polling via lightweight /status endpoint - #446
Merged
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit:d162501efc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
SunnySoldier357force-pushed
the
sandeep/poll-status
branch
from
April 24, 2026 18:52
e5ef516 to
d191e12CompareUh oh!
There was an error while loading. Please reload this page.
SunnySoldier357force-pushed
the
sandeep/poll-status
branch
from
April 24, 2026 21:54
d191e12 to
a17c9b8CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Use the lightweight status endpoint from RemoteRolloutProcessor via the shared aiohttp session and avoid the logs backfill after terminal status is observed. Made-with: Cursor
Continue polling when the lightweight status endpoint returns RUNNING so the remote rollout processor only exits the poll loop for terminal statuses. Made-with: Cursor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8e0e0ef. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
Read rollout status extras from the top-level status response so RemoteRolloutProcessor preserves metadata that previously came from log entries. Made-with: Cursor
xzrderek
enabled auto-merge (squash)
April 29, 2026 03:12
xzrderek
disabled auto-merge
April 29, 2026 03:12
Uh oh!
There was an error while loading. Please reload this page.
3 tasks
SunnySoldier357 added a commit
that referenced
this pull request
May 6, 2026
The lightweight `/status` endpoint on the tracing gateway only returns the status code; `Message`, `Details`, and `Extras` still live on the Logs table. After PR #446 stopped reading from `/logs` on terminal status, the SDK was constructing `Status(code=..., message="", details=[])` for every completed rollout and `EvalProtocolError(message="")` for failures, which broke `tests/remote_server/test_remote_fireworks_propagate_status.py` (`assert row.rollout_status.message == "test error"`). Restore the two-phase polling shape from the original PR: poll `/status` for the code, and on a terminal (non-RUNNING) code do one `async_search_logs` call to backfill `message`/`details`/`extras` from the matching log row. This is still ~1000x cheaper on the Logs table than the pre-#446 polling loop because the search runs once per rollout completion instead of every poll interval. Made-with: Cursor Co-authored-by: Cursor <cursoragent@cursor.com>
SunnySoldier357 added a commit
that referenced
this pull request
May 7, 2026
…451) * fix: backfill rollout status fields from logs when polling completes The lightweight `/status` endpoint on the tracing gateway only returns the status code; `Message`, `Details`, and `Extras` still live on the Logs table. After PR #446 stopped reading from `/logs` on terminal status, the SDK was constructing `Status(code=..., message="", details=[])` for every completed rollout and `EvalProtocolError(message="")` for failures, which broke `tests/remote_server/test_remote_fireworks_propagate_status.py` (`assert row.rollout_status.message == "test error"`). Restore the two-phase polling shape from the original PR: poll `/status` for the code, and on a terminal (non-RUNNING) code do one `async_search_logs` call to backfill `message`/`details`/`extras` from the matching log row. This is still ~1000x cheaper on the Logs table than the pre-#446 polling loop because the search runs once per rollout completion instead of every poll interval. Made-with: Cursor Co-authored-by: Cursor <cursoragent@cursor.com> * fix: match backfilled log status to terminal status code Bugbot pointed out that the backfill loop could pick an earlier RUNNING/partial status log instead of the terminal one when a rollout emits multiple status-bearing logs. The reported `code` was always correct (it came from /status), but `message`/`details`/`extras` could be attached from the wrong row and the raised exception would carry misleading text. Match the log row's status code to the terminal code returned by /status so the backfill is deterministic. Made-with: Cursor --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
get_status()toFireworksTracingAdapterthat calls the new/statusendpoint on the tracing gateway for fast rollout status lookups.RemoteRolloutProcessorwith a two-phase approach: poll/statusfor the status code (lightweight point-read), then make a singlesearch_logscall to backfill message/details/extras.This reduces the hot-path read load on the Logs table from ~1000 RPS to a single read per rollout completion. Depends on the tracing gateway PR that adds the
/statusendpoint andStatusSpanner table ([mono PR](https://github.com/fw-ai/fireworks/pull/$(cd /Users/sandeepsingh/src/fireworks/mono && gh pr view --json number -q .number 2>/dev/null || echo "TBD"))).Test plan
get_status()returns status when available,Nonewhen notRemoteRolloutProcessorpolls/statusfirst, then fetches full logs onceMade with Cursor
Note
Medium Risk
Changes remote rollout completion polling logic and introduces a new tracing gateway call path, which could affect rollout completion detection and timeout behavior if the
/statusresponse shape/availability differs across deployments.Overview
Adds
FireworksTracingAdapter.get_status()to query a lightweight tracing gateway/status(with/v1/statusfallback) for rollout status codes.Updates
RemoteRolloutProcessorto use a two-phase polling flow: repeatedly pollget_status()until the rollout leavesRUNNING, then perform a singleasync_search_logs()call to backfill statusmessage/detailsand propagateextrasintoexecution_metadata.extra, reducing repeated log-table reads during polling.Reviewed by Cursor Bugbot for commit a17c9b8. Bugbot is set up for automated code reviews on this repo. Configure here.