Skip to content

fix(engine): stop non-LLM steps constructing a provider for a context window - #456

Merged
Jason Robert (jrob5756) merged 1 commit into
mainfrom
fix/non-llm-step-provider-lookup
Aug 18, 2026
Merged

fix(engine): stop non-LLM steps constructing a provider for a context window#456
Jason Robert (jrob5756) merged 1 commit into
mainfrom
fix/non-llm-step-provider-lookup

Conversation

@jrob5756

Copy link
Copy Markdown
Collaborator

Fixes the two failing jobs on main: Web BG Smoke (windows-latest) (failing 3 of the 4 runs since it was added in #450) and a flaky Test (Python 3.13, ubuntu-latest).

1. A step with no model was constructing a provider

_execute_loop built every agent_started payload with a context_window_max resolved through _get_context_window_for_agent, which resolves a provider — and ProviderRegistry builds providers lazily. A wait / set / script / terminate / human_gate step has no model, so that construction bought a guaranteed None.

It is also charged to limits.timeout_seconds, because it runs inside the engine's timed loop. Profiling a provider-free wait workflow, the first step spent 0.434s of the budget building a Copilot SDK client:

gap workflow_started -> first agent_started: 0.451s # before
gap workflow_started -> first agent_started: 0.000s # after

On a cold Windows CI runner that (plus general per-step overhead) consumed enough of examples/wait-smoke.yaml's 3s cap to time the workflow out. The child then exits non-zero, and the launch gate reports it as Background process exited before the workflow started (code 1) — failing the smoke job for something it does not test. In-engine time for that workflow drops from 1.46s to 1.01s, of which 1.00s is the sleeps it is meant to perform.

is_llm_agent was already computed two lines above, so the fix gates the lookup on it. Provider-backed agents are unaffected; the two completion-side call sites already run only in the LLM branches.

TestNonLlmStepsSkipTheProviderLookup pins it with a spy on ProviderRegistry.get_provider — verified to fail on the unfixed engine, so it is not vacuous.

2. test_a_scan_failure_does_not_wedge_the_screen was racing its own poll timer

The two TestRunsScreenGuardRecovery tests asserted _refreshing is False a single instant after settle(), while shrinking POLL_INTERVAL_SECONDS to 0.05s — so a tick landing during settle()'s trailing pilot.pause() legitimately set the flag back to True before the assertion read it (assert True is False). Reproduced locally at ~20%.

They now wait for the flag to clear, which is what the finally actually guarantees: without it the flag latches True for the session and the wait times out, so the regression is still caught. 20/20 clean, from 2/10 failing.

3. Headroom for the smoke fixture

examples/wait-smoke.yaml's 3s cap left no margin on a cold Windows runner even after fix 1, so it is now 15s, with examples/README.md and the ci.yml job comment updated. The timeout path the example demonstrates is unchanged — drive it with a larger --input middle_duration_ms.

Validation

  • make check (ruff, ruff format, ty) — clean
  • tests/test_engine + tests/test_fleet — 1602 passed, 1 skipped
  • make validate-examples — exit 0
  • Real conductor run examples/wait-smoke.yaml --web-bg — launches, reaches workflow_completed in 1.013s in-engine, conductor stop cleans up

… window
`_execute_loop` built every `agent_started` payload with a
`context_window_max` resolved through `_get_context_window_for_agent`,
which resolves a provider -- and the registry builds providers lazily. A
`wait` / `set` / `script` / `terminate` / `human_gate` step has no model,
so that construction bought a guaranteed `None`.
It is also charged to `limits.timeout_seconds`, because it runs inside
the engine's timed loop. Measured on a provider-free wait workflow, the
first step spent 0.434s of the budget building a Copilot SDK client; on a
cold Windows CI runner it consumed enough of `examples/wait-smoke.yaml`'s
3s cap to time the workflow out, exit the child non-zero, and fail the
`Web BG Smoke (windows-latest)` job as a launcher failure (3 of the 4
runs since that job was added). In-engine time for that workflow drops
from 1.46s to 1.01s, of which 1.00s is the sleeps it is meant to perform.
`is_llm_agent` was already computed two lines above, so the fix is to
gate the lookup on it. Provider-backed agents are unaffected; the
completion-side sites already run only in the LLM branches.
Also fixed, both surfaced by the same CI failures:
- `test_a_scan_failure_does_not_wedge_the_screen` and its render-failure
sibling asserted `_refreshing is False` a single instant after
`settle()`, while the test's own 0.05s poll timer legitimately flips
the flag back to `True` -- including inside `settle()`'s trailing
pause. That failed ~20% of local runs and once on CI (Python 3.13,
ubuntu). They now wait for the flag to *clear*, which is what the
`finally` actually guarantees: without it the flag latches `True`
forever and the wait times out, so the regression is still caught.
20/20 clean, from 2/10 failing.
- `examples/wait-smoke.yaml`'s 3s cap left no headroom for a cold
Windows runner even after the engine fix, so it is now 15s.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jrob5756
Jason Robert (jrob5756) merged commit f0849c5 into mainAug 18, 2026
13 checks passed
@jrob5756
Jason Robert (jrob5756) deleted the fix/non-llm-step-provider-lookup branch August 18, 2026 15:26
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@jrob5756