Skip to content

Fix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readiness - #52994

Merged
pelikhan merged 6 commits into
mainfrom
copilot/aw-debug-daily-ambient-workflow
Aug 16, 2026
Merged

Fix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readiness#52994
pelikhan merged 6 commits into
mainfrom
copilot/aw-debug-daily-ambient-workflow

Conversation

CopilotAI commented Aug 16, 2026

Copy link
Copy Markdown
Contributor

Daily Ambient Context Optimizer (and other docker-sbx + Copilot SDK workflows) failed repeatedly because the harness sent its first chat-completion request to the api-proxy sidecar before its per-provider listener was actually accepting connections, exhausting all retries on ECONNREFUSED and ending the run with zero agent turns (driver_exit).

Root cause

  • api-proxy reports /reflect healthy and passes key validation before the per-provider listener (e.g. http://api-proxy:10002) is actually accepting TCP connections.
  • The Copilot SDK driver's 3 retry attempts all land inside this startup window and fail identically, so the run dies within ~1 minute with no agent turns recorded.
  • This exact defect was previously diagnosed and fixed in Harden Copilot api-proxy startup: verify listener accept readiness and absorb first-request ECONNREFUSED #52619, but that PR was closed without merging, leaving the underlying race unresolved.

Fix

  • awf_reflect.cjs: added waitForProviderListenerReady(), which probes a real TCP connect (TLS handshake for https:// baseUrls) against a provider listener with a bounded timeout/retry cadence.
  • copilot_harness.cjs: in SDK multi-provider mode, gates on listener readiness for every unique provider baseUrl before the first request, and classifies a first-attempt ECONNREFUSED as a one-shot short-backoff retry instead of burning the full retry budget on a known-transient race.
  • Re-applied the prior fix's test coverage in awf_reflect.test.cjs / copilot_harness.test.cjs, resolving conflicts with unrelated test additions merged since.
// copilot_harness.cjs (SDK mode): require real listener readiness before first requestconstreadiness=awaitwaitForProviderListenerReady({baseUrl: providerBaseUrlToProbe,timeoutMs: AWF_PROVIDER_LISTENER_READY_TIMEOUT_MS,logger: log,});if(!readiness.ok){emitInfrastructureIncomplete(`api-proxy provider listener was not ready at ${providerBaseUrlToProbe} before first Copilot SDK request (${readiness.error}).`);process.exit(1);}

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.49 AIC · ⌖ 5.67 AIC · ⊞ 6.3K ·
Comment /souschef to run again


Run: https://github.com/github/gh-aw/actions/runs/31921676473> Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.82 AIC · ⌖ 5.65 AIC · ⊞ 8.7K ·

Comment /souschef to run again

CopilotAI linked an issue Aug 16, 2026 that may be closed by this pull request
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

Triage

  • Category: bug (WIP self-debug)
  • Risk: low
  • Score: 20/100 (impact 5, urgency 10, quality 5)
  • Recommendation:defer

Zero-diff WIP PR auto-created by an agent to debug a failed Daily Ambient Context Optimizer run. No files changed yet. Revisit once a diff lands or close if stale.

Generated by 🔧 PR Triage Agent · auto · 40.8 AIC · ⌖ 2.37 AIC · ⊞ 8.1K ·

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
CopilotAI changed the title [WIP] Debug daily ambient context optimizer workflow failureFix ECONNREFUSED driver_exit failures in Copilot SDK engine by gating on api-proxy listener readinessAug 16, 2026
CopilotAI requested a review from pelikhanAugust 16, 2026 00:33
@pelikhan
pelikhan marked this pull request as ready for review August 16, 2026 00:35
CopilotAI balanced review requested due to automatic review settings August 16, 2026 00:35

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds provider-listener readiness checks to prevent Copilot SDK startup races.

Changes:

  • Probes provider TCP/TLS listeners before SDK requests.
  • Adds one-shot ECONNREFUSED retry handling.
  • Adds readiness and retry tests.
Show a summary per file
FileDescription
actions/setup/js/awf_reflect.cjsImplements listener readiness probing.
actions/setup/js/awf_reflect.test.cjsTests probe behavior and timeouts.
actions/setup/js/copilot_harness.cjsGates SDK startup and handles refused connections.
actions/setup/js/copilot_harness.test.cjsAdds detection and retry tests.

Review details

  • Files reviewed: 4/4 changed files
  • Comments generated: 3
  • Review effort level: Balanced

Comment threadactions/setup/js/copilot_harness.cjs Outdated
// provider died in the narrow window between the probe and the first request — retry
// once as a fresh run. Later attempts (attempt > 0) fall through to the generic retry
// handling below instead of taking this one-shot path.
if (attempt === 0 && isConnectionRefused && maxRetries > 0) {
Comment on lines +495 to +509
socket.once(readyEvent, () => {
clear();
// Remove the error listener before tearing down the socket: a successful handshake has
// already settled this probe as ready, and a late/trailing error (e.g. an abrupt RST)
// must not overwrite lastError or otherwise affect the already-settled result.
socket.removeAllListeners("error");
socket.destroy();
settle(true);
});
socket.once("error", err => {
clear();
lastError = getErrorMessage(err);
socket.destroy();
settle(false);
});
Comment on lines +1882 to +1886
function decideConnectionRefusedRetry(isConnectionRefused, attempt, maxRetries) {
if (attempt === 0 && isConnectionRefused && maxRetries > 0) {
return { action: "retry", nextDelayMs: FIRST_CONNECTION_REFUSED_RETRY_DELAY_MS };
}
return { action: "continue" };
@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot this PR looks ready for a finishing pass.

Please:

  • address any open inline review feedback from copilot-pull-request-reviewer
  • refresh the branch if needed
  • run the pr-finisher skill before returning it for maintainer review

Run: https://github.com/github/gh-aw/actions/runs/31918518969

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 8.49 AIC · ⌖ 5.67 AIC · ⊞ 6.3K ·
Comment /souschef to run again

…or listener, test production helper
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>

CopilotAI commented Aug 16, 2026

Copy link
Copy Markdown
ContributorAuthor

@copilot this PR looks ready for a finishing pass.

Please:...

Finishing pass done in 047e2a8:

  • SDK-mode scoping: the one-shot ECONNREFUSED retry now goes through a new exported shouldRetryFirstConnectionRefused({ copilotSDKMode, attempt, isConnectionRefused, maxRetries }), so CLI failures keep the generic retry policy (including --continue) instead of being forced into a fresh run.
  • Socket error listener: waitForProviderListenerReady() no longer calls removeAllListeners("error"). The probe settles as ready before destroy(), the error listener stays installed, and the handler ignores already-settled probes — so a late RST can't surface as an uncaught EventEmitter error. The timeout path settles before destroy() for the same reason.
  • Tests: the inlined retry-decision helper in copilot_harness.test.cjs was replaced with tests that call the exported production helper, including a CLI-mode guard case; awf_reflect.test.cjs now asserts the error listener is still installed at destroy() time.

Branch is up to date with main (0 commits behind). Prior CI on this branch had no failures — the completed CJS run on 839b739 was green, and the remaining runs are action_required (awaiting maintainer approval). Local validation (make agent-report-progress, incl. format, JS lint, typecheck, impacted tests, and make recompile lock-file sync) passes. CI on the new HEAD is unverified until a maintainer re-triggers it.

@gh-aw-bot

Copy link
Copy Markdown
Collaborator

@copilot this PR looks ready for a finishing pass.

Please:

Run: https://github.com/github/gh-aw/actions/runs/31921676473

Generated by 👨‍🍳 PR Sous Chef · gpt54 · 3.82 AIC · ⌖ 5.65 AIC · ⊞ 8.7K ·
Comment /souschef to run again

@pelikhan
pelikhan merged commit 1f1a46a into mainAug 16, 2026
1 check failed
@pelikhan
pelikhan deleted the copilot/aw-debug-daily-ambient-workflow branch August 16, 2026 02:26
Copilot stopped work on behalf of gh-aw-bot due to an error August 16, 2026 02:26
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This pull request is included in a new release.

Release: v0.87.1

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[aw] Daily Ambient Context Optimizer failed

4 participants

@gh-aw-bot@pelikhan