Skip to content

ci(docs-drift): tolerate transient API failures when delivering the advisory, loudly - #9423

Merged
os-steve merged 1 commit into
mainfrom
claude/issue-9373-docs-flag-503-tolerance
Aug 18, 2026
Merged

ci(docs-drift): tolerate transient API failures when delivering the advisory, loudly#9423
os-steve merged 1 commit into
mainfrom
claude/issue-9373-docs-flag-503-tolerance

Conversation

@os-steve

Copy link
Copy Markdown
Collaborator

Fixes#9373

What was measured, not assumed

The card attributes the failure to "posting the advisory comment". Verified against a tree
asserted equal to origin/main (51a46a440), with one refinement worth recording: the URL in
the recorded error, /repos/.../issues/9370/comments, is the endpoint of both
issues.listComments (GET) and issues.createComment (POST). The job log therefore cannot say
which of the two calls was rejected, so both are covered here.

Propagation path, confirmed rather than presumed. There is no continue-on-error anywhere in
this workflow — not on the step, not on the job. actions/github-script registers
main().catch(handleError), and handleError calls core.setFailed("Unhandled error: " + err).
That is exactly the ##[error]Unhandled error: HttpError: ... signature in the card. So any throw
from the inline script fails the step, and the step fails the job.

For calibration: this job's name, Flag docs affected by code changes, is not in the
REQUIRED_CONTEXTS registry, so the red did not block branch protection. Its measured cost is the
red check itself, the four re-run cycles, and the noise in mergeable_state — not a hard merge
block. Recording that so nobody inherits an overstatement.

The fix, and the half that is binding

Delivery of the advisory — and only delivery — now tolerates transient platform weather. The
verdict is unchanged and still governs the job's conclusion.

It does not fall silent. When the bounded retries are spent the run says so, in its own output,
where a reader looks for this advisory's verdict:

  • a warning annotation titled "Docs drift advisory not delivered";
  • a job summary that names the failed call and status, states plainly that the verdict is
    not on the pull request, warns that any advisory comment shown there is from an earlier
    push
    and was not refreshed by this run, and then reproduces the verdict this run actually
    computed
    .

No verdict is emitted that was not computed. This matches the posture the file already takes for
its derivation limits — "this run has no opinion", plus a named list of what it could not anchor.

Why not continue-on-error: true, and why not a bare catch. The step also parses
affected.json and builds the comment body. Blanket tolerance would let a malformed scan result,
or a 422 from an over-long body, read as a clean run — real breakage wearing a green tick. And a
swallowed failure leaves the run saying nothing at all, so "could not tell" renders exactly like
"no drift", which is the same defect wearing the opposite mask.

503 is not the right granularity — the class is. Tolerated: 5xx, 429, 403carrying a
secondary-rate-limit signature
(GitHub answers a secondary limit with 403 as well as 429, so the
signature and not the status alone separates it from a real permission denial), and network-level
error codes. Still fatal, on purpose: 401, a plain 403 (the permissions: block is wrong), 404,
422, and every non-HTTP error such as a TypeError in the code above. Those are this repo's own bugs.

Retries are bounded and short — 3 attempts, 3s then 9s. The measured incident ran over an hour; no
retry budget rides that out, and pretending otherwise only burns runner minutes before degrading
anyway. The retries are for a blip; the visible-degradation path is what handles an incident.

Verification — a simulated failure path, not an observed 503

A real 503 cannot be summoned, and none was observed. Instead the inline github-script source was
extracted from the workflow and executed against a mocked octokit and @actions/core, reproducing
github-script's own main().catch(handleError) propagation. Backoff was made instant by overriding
setTimeout in the harness, so the shipped workflow carries no test hook. The harness lives in
the session scratchpad, not in this diff — adding a workflow-script test rig is outside this card's
scope.

Baseline, against this file as it stands on origin/main:

scenariobefore
503 on the write, persistentFAILUREUnhandled error: HttpError: No server is currently available to service your request.
503 on the listing, persistentFAILURE — same signature
happy pathsuccess

After, all ten scenarios:

scenarioAPI callsjobwarningsummary
happy pathlist, createsuccess0
503 on write, clears on retrylist, create, createsuccess0
503 on write, persistentlist, create x3success11832 B
503 on listing, persistentlist x3success11907 B
503 while updating an existing commentlist, update x3success11887 B
network ECONNRESETlist, create x3success11829 B
403 secondary rate limitlist, create, createsuccess0
422 body too longlist, createFAILURE0
403 plain permission deniallist, createFAILURE0
malformed affected.jsonnoneFAILURE0

Both halves of the ruling, demonstrated: the job does not fail on the transient class, and the
run states what happened. The degraded warning reads:

The docs-drift advisory was computed but could not be posted to this PR: issues.listComments
failed 3x with HTTP 503: No server is currently available to service your request. This run could
not even determine whether an advisory comment exists on this PR; if one is shown there, it is
from an earlier push. The verdict is in this run's job summary. Transient GitHub API failure
(#9373) — the job stays green because the scan itself is unaffected.

The three FAILURE rows are the important ones: they are what separates this from the swallow-it fix.

One design note recorded in the file: when the listing is what fails, the run degrades without
posting blind. It cannot tell an update from a create, and posting anyway would strand a second
advisory comment that the marker dedup would then update forever alongside the first — and every
comment here is relayed into subscribed agent sessions.

Scope

One file, .github/workflows/docs-drift-check.yml, and within it only the delivery tail plus the
file-header comment that described the old contract (it claimed "never fails the build", which was
untrue in exactly the way this card records). The advisory's derivation logic is untouched — that
is #9282 / #9294 / #9331 territory, deliberately not entered here.

Because this PR edits the workflow's own file, which is in its own paths: trigger, the modified
job runs on this PR and exercises the happy delivery path live.

Gates

Re-derived from the actual changed path via node scripts/pm/dispatch-gates.mjs, which added one
family the dispatch list did not name: the workflow's own self-test. All run at final commit
86aab217e:

pnpm check:node-version exit=0
pnpm check:required-contexts exit=0
pnpm check:shard-attestation exit=0
pnpm check:workflow-status-functions exit=0
pnpm check:nul-bytes exit=0
node scripts/docs-audit/check-affected-docs.mjs exit=0 (197 cases)
node scripts/check-shard-attestation.mjs exit=0

check:workflow-status-functions is a no-op for this change by construction: no job-level if: is
added or altered, and the job reads no needs.*.outputs.* at all.

Workflow-only, publishes nothing, so skip-changeset rather than a changeset file.


Generated by Claude Code

…dvisory, loudly
The `Flag docs affected by code changes` job died four consecutive times on
PR #9370 (2026-08-17, 17:17Z-18:24Z) with a 503 from
`/repos/.../issues/9370/comments`. actions/github-script routes any throw from
the inline script to `main().catch(handleError)` -> `core.setFailed(...)`, so a
degraded GitHub API turned an advisory-only check red and cost four re-runs
that no local change could have fixed.
Delivery of the advisory comment now retries a bounded, narrow transient class
(5xx, 429, 403 carrying a secondary-rate-limit signature, and network-level
codes) and, once the retries are spent, degrades VISIBLY instead of failing:
a warning annotation plus a job summary that names the failed call, states that
the run's verdict is NOT on the pull request, warns that any advisory comment
shown there is from an earlier push, and reproduces the verdict it did compute.
The tolerance is scoped to delivery only. A malformed `affected.json`, a 422
over-long body, a plain 403 permission denial and any non-HTTP error still fail
the job — swallowing those would let "could not tell" render as "no drift",
which is the same defect wearing the opposite mask.
Fixes#9373
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

Nothing in this diff resolved to a documentable surface (no symbol, route or SDK anchor derived from 0 changed package(s)), so this run has no opinion about the docs.

@claudeclaudeBot added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 18, 2026
@os-steve
os-steve marked this pull request as ready for review August 18, 2026 00:53
@os-steve
os-steve added this pull request to the merge queueAug 18, 2026
Merged via the queue into main with commit f099dcaAug 18, 2026
25 checks passed
@os-steve
os-steve deleted the claude/issue-9373-docs-flag-503-tolerance branch August 18, 2026 01:17
os-steve pushed a commit that referenced this pull request Aug 18, 2026
…onger destroys the triage
The triage comment is the machine-readable signal the PM dispatch loop keys on,
and the cross-PR flake evidence lives in it — but it was posted through a bare
`github-script` call with no retry and no handler, so a transient GitHub API
failure killed the job and took the diagnosis with it: `body` exists nowhere but
the failed request, and the run page was left showing `Unhandled error:
HttpError` and nothing else.
Declare the retry policy on the step (`retries: 3`, with 403 removed from the
exempt list because GitHub answers a secondary rate limit with 403 as well as
429, and this job paginates jobs, logs and 24 h of runs). Nothing is swallowed:
a spent retry still throws, so detection failures fail exactly as before.
On a refused delivery, write the whole computed triage into the job summary —
where it outlives the request and can be pasted onto the PR — and then fail.
Failing is deliberate and diverges from the sibling fix in docs-drift-check.yml
(#9373/#9423): this is a `workflow_run` job whose conclusion is on no check
list, in no required context and gates nothing, so red costs nothing while a
green that quietly delivered nothing recreates the silence one layer further
in. That buys the invariant: green ⇔ the triage comment is on the PR.
The de-duplication listing degrades to at-least-once instead: its marker is
scoped to one run id, so a duplicate is inert while a miss loses the signal.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XqDQYVU5smx29ts9pAErja
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cdsize/mskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

2 participants

@os-steve@claude