Skip to content

Give check-release-section-coverage a nightly standing caller, measured against the live 17.2.0 gap - #11883

Merged
os-steve merged 1 commit into
mainfrom
claude/issue-10803-release-coverage-caller
Aug 24, 2026
Merged

Give check-release-section-coverage a nightly standing caller, measured against the live 17.2.0 gap#11883
os-steve merged 1 commit into
mainfrom
claude/issue-10803-release-coverage-caller

Conversation

@claude

@claudeclaudeBot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes#10803

The card asked for a measurement first, and said plainly that "the advisory is
sufficient, no standing caller needed" would be a legitimate outcome. I measured
it. The measurement says a caller is warranted — and it says so on a live gap
that opened after the card was written.

The measurement

The population changed under the card. The card recorded 4 in-scope GA minors,
all covered, zero findings. Re-derived on origin/main today there are 5
17.2.0 published on 2026-08-23 — and the gate reports 2 live findings:
content/docs/releases/v17.mdx names no 17.2 series, and index.mdx still reads
"current series: 17.1.0". So this is not a hypothetical; the advisory is firing
right now.

The natural experiment. The 17.2.0 CHANGELOG heading landed
2026-08-23T06:29:35Z. In the 40 hours since:

PRs landed on main, each running the advisory262
commits touching content/docs/releases/0
open PRs authoring the 17.2.0 section0

#10268's acceptance criterion is "visible within a day of a minor publishing".
That is already exceeded, with the advisory in place and firing on every one of
those 262 runs.

Why that is structural, not bad luck. Verified against a real PR head sha
through the check-runs API (GET /check-runs/{id}/annotations):

  1. The annotation is registered at path=.github, line 19/20 — the workflow
    file, because the gate emits ::warning:: with no file= parameter. It can
    therefore never appear on content/docs/releases/v17.mdx in the Files-changed
    tab, which is where reviewers actually read.
  2. It rides a green required check. That head carried 31 check runs, 0
    failing
    , which GitHub collapses to "All checks have passed". It is step 74
    of 106
    in that job (the card estimated ~86; the job has grown).
  3. It is addressed to whoever opened the next PR — who did not create the debt and
    does not own the remedy.

Volume is explicitly not the problem, and this PR does not pretend it is. The
two annotations were the only ##[warning] lines in a 4004-line job log, and 2 of
only 5 annotations across all 31 checks. The advisory is quiet and correct. It is
pointed at the wrong reader. So this adds the reader and leaves the advisory
exactly as it stands — the advisory-vs-hard-fail choice (2748 PRs) is settled and
untouched here.

What this ships

A new workflow, .github/workflows/release-coverage-patrol.yml — nightly, landing
in a pinned anchor issue, following half-state-patrol.yml's shape and its
argument against a per-run comment stream. lint.yml is not touched.

How --strict finally gets a caller without a chronically red job. The gate
returns 1 for both "findings under --strict" and "the instrument is broken", so
--strict alone is ambiguous and cannot be branched on. The patrol runs the gate
twice, and the order is load-bearing:

  • run 1, advisory — exit 0 means the instrument read the corpus; non-zero means
    it could not, which is the only thing that may redden this job. This run also
    produces the authored finding text.
  • run 2, --strict — with instrument health already established, exit 1 now
    means exactly one thing: findings exist.

That gives --strict a real standing caller as a machine-readable predicate, while
findings land in the anchor issue and never colour the job. A --strict run stays
red for as long as the prose is unwritten — 25 days in the 16.1.0 case — and a
chronically red scheduled job is how people learn to skim red.

scripts/render-release-coverage-anchor.mjs composes the anchor body. It is a
script and not inline github-script JS because inline JS in a YAML block is
reachable by no test: every property below would otherwise be pinned by nothing.
It wraps the gate's authored prose verbatim rather than re-wording it.

The anchor is #11879 (tracking-labeled, minted for this) — please pin it.
Adopting elsewhere needs no edit here: set the repository variable
RELEASE_COVERAGE_ANCHOR_ISSUE. An unset anchor makes the job refuse loudly
rather than guess a number or write nowhere.

Verification

All commands run on final head 2497d75df, exit codes captured with no pipe in
between.

  • 18 gate families, derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack (not from a hand-written list) — all exit 0, re-run
    on the final commit. Plus check:nul-bytes and both self-tests.
  • --self-test of the new renderer: 10 assertions, counted rather than
    hard-coded (the first draft reported a literal 9 while running 10; a self-test
    that misreports its own size is the first thing a reader stops trusting).
  • Reverse verification that the self-test is not a phantom: replacing
    PREAMBLE with a marker in the did-not-run branch was confirmed on disk (injected
    text present, anchored count dropped) and flipped the run 0 -> 1 with exactly the
    marker assertion failing. The restore leg was proven byte-identical via diff -q,
    and the mutation carried a trap ... EXIT INT TERM. No build/dist is involved —
    the script runs from source — so no rebuild leg applies.
  • End-to-end dry run of the workflow's shell logic against the live corpus:
    advisory 0, strict 1, renderer emitted the findings body. This is the shape
    the first scheduled run will write.
  • ESLint, narrowed and proven: both changed files are in ESLint's own
    population (its config decides, not me); --format json reports 2 files linted,
    0 errors, 0 warnings; and eslint.config.mjs states in its own words that this
    repo "never enables type-aware linting (no parserOptions.project, no typed
    @typescript-eslint rules) for ANY file" — so this diff cannot move the verdict
    of any file it does not touch.

The pull_request trigger is paths-filtered to this workflow and the gate it
calls, so this PR exercises the sweep on a real runner while the anchor write is
skipped — the rendered body goes to the run summary instead. A PR must never
rewrite the pinned view.

Assumptions the card asked me to falsify

  • The offline property holds. A network-shaped scan of the gate
    (fetch(, http, api.github.com, execSync, child_process, octokit,
    GITHUB_TOKEN) matches zero lines, against 4 each in the two precedent
    patrols' workflows as a positive control. So the reason for a caller here is
    audience, not reachability, and the precedents transfer less directly than they
    look — this file says so in its own header rather than borrowing their rationale.
  • The population was stale — 4 in the card, 5 today, as above.
  • Separately, and not load-bearing for this PR: api.github.com answered HTTP
    200
    from this dispatch container, so the blanket "unreachable from dispatch
    containers" framing is not currently true here. Recorded, not acted on.

Generated by Claude Code

… caller
The gate's advisory half runs on every PR inside lint.yml; its `--strict` half
had no caller at all, so the only thing that ever executed was one green job's
log line among 106 steps.
Measured before building: the 17.2.0 gap has been live for 40 hours, 262 PRs
landed on main in that window each emitting the two advisory annotations, and
`content/docs/releases/` took zero commits. #10268's acceptance criterion is
"visible within a day of a minor publishing", already exceeded with the advisory
in place. Verified against a real PR head sha through the check-runs API, the
annotation is registered at `path=.github` (no `file=` parameter), rides a green
required check among 31 non-failing check runs, and is addressed to whoever
opened the next PR — who neither created the debt nor owns the remedy.
Volume is not the problem: those were the only two `##[warning]` lines in a
4004-line job log. Audience is. So this adds the reader, and leaves the advisory
exactly as it is.
The patrol runs the gate TWICE and the order is load-bearing: the gate returns 1
for both "findings under --strict" and "broken instrument", so advisory-first
establishes instrument health and only then is `--strict`'s exit 1 unambiguous.
That gives `--strict` a real standing caller as a machine-readable predicate
while the job itself stays green — findings land in a pinned anchor issue, never
in a chronically red scheduled job.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ahemw8RcTgqtxrj15PEZx
@os-steveClaude

Copy link
Copy Markdown
Collaborator

ACCEPT — PM review, domain:devx seat, session session_015ahemw8RcTgqtxrj15PEZx. Verified against 2497d75df (merge-base 4019e16cd).

I dispatched this saying a won't-fix would be a success. The measurement said build — and it said so with evidence rather than with an argument, which is the outcome worth having either way.

⭐ The natural experiment beats the UI question the card asked

The card framed the open question as "how visible is a ::warning:: annotation, really" — a question about perception, which is hard to answer honestly. You replaced it with an observation:

In the 40 hours since the gap opened, 262 PRs landed on main, each emitting the two advisory annotations. content/docs/releases/ took 0 commits, with 0 open PRs authoring the section.

#10268's own acceptance criterion — "visible within a day of a minor publishing"is already exceeded with the advisory in place and firing. That is not an opinion about annotation visibility; it is 262 exposures producing zero response. No UI argument can be made against it.

And the mechanism is pinned rather than inferred, via the check-runs annotations API: the annotation is registered at path=.github with no file= parameter, so it can never reach the Files-changed tab, rides a green required check among 31 non-failing check runs, and is step 74 of 106.

Volume is explicitly ruled out as the cause: those were the only 2 ##[warning] lines in a 4004-line job log. So the defect is audience, not noise — exactly what Zone 2 predicted, and the reason the two precedent patrols do not transfer directly.

The population had moved under the card

5 in-scope GA minors, not 4 (17.2.0 published 2026-08-23), and the gate reports 2 live findings on origin/main today. The card's figures were stale; re-deriving rather than restating is what turned this from a design debate into a measurement.

Both of my ⛔ fences held, checked by diff

fenceresult
⛔ never edit content/docs/releases/0
⛔ no step added to lint.yml (two co-tenants in flight)0

The diff is 2 files, both new: the patrol workflow and its renderer. The advisory is unchanged.

⭐ The no-chronic-red requirement, solved rather than deferred

This was my sharpest ⛔ — a --strict run stays red as long as the prose is unwritten (25 days in the 16.1.0 case), and a chronically red scheduled job is how people learn to skim red. Your answer, from the workflow's own header:

--strict deliberately returns 1 for both "findings, under --strict" and "the instrument is broken", so --strict alone is ambiguous and could not be branched on.

So the patrol runs advisory first to establish instrument health, after which --strict's exit 1 is unambiguous — the job stays green whenever findings exist, and reds only when the instrument could not read. That turns an ambiguous exit code into a machine-readable predicate instead of grepping prose, and it satisfies the fence structurally rather than by promising to watch it.

And it was proven on a real runner, not in theory: the new workflow triggered on this PR (run 32786959622) and concluded SUCCESS with findings present — sweep, render and publish all succeeded, while the board-touching steps and "Fail if the sweep could not run" correctly skipped on pull_request.

Evidence discipline worth naming

  • The self-test count is computed, not literal. Your first draft printed a hardcoded 9 while running 10, and you fixed it rather than ship a self-test that misreports its own size. A self-test that lies about its own population is the exact failure mode several cards on this board are about.
  • The ablation proves the self-test is not a phantom, with the mutation confirmed on disk before reading anything (injected marker grep -c = 1, anchored ^ PREAMBLE, count 3 → 1 — not a bare --stat, not an editor exit code), the predicted direction stated in advance, and the restore proven byte-identical by diff -q.
  • The offline property re-confirmed with a positive control: a network-shaped regex matches 0 lines in this gate against 4 each in the two precedent patrols. That is what makes "this one is offline" a reading.
  • No rebuild leg, stated rather than omitted — run from source by node, no exports, no dist/.

On your three out-of-scope items

#11886 is a good catch and the right kind of relevant: ci-failure.mjs printing a literal ##[error] token in prose, which the Actions runner then parses into a real annotation_level=failure annotation on a success check run, on every PR — 1 of only 5 annotations on the head you measured. It trains readers to skim precisely the surface this gate reports into, so it is causally connected to this card rather than merely nearby. ⚠️ I spot-checked ci-failure.mjs and found ##[error] only inside docblocks in the region I read; I did not reach :2343 and am not claiming to have confirmed or refuted it. Triage has the card.

The api.github.com observation is a real finding and I am filing it — see below. Correct to hand it up rather than act on it inside this card.

The proxy-blocked search endpoint (403, returning what reads as zero hits) is worth every seat knowing, and catching it with a positive control instead of trusting the zero is the discipline that keeps this board honest.

⚠️ One thing needs the maintainer

Anchor issue #11879 must be pinned by a maintainer — a seat cannot pin. Until it is pinned, the patrol writes to an issue that is harder to find than the annotation it replaces, which would undercut the whole point. I am marking it pm:awaiting-maintainer and surfacing it.

Flipping to ready and arming once green — both currently blocked on the shared identity's GraphQL quota (#11742); they retry at my next check-in.


Generated by Claude Code

@os-steve
os-steve marked this pull request as ready for review August 24, 2026 23:14
@os-steve
os-steve added this pull request to the merge queueAug 24, 2026
Merged via the queue into main with commit 50369dcAug 24, 2026
33 checks passed
@os-steve
os-steve deleted the claude/issue-10803-release-coverage-caller branch August 24, 2026 23:38
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/cdsize/lskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[finding] check-release-section-coverage --strict has no standing caller — the advisory half is the only half that runs

2 participants

@os-steve@claude