You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[finding] os-verify-lock.sh's clock-stopped backstop misfires on an ORDINARY full-budget timeout, accusing the clock of a fault the same run's own progress lines disprove #10717
Filed unassigned from the domain:services lane while implementing #10424. Observation only — ⛔ I did not touch the script: three sibling agents were using it during the measurement.
The verdict line, verbatim — this line IS the finding
os-verify-lock: VERDICT queue-timeout (exit 99) · never acquired · spent 540s of a 540s
budget in flock slices while the clock this script polls reported none of it, so the
deadline could never expire · refusing to spin · nothing was built or tested ·
holder pid 18222, held 851s — bash .../issue-10556/gates.sh
That sentence asserts a specific fault: the polled seconds clock stopped advancing. In the run that printed it, the clock was working perfectly, and the same run's own output proves it.
Why the claim is false, from the same run
The 18 progress lines immediately above the verdict decrement monotonically and correctly, one per 30s:
os-verify-lock: waiting: at the head of the queue, 510s of budget left · holder pid 18222, held 341s
os-verify-lock: waiting: at the head of the queue, 480s of budget left · holder pid 18222, held 371s
…
os-verify-lock: waiting: at the head of the queue, 30s of budget left · holder pid 18222, held 821s
Both numbers are computed from now_s(). 510 → 30 in exact 30s steps, and the holder's held counter advancing 341 → 821 in lockstep, is direct evidence that the clock answered every single poll. The backstop then declared that same clock had "reported none of it".
Mechanism — it is an ordering race, not a clock fault
A waiter that reaches the head of the queue immediately spends its whole budget in slices: 18 × 30s = 540s = BUDGET. slices_spent >= BUDGET becomes true at the bottom of iteration 18 — one check earlier than remaining <= 0 would fire at the top of iteration 19. The two conditions become true at essentially the same instant, and the backstop wins the tie by position.
So for a head-of-queue waiter that consumes its full budget, the clock-fault message is not an edge case — it is the default message for an ordinary timeout.
This also explains why the other two timeouts I hit in the same session printed the ordinary verdict instead:
Those runs spent part of the budget in the pre-queue sleep POLL_S phase before reaching the head, so slices_spent was still below 540 when the wall-clock deadline expired. Which branch you get depends only on how much of your wait happened before vs. after reaching the head of the queue — not on the health of any clock.
Measurements
Three queue-timeout exits in one session, one card, all on the shared container:
#
Verdict branch
Holder observed
1
"clock reported none of it" (the misfire)
pid 18222, issue-10556/gates.sh, held up to 851s
2
ordinary waited 540s (9m00s)
pid 1071, issue-10556/gates2.sh, held 425s
3
ordinary waited 540s (9m00s)
pid 1071, issue-10556/gates2.sh, held 1111s
Total dead queueing ≈ 27 minutes. The heavy steps did eventually acquire (VERDICT command-exit 0 · held the lock 327s · waited 138s, and · held the lock 36s · waited 407s), so the lock itself functions.
Why this is worth a card even though nothing is "broken"
The refusal behaviour is correct — it refuses to spin and says nothing was built or tested, which is the right answer and appears to be exactly what the "bound every acquire path" work (PR #10609) set out to guarantee. That part is working.
What is wrong is only the explanation, and it is wrong in the direction that costs the most: it names a cause the evidence contradicts. An agent reading it reasonably concludes the container's clock is unreliable and that waiting is futile in a way no retry can fix. The honest reading is far more mundane and far more actionable: a sibling held the lock for longer than one full budget. Those two readings lead to opposite next moves.
That matters because this lock has already cost a seat a whole agent: the #10009 reconciliation agent queued behind a sibling here, armed a monitor and exited — and a monitor armed by an agent that then exits is not a plan. Its PR sat dirty with a live merge conflict until the PM re-dispatched it. A waiter that can consume its entire budget and then report an unfalsifiable-sounding infrastructure fault is precisely the shape that invites an agent to give up and hand back nothing.
I believe this is a residual of the backstop added for the bash-3.2 hardening rather than a duplicate of any of them, but I did not verify that attribution against the PR diff and triage should not take it from me.
Possible directions (not a decision — for triage)
Check the wall-clock deadline before the slice backstop, so a budget that genuinely elapsed reports the ordinary verdict and the backstop keeps only the case it was built for.
Fire the clock-fault wording only when slice accounting and wall-clock elapsed actually disagree (e.g. slices_spent >= BUDGET while now - started is materially below BUDGET) — that comparison is the thing the message already claims to have made.
Leave the branch and re-word it so it stops asserting a clock fault it has not established.
Option 2 makes the message falsifiable, which is what it is missing. I have no opinion strong enough to override triage.
Filed unassigned from the
domain:serviceslane while implementing #10424. Observation only — ⛔ I did not touch the script: three sibling agents were using it during the measurement.The verdict line, verbatim — this line IS the finding
That sentence asserts a specific fault: the polled seconds clock stopped advancing. In the run that printed it, the clock was working perfectly, and the same run's own output proves it.
Why the claim is false, from the same run
The 18 progress lines immediately above the verdict decrement monotonically and correctly, one per 30s:
Both numbers are computed from
now_s().510 → 30in exact 30s steps, and the holder'sheldcounter advancing341 → 821in lockstep, is direct evidence that the clock answered every single poll. The backstop then declared that same clock had "reported none of it".Mechanism — it is an ordering race, not a clock fault
Constants (
scripts/pm/os-verify-lock.sh):HARD_CAP_S=540,DEFAULT_WAIT_S=540,SLICE_S=30,PROGRESS_EVERY_S=30.In the head-of-queue loop the deadline test sits at the top of the iteration and the slice-accounting backstop at the bottom:
A waiter that reaches the head of the queue immediately spends its whole budget in slices:
18 × 30s = 540s = BUDGET.slices_spent >= BUDGETbecomes true at the bottom of iteration 18 — one check earlier thanremaining <= 0would fire at the top of iteration 19. The two conditions become true at essentially the same instant, and the backstop wins the tie by position.So for a head-of-queue waiter that consumes its full budget, the clock-fault message is not an edge case — it is the default message for an ordinary timeout.
This also explains why the other two timeouts I hit in the same session printed the ordinary verdict instead:
Those runs spent part of the budget in the pre-queue
sleep POLL_Sphase before reaching the head, soslices_spentwas still below 540 when the wall-clock deadline expired. Which branch you get depends only on how much of your wait happened before vs. after reaching the head of the queue — not on the health of any clock.Measurements
Three
queue-timeoutexits in one session, one card, all on the shared container:issue-10556/gates.sh, held up to 851swaited 540s (9m00s)issue-10556/gates2.sh, held 425swaited 540s (9m00s)issue-10556/gates2.sh, held 1111sTotal dead queueing ≈ 27 minutes. The heavy steps did eventually acquire (
VERDICT command-exit 0 · held the lock 327s · waited 138s, and· held the lock 36s · waited 407s), so the lock itself functions.Why this is worth a card even though nothing is "broken"
The refusal behaviour is correct — it refuses to spin and says
nothing was built or tested, which is the right answer and appears to be exactly what the "bound every acquire path" work (PR #10609) set out to guarantee. That part is working.What is wrong is only the explanation, and it is wrong in the direction that costs the most: it names a cause the evidence contradicts. An agent reading it reasonably concludes the container's clock is unreliable and that waiting is futile in a way no retry can fix. The honest reading is far more mundane and far more actionable: a sibling held the lock for longer than one full budget. Those two readings lead to opposite next moves.
That matters because this lock has already cost a seat a whole agent: the #10009 reconciliation agent queued behind a sibling here, armed a monitor and exited — and a monitor armed by an agent that then exits is not a plan. Its PR sat
dirtywith a live merge conflict until the PM re-dispatched it. A waiter that can consume its entire budget and then report an unfalsifiable-sounding infrastructure fault is precisely the shape that invites an agent to give up and hand back nothing.Neighbours checked before filing
scripts/pm/os-verify-lock.shspins forever on bash 3.2 (macOS):mapfile/EPOCHSECONDSare bash 4+/5+, so the lock is never acquired and the command never runs #10289 (closed) — the same script spinning on macOS/bash 3.2 becausemapfile/EPOCHSECONDSare bash 4+/5+. Not this. My run is Linux/bash 5,EPOCHSECONDSis present, and the progress lines prove it answered.os-verify-lock.shnow refuses loudly on stock macOS instead of spinning — but it still cannot take the lock there, because macOS ships noflock#10611 (open) — still cannot take the lock on stock macOS, noflock. Different failure, different platform.-wbudgets differ 6x, andflockgrants to whoever is currently waiting #9661, [finding] os-dev stalls by backgrounding a contended verify job and yielding — measured twice in one session, and concurrency makes it likelier #9687, os-dev agents park on the shared verification flock expecting a wake-up that never comes — the active-wait clause belongs in the role file #8448, [finding] Three mechanizable items from a 16-hour spec-lane shift: os-dev's background-wait stall (6 instances), the verify-lock convention devs invented, and enable_pr_auto_merge's silent no-op #8294 (closed) — starvation and background-wait stalls. Related territory; none covers a misfiring diagnostic branch.I believe this is a residual of the backstop added for the bash-3.2 hardening rather than a duplicate of any of them, but I did not verify that attribution against the PR diff and triage should not take it from me.
Possible directions (not a decision — for triage)
slices_spent >= BUDGETwhilenow - startedis materially belowBUDGET) — that comparison is the thing the message already claims to have made.Option 2 makes the message falsifiable, which is what it is missing. I have no opinion strong enough to override triage.
Generated by Claude Code