Skip to content

feat(pm): record a passive boot marker on every verify-lock invocation, so cross-restart evidence about /tmp accumulates by itself - #13145

Open
os-elon wants to merge 4 commits into
mainfrom
claude/issue-12919-boot-marker
Open

feat(pm): record a passive boot marker on every verify-lock invocation, so cross-restart evidence about /tmp accumulates by itself#13145
os-elon wants to merge 4 commits into
mainfrom
claude/issue-12919-boot-marker

Conversation

@os-elon

Copy link
Copy Markdown
Collaborator

Part of #12919. It builds the mechanism the ruling ordered and deliberately writes none of the card's conclusions, so the card stays open to receive readings — that is the half left behind, and it is left behind on purpose.

The ruling, and the one word in it that points the wrong way

The maintainer ruling (2026-08-29) says "adopting option B (piggyback passive measurement)", but triage's own table labels A as the passive marker and B as the active multi-container fleet campaign. The ruling's description is unambiguous in the other direction — "piggyback passive measurement", "no dedicated multi-container restart campaign", "the measurement happens because of what gets typed, not because someone remembers to run it". This PR implements the description: the passive marker. No fleet campaign, which would need explicit authorisation in any case.

First, the thing triage said had to be measured before the design

Triage's confidence gap 3: if the ledger rolls faster than the restart interval, a passive marker is truncated before it is ever read. Measured on this container before anything was written:

measured
LEDGER_MAX_BYTES8388608 bytes (8 MiB), a readonly constant — not env-overridable
live ledger150 records / 31547 bytes spanning 84056 s (23h20m)
bytes per record210 measured (the source comment says ~150)
accumulation rate1351 bytes/hour, 6.4 records/hour
time from empty to the bound at that rate~6209 hours ≈ 259 days
restart interval on this boxthe same ledger spans a boot boundary: 128 records before the derived boot, 22 at or after it, with the derived boot 81 minutes old

So the bound sits roughly three orders of magnitude away from the restart interval. The hazard triage flagged does not bite at the observed rate.

⚠️The shape of that hazard is also wrong, and that matters for where the marker goes.ledger_append returns early when the file exceeds the bound — it does not trim, roll or rotate. Nothing is truncated; new records simply stop being written. So the failure mode a marker in the ledger would have had is not "truncated before it is read", it is "never written after the bound" — silently, at exactly the point the population became long enough to be worth reading. Same remedy, different reason, and the reason is now in the code.

The mechanism

A second file beside the lock, os-heavy-verify.lock.boots, plus a fixed-name probe. boot_marker_observe runs in main on every invocation — a run, a refusal, a --status, even a usage error — before dispatch. It appends one record per distinct boot, never one per run.

Where it lives is the measurement. The marker sits in the same directory as the ledger it is about, so it is itself subject to whatever happens to that directory across a reset. A marker written somewhere known-durable would measure nothing. It satisfies all three: no size bound of this file's can stop it being written, normal operation writes it without anyone remembering, and it is exposed to the phenomenon under study.

The boot identity is the kernel's, not a clock subtraction./proc/sys/kernel/random/boot_id is exact, needs no tolerance, and jitters not at all — the derived boot is still recorded, as the quantity the two existing readings used, but it is no longer the key. Where no boot id exists the key falls back to the derived boot with a tolerance and the field records -, so a reader knows which key was used.

Both controls travel on every record, because a reading without its controls is not a reading:

  • positivepos=: a file written at that instant, read back through the same stat and compared against the same derived boot, must read after. Anything else means the comparison was not live for that reading and its markerpre/ledgerpre are worth nothing.
  • negativebtime= and pid1after=: the kernel's own record of when it booted, and how long after the derived boot PID 1 started. A small pid1after says the process tree restarted with the clock, so a file older than the boot means the filesystem did not restart with them. It can read slightly negative from whole-second rounding; what the control says is that the number is small in either direction.

The strongest field is prev=, and it carries no timestamp at all. It names the boot identity on the previous line of the same file. If it names a different boot, this file was present before that restart and is being appended to after it — evidence by content, with no clock comparison anywhere in it. markerbirth/markerpre and ledgerbirth/ledgerpre corroborate it from the timestamp side and reproduce the exact row the two existing readings were filed over.

Cost, measured and decomposed

It runs on every invocation in every agent container, so #12783's fence binds: nothing that outlives a process and needs reaping, and nothing that can redden a gate.

A/B against the same file without the mechanism, 100 invocations per arm, three interleaved rounds, worst case (--show-budget, which does nothing else):

ms per invocation
total added1.20 – 1.47
of which: bash parsing the added source~1.03 – 1.23
of which: the mechanism itself~0.23 – 0.80

The dominant term is bash parsing 691 added lines, most of them comment — a cost this file pays for any comment. Against a run that holds this lock for tens to hundreds of seconds that is under one part in a hundred thousand. Shared-box milliseconds, per this file's own doctrine.

The fast path forks nothing on bash 5.0+: /proc/uptime and the boot id are read with the read builtin, the marker with one builtin read. Counted rather than claimed — comments stripped and arithmetic $(( excluded, the four fast-path functions contain exactly one command substitution, the now_s fallback reached only below bash 5.0; the control, the same filter over the once-per-boot path, counts 12.

Nothing here needs reaping. Two fixed paths, exactly as .holder and .ledger already are: the marker is append-only and bounded at 65536 bytes, the probe is overwritten in place and never grows. Neither accumulates a file per process, which is what the fence was against. Every write is best-effort and every failure swallowed; a self-test case pins that an unwritable marker loses boots, never runs, and a second pins that it stays quiet about it.

Concurrency

Several agents in one container invoke this concurrently. Appends are a single short printf to an O_APPEND fd, well under PIPE_BUF, atomic on Linux — the same argument the ledger's format already rests on. The read-then-append sequence is not atomic and deliberately takes no lock: a lock round-trip here would cost more than the measurement. Two runs that both notice the same new boot can each append a record for it; that is benign, stated rather than prevented, and --report groups by boot identity so a duplicated boot reads as one boot — and a duplicate carries its own controls anyway.

⚠️ One existing pin was narrowed. Read this part.

--self-test case "and a kill -9 mid-run leaves nothing behind to reap either" asserted that nothing at all sat beside the lock path after an unlocked run. It went red, correctly, the first time the marker existed. That case's own comment states the invariant it protects: a hand-rolled lockfile is the option the 2026-08-22 ruling rejected, and a killed holder leaving the lock held is its whole cost. The blanket glob was a convenient spelling of that back when the only files that could be beside the lock were lock state.

The marker is not lock state: nothing reads it to decide whether the lock is held, a kill -9 leaves it valid because an append either landed whole or did not land, and reaping it would destroy the measurement rather than tidy anything. So the assertion is re-spelled to the property it was protecting — no lock state survives an unlocked run — and paired with a positive that counts the residue and requires it to be exactly the marker pair, because a narrowed assertion that counts only what it chose to look at is how a pin stops catching the regression it exists for. The run_unlocked header now says the claim is about that function, not about the invocation.

What it can and cannot conclude — printed with the numbers, not filed beside them

--report gained a boot section that ends in a limits block, and --status names the marker so a reader meets it:

  • CAN: that this marker file was present before a restart and after it, once per counted crossing — a positive fact about one path on one box, carried by content, with both controls per reading.
  • CANNOT: that /tmp survives a reset. A wipe takes the file with it, so on-box a wipe is indistinguishable from a first run on a fresh container — both read as a marker whose only record says prev=-. The negative direction is visible only off-box, comparing a reading carried away earlier against a later one from the same machine=. That is why machine= is recorded at all.
  • CANNOT: that any of it generalises. One file, one path, one container, and only the restart kinds that box happened to perform.
  • What would warrant the dated line. Not a count of same-direction observations: two disqualified a mechanism and did not establish its negation, and twenty would not either. What is establishable is a bound — with k observed restarts all survived, the one-sided 95% upper bound on the wipe rate is 1-0.05^(1/k): k=10 gives ~26%, k=20 ~14%, k=30 ~9.5%. So: at least 10 crossings on each of at least 3 distinct machine= values, and the line phrased as that bound, never as "/tmp survives". One observed non-survival settles the other direction at k=1.

⛔ What was deliberately not touched

The "the reason it sits there is left unasserted" wording in the --report scope block, and the "AND THE OPPOSITE PREMISE IS NOT CLAIMED" paragraph in the slots block. Two same-direction observations disqualify a mechanism; they do not establish its negation, and this PR adds a third same-direction observation without touching either sentence. Proven byte-for-byte rather than asserted — both blocks extracted from the base commit and from this branch and hashed:

block [WHAT USED TO STAND WHERE THAT FLOOR STANDS] base 24 lines 26a367982e09cb3b branch 24 lines 26a367982e09cb3b IDENTICAL
block [THE POPULATION, AND WHERE ITS FLOOR ACTUALLY IS] base 34 lines 170153d4d92b8dcb branch 34 lines 170153d4d92b8dcb IDENTICAL

Nor was platform-readings.md touched: that line is the future step, and this round builds the mechanism rather than writing a conclusion. Nor was the scope of #12795 entered — whether CPU-heavy gate runs should be routed through this lock is that card's question and is untouched here.

The first datapoint

Taken from this container the moment the mechanism existed, in the format of the two readings already on the card. It is a third same-direction observation of the ledger outliving a reported boot — recorded automatically this time, with both controls, and carrying no conclusion.

1787982935 bootid=709d073eb010 boot=1787976782 up=6153 btime=1787976782
pid1after=0.46 prev=- prevboot=- markerbirth=- markerpre=- ledgerbirth=1787897554
ledgerpre=yes pos=after fs=ext2/ext3 machine=0d0af05ee8fd pid=18059

markerbirth/markerpre read - on a container's first record, and correctly: the marker is created by that very append, so at that moment there was no marker to date. It says the same thing prev=- says. The full table is on the card.

Verification

Union re-run after the final commit, at 05e7d98bfe. Exit codes captured before any pipe; each gate's own verdict line quoted rather than a bare status.

gateresult
bash scripts/pm/os-verify-lock.sh --self-test✓ os-verify-lock self-test: all cases pass.225 cases, up from 195 on the base commit, measured by running the base tree's own suite in a comparison worktree rather than by recalling a number. 33 cases added in source; 2 of them belong to the no-/proc/uptime branch this host does not take, and one replaced case nets +1
pnpm check:bash32-floor✓ check-bash32-floor: 22 tracked shell file(s) ... name no bash 4+ construct
pnpm check:entry-guard✓ check:entry-guard: 177 scripts/ file(s)
pnpm check:parse-guard✓ check:parse-guard: 176 scripts/ file(s)
pnpm check:pnpm-filter-targets✓ check:pnpm-filter-targets: 140/177 --filter occurrence(s)
pnpm check:agent-test-spelling✓ check-agent-test-spelling: 0 violations
pnpm check:cli-command-ids✓ check-cli-command-ids: 294 command-id literal(s)
pnpm check:watch-hint-literal✓ check-watch-hint-literal: 18 ROOT_DIR_WATCH_HINTS declaration(s)
pnpm check:cross-package-test-inputsOK: 23 package(s) read outside themselves
pnpm check:nul-bytescheck-nul-bytes: OK (scanned 7288 text file(s) ... no raw ASCII control bytes)
node scripts/check-ci-filter-parity.mjsOK: all 121 declared cross-package glob(s)
node scripts/check-shard-attestation.mjs✓ check-shard-attestation: 2 aggregate gate(s)

Gate family derived with node scripts/pm/dispatch-gates.mjs --repo objectstack-ai/objectstack after merging origin/main, so the list is not stale: 12 families matched. The twelfth, check-test-completeness.mjs, answers a standalone run with exit 1 and a usage line — an invocation error, filed as #13110, not a gate result and not touched here.

pnpm lint was narrowed, and the narrowing is measured rather than assumed. ① The population comes from eslint's own config, not from a guess: run against the changed path it answers "File ignored because no matching configuration was supplied". ② The count comes from --format json: 1 result, 0 errors, and the sole message is that ignore notice; the control, a real .mjs path, returns 1 result with 0 errors and no messages, so the instrument distinguishes "linted clean" from "not in the population". ③ Invariance for untouched files: eslint.config.mjs carries no parserOptions.project and no typed rules, so a diff touching no JS or TS file cannot move any untouched file's verdict. CI runs the full sweep regardless.

No changeset, derived rather than asserted.pnpm check:published-files is green over 69 publishable packages, and a sweep of all 75 workspace manifests finds zero whose files whitelist mentions scripts — so scripts/pm/os-verify-lock.sh reaches no tarball and nothing here is user-visible. The PR carries skip-changeset.

Generated by Claude Code


Generated by Claude Code

@os-elonos-elon added the skip-changeset PR has no user-facing published change; bypasses the changeset gate label Aug 29, 2026 — with Claude
@os-elon
os-elon marked this pull request as ready for review August 29, 2026 06:42
@os-elon
os-elon enabled auto-merge August 29, 2026 06:42
@os-elon
os-elon added this pull request to the merge queueAug 29, 2026
@github-merge-queue
github-merge-queueBot removed this pull request from the merge queue due to failed status checks Aug 29, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⛔ merge queue 构建失败 — 先分诊,再决定要不要重排

队列构建 33241594487 红了。队列跑的是全量套件(PR 侧 CI 只跑 affected 子集),
所以失败的测试可能在本 PR 没碰过的包里 —— 那不是重排能修的。每次盲目重排都会让排在后面的所有 PR 重建一轮。

失败的 job(日志抽取,best effort):

  • Test Core (1/6) — 失败步骤: Run this shard's tests

    @objectstack/cli:test: FAIL test/serve-publishes-bound-port.e2e.test.ts > #13062 `os serve --port 0` — the request that can never be the answer > announces the BOUND port on all three channels, and ↳ 失败原因: @objectstack/cli:test: Error: ENOENT: no such file or directory, open '/tmp/os-bound-port-home-oNo2Gv/runtime.env_local.json'
    

↳ 失败原因 是判读的关键:超时Test timed out in … / Hook timed out in …)多半是负载/时序,不是本 PR 的回归;
断言AssertionError: …)才指向真实的行为改变。两者的 FAIL 行长得一模一样,只有这一行能区分。

跨 PR 相同签名(24h,按失败测试文件聚合):

历史信号:

  • 本 PR 过去 24h 无队列失败记录(首次)。
  • 过去 24h 队列共有 7 个失败构建(不含本次)。

分诊清单:

  1. 失败测试在本 PR 改动的包里 → 真回归,修 PR。
  2. 失败测试与本 PR 无关 → 看上面的「跨 PR 相同签名」;已有汇总 issue ⇒ flaky/环境问题实锤,去那张 issue 上谈,修好前重排只会再烧一轮全队列。
  3. 两者都不是 → 可能与同组 PR 语义冲突;等前面的 PR 落地或失败出队后再重排一次即可,不要连续重排。

Generated by Claude Code · merge-queue-triage workflow (#4859)

This was referenced Aug 29, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/lskip-changesetPR has no user-facing published change; bypasses the changeset gate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@os-elon@claude