diff --git a/.claude/agents/os-dev.md b/.claude/agents/os-dev.md index 0fe8f69fbc..abfd6ba393 100644 --- a/.claude/agents/os-dev.md +++ b/.claude/agents/os-dev.md @@ -66,13 +66,13 @@ JSON,所以终报消息就是 JSON 本身,别无其它。 ## 资源纪律 —— 并行 agents 共享同一个容器 -1. **重活串行 —— 共享验证锁是具名约定。** 每容器一把锁,`/tmp/os-heavy-verify.lock`,包 - 住每次 build/test:`flock -E 99 -w 540 /tmp/os-heavy-verify.lock -c ''`。纪 - 律:**释放归 `flock`**(fd 持有,命令进程树退出即释放 —— 永不手搓 lockfile);只包 - **命令本身**,不包你的阅读与判断;保留 **`-E 99`**,让排队超时与测试失败可区分。 - `-w` 要压在一次前台调用之内(本 harness 单次调用上限 10 分钟),循环重试获取 —— 盲等 - 不能比承载它的调用活得久,为逃上限把它丢后台正是规则 7 要止的停摆。排队是常态,不是 - 挂死。 +1. **重活串行 —— 共享验证锁只有一个入口。** 每次 build/test 都从这里走: + `bash scripts/pm/os-verify-lock.sh -c ''`(或 `-- `)。⛔ 永不手搓 + `flock`/lockfile —— 手写的 `-w` 正是「守约者饿死、越界者通吃」的来源(实测:同一容器 + 五个等待者、三个越界 6 倍;越界者只凭「一直在场」就 12/12 赢走交接)。入口点保证:等待 + 预算钉死在一次前台调用之内(更长的表达不出来)· 按到达顺序授予 · 99 仍专指「没排到」· + 收尾打印本次持锁时长,过长自己喊出来。只包**命令本身**,不包你的阅读与判断;结论读它 + 印的 VERDICT 行,不读裸 `$?`。排队是常态,不是挂死。 2. **压住堆**:重命令前缀 `NODE_OPTIONS=--max-old-space-size=4096`(要抬需给理由)。 3. **定向,不扫全**:只 build/test 受影响的包(`pnpm --filter …`),vitest `--maxWorkers=2`,turbo `--concurrency=2`。 @@ -85,13 +85,13 @@ JSON,所以终报消息就是 JSON 本身,别无其它。 对那个 PID 操作。 6. **整条流水线在前台跑。** build 与 test 都是本任务的步骤:阻塞运行、读真实输出、继续。 ⛔ 永不把验证挂在后台 watcher 上然后停轮(禁令与两种合法终态见「干净收尾」)。唯一合 - 法的长等待是规则 1 的 `flock` 排队——主动、在轮内(规则 7),从不是停轮的理由。 + 法的长等待是规则 1 的锁排队——主动、在轮内(规则 7),从不是停轮的理由。 7. **排队不是停摆 —— 在轮内主动等。** 持锁的是你不拥有的进程,它的完成不会以任何方式唤 醒你:⛔ 永不为「等锁」结束一轮(实测:这么做的每个 agent 都无通知地停摆,赔进一轮探 - 活)。循环:限时获取 ⇒ 退出码 99 时把间隔花在无锁工作上(写测试、changeset、PR 正 - 文、包内 `typecheck`)⇒ 再获取。**排队 ~20 分钟无进展 ⇒ 先看这次检查能否收窄到不必持锁 - (收窄要申报,见「干净收尾」);收窄不了就停下报 `blocked` 并点名持锁者**:`fuser -v /tmp/os-heavy-verify.lock`(或 `lsof`)打印其 PID 与命令 —— 一动不动 - 的持锁者本身就是真发现。报告它;沉默是唯一错误答案。 + 活)。循环:拿到 99 就把间隔花在无锁工作上(写测试、changeset、PR 正文、包内 + `typecheck`)⇒ 再跑一次。**排队 ~20 分钟无进展 ⇒ 先看这次检查能否收窄到不必持锁(收窄 + 要申报,见「干净收尾」);收窄不了就停下报 `blocked` 并点名持锁者** —— `os-verify-lock.sh + --status` 打印持锁者、已持时长与队列;一动不动的持锁者本身就是真发现。沉默是唯一错误答案。 ## Toolchain traps(每条都至少让一个 agent 白跑一轮) diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml index 7faae9e686..00f99a1035 100644 --- a/.github/workflows/lint.yml +++ b/.github/workflows/lint.yml @@ -343,6 +343,28 @@ jobs: - name: Release-rehearsal clone preflight self-test run: node scripts/pm/release-rehearsal-clone.mjs --self-test + # Verify-lock entry-point self-test (#9661). `scripts/pm/os-verify-lock.sh` + # is the ONE way an agent takes the container's shared heavy-verify lock, + # and it is the enforcement of a rule that used to live only in prose: the + # acquisition budget os-dev.md declares is now unrepresentable above its + # cap, grants are ordered by a ticket file, and every run reports how long + # it held. Measured before it existed, in one container: five live waiters, + # three of them 6x over the declared cap, and the compliant ones were the + # ones not verifying — a convention whose violators win is not a convention. + # + # Nothing in CI *uses* the lock (GitHub runners are one job per machine), + # so this step exists to keep the mechanism honest rather than to serialise + # anything: 29 cases over the budget clamp, ticket liveness (dead pid, + # REUSED pid, over-age), fd hygiene, exit-code passthrough, and real + # multi-process contention — queue timeout, legacy free-hand coexistence in + # both directions, and three staggered waiters acquiring in arrival order. + # It runs entirely on a private lock under a temp dir; it never touches + # /tmp/os-heavy-verify.lock, so a runner and an agent container behave the + # same. Unconditional and un-`if:`-ed, like the self-tests above — an + # exemption is what a self-test must not have. No network, ~15 s. + - name: Verify-lock entry-point self-test + run: bash scripts/pm/os-verify-lock.sh --self-test + # Docs/skills authoring guard (#2035 / ADR-0059): TS code blocks in # Markdown/MDX are not type-checked or ESLinted, so skills/ and # content/docs/ can drift back to teaching the bare `: Page = {}` literal diff --git a/scripts/pm/os-verify-lock.sh b/scripts/pm/os-verify-lock.sh new file mode 100755 index 0000000000..b8657ae741 --- /dev/null +++ b/scripts/pm/os-verify-lock.sh @@ -0,0 +1,627 @@ +#!/usr/bin/env bash +# os-verify-lock.sh — the single entry point for this container's shared +# heavy-verify lock (`/tmp/os-heavy-verify.lock`). +# +# scripts/pm/os-verify-lock.sh -c 'pnpm --filter @objectstack/core test' +# scripts/pm/os-verify-lock.sh -- pnpm --filter @objectstack/core test +# scripts/pm/os-verify-lock.sh --status # holder, how long it has held, the queue +# scripts/pm/os-verify-lock.sh --show-budget # the acquisition budget this call would use +# scripts/pm/os-verify-lock.sh --self-test # verify this script +# +# Exit codes: 99 means THIS CALL NEVER ACQUIRED the lock (the same code the +# free-hand `flock -E 99` convention this replaces used, so callers migrate +# without changing how they branch). Anything else is the wrapped command's own +# exit code — including a command that itself exits 99, which is why every run +# ends with a VERDICT line naming which of the two happened. Read the verdict +# line, never a bare `$?`. +# +# --------------------------------------------------------------------------- +# WHY AN ENTRY POINT AND NOT A CONVENTION +# +# The convention it replaces was: `flock -E 99 -w 540 /tmp/os-heavy-verify.lock +# -c ''`, with the 540 explained in prose. Three mechanisms were measured +# in this container while that was the whole mechanism: +# +# 1. WAITER ASYMMETRY — the one that makes obeying the rule a losing strategy. +# `flock(2)` is not FIFO: it grants to whichever waiter happens to be +# blocked when the lock frees, so the DUTY CYCLE of a waiter decides who +# wins, not its arrival. A compliant waiter (`-w 540`) is present for nine +# minutes, times out, goes off to do lock-free work, and comes back — it is +# absent from the queue for part of every cycle. A waiter that wrote +# `-w 3000` is continuously resident for fifty. Measured, one container, +# five live waiters: three exceeded the declared cap, by up to 6x, and the +# two compliant ones were the ones not verifying. One dev burned 68 minutes +# over 9 attempts, every one returning 99, and correctly declared its +# ablation unrun rather than running unlocked. +# +# 2. LONG HOLDERS — a single run held the lock 28+ minutes straight, which +# lengthens every cycle underneath (1). Nothing made that holder visible +# except another agent going and looking with `fuser`. +# +# 3. ORPHANED FD HOLDERS — a backgrounded child inherited the caller's lock +# fd and kept the lock long after the caller was gone (see the lifecycle +# block in scripts/gen-sdui-manifest.sh, where that was diagnosed). +# +# A declared cap that nothing enforces is not a convention when its violators +# win. So the cap moves from prose into the call site: this entry point takes no +# `-w` at all, and the only knob (`OS_VERIFY_LOCK_WAIT`) can lower the budget, +# never raise it. That is (a). Grants are ordered by a ticket file, so presence +# stops deciding winners — (b). And every run reports how long it held, loudly +# past a threshold, so the next long holder names itself instead of waiting to +# be found — (c). +# +# --------------------------------------------------------------------------- +# HOW THE ORDERING WORKS, AND WHAT IT DELIBERATELY DOES NOT DO +# +# `flock` remains the ONLY mutual-exclusion primitive. The ticket queue is +# ADVISORY ORDER layered on top of it and holds no exclusion of its own. That +# split is the whole coexistence story, so it is stated rather than implied: +# +# - Every entry-point call drops a ticket file named by arrival time into +# `.q/`. Only the ticket at the HEAD of the live queue ever calls +# `flock`; everyone else polls. Head-only means no thundering herd, and +# headship is stable — tickets sort by arrival and the ones ahead of you can +# only disappear. +# +# - A LEGACY free-hand `flock` user (an agent still on the old line, or any +# script that locks this file directly) contends on the same file with the +# same primitive. Mutual exclusion is unaffected — it cannot corrupt the +# queue, deadlock it, or run concurrently with an entry-point holder. What +# it can do is win a grant ahead of the queue head, because it never took a +# ticket. So during rollout the guarantee degrades to: entry-point callers +# are FIFO AMONG THEMSELVES, legacy callers behave exactly as they do today, +# and nobody loses exclusion. `--status` still names a legacy holder (via +# `fuser`/`lsof`), it just cannot report its duration — it never registered. +# +# - A waiter that dies, is killed, or times out leaves at most one stale +# ticket, and a stale ticket cannot wedge the queue: tickets are pruned by +# liveness (pid present AND its `/proc` start time unchanged, so a reused +# pid does not resurrect a dead ticket) and by an absolute age bound. If the +# queue directory cannot be used at all, acquisition FALLS BACK to a plain +# capped `flock` with a warning: the ordering layer is best-effort, the cap +# and the exclusion are not. +# +# The lock fd is closed in the wrapped command's child (`{LFD}>&-`), so a +# process the command leaves behind cannot inherit the lock — mechanism 3 above. +# Release still belongs to the fd: this script never hand-rolls a lockfile, and +# a kill -9 at any point releases the lock the moment the process dies. +# --------------------------------------------------------------------------- + +set -uo pipefail + +# The declared invariant, in one place: an acquisition wait must fit inside a +# single foreground agent call (harness ceiling: 10 minutes). Everything above +# this is unrepresentable through this entry point — that is the point of the +# entry point, so it is a constant and not an option. +readonly HARD_CAP_S=540 +readonly DEFAULT_WAIT_S=540 + +# Past this, a holder is loud about itself on release. 15 minutes: the measured +# long holder was 28+, an ordinary targeted package build is well under. +readonly LONG_HOLD_WARN_S="${OS_VERIFY_LOCK_LONG_HOLD_WARN:-900}" + +readonly POLL_S=1 # queue poll while not head +readonly SLICE_S=30 # flock slice while head, so waiting still reports +readonly PROGRESS_EVERY_S=30 + +# `OS_VERIFY_LOCK_FILE` exists so --self-test can run real two-process +# contention without touching the shared lock. Pointing real verification at a +# private lock defeats the serialisation the lock exists for; don't. +LOCK_FILE="${OS_VERIFY_LOCK_FILE:-/tmp/os-heavy-verify.lock}" +QUEUE_DIR="${LOCK_FILE}.q" +HOLDER_FILE="${LOCK_FILE}.holder" +readonly TICKET_MAX_AGE_S=$((HARD_CAP_S + 300)) + +SELF="${BASH_SOURCE[0]}" +TICKET="" +HOLDING=0 +BUDGET_NOTE="" + +log() { printf 'os-verify-lock: %s\n' "$*" >&2; } + +now_s() { printf '%s' "${EPOCHSECONDS}"; } + +# Microsecond arrival stamp, zero-padded so plain lexical (glob) order IS +# arrival order. `EPOCHREALTIME` renders its separator per locale, hence the +# character class; `date` is the fallback if the shell ever stops providing it. +now_stamp() { + local raw="${EPOCHREALTIME:-}" + raw="${raw/[.,]/}" + case "$raw" in '' | *[!0-9]*) raw="$(date +%s%N 2> /dev/null || echo 0)" ;; esac + printf '%020d' "$raw" +} + +# Field 22 of /proc//stat (process start time). Strips through the LAST +# ') ' first: field 2 is the comm, which may itself contain spaces and parens. +proc_starttime() { + local pid="$1" stat rest + [[ -r "/proc/${pid}/stat" ]] || return 1 + stat="$(< "/proc/${pid}/stat")" || return 1 + rest="${stat##*) }" + [[ "$rest" != "$stat" ]] || return 1 + awk '{ print $20 }' <<< "$rest" +} + +# --- budget ----------------------------------------------------------------- + +# Sets BUDGET (seconds) and BUDGET_NOTE. The clamp is the enforcement half: +# a caller asking for more than the cap gets the cap, and gets told. +effective_budget() { + local want="${OS_VERIFY_LOCK_WAIT:-$DEFAULT_WAIT_S}" + BUDGET_NOTE="" + case "$want" in + '' | *[!0-9]*) + BUDGET_NOTE="OS_VERIFY_LOCK_WAIT='${want}' is not a number — using the default ${DEFAULT_WAIT_S}s" + want="$DEFAULT_WAIT_S" + ;; + esac + if ((want > HARD_CAP_S)); then + BUDGET_NOTE="OS_VERIFY_LOCK_WAIT=${want} exceeds the declared cap — clamped to ${HARD_CAP_S}s (an acquisition wait must fit inside one foreground call)" + want="$HARD_CAP_S" + fi + ((want < 1)) && want=1 + BUDGET="$want" +} + +# --- ticket queue ----------------------------------------------------------- + +queue_usable() { + mkdir -p "$QUEUE_DIR" 2> /dev/null || return 1 + chmod 1777 "$QUEUE_DIR" 2> /dev/null || true + [[ -w "$QUEUE_DIR" ]] +} + +# ticket file: "