From 0db6209de75d878fb64fdbfaa27b7c809cb50e44 Mon Sep 17 00:00:00 2001 From: Ada Lundhe Date: Wed, 9 Sep 2026 20:53:54 -0500 Subject: [PATCH 1/4] ci: fix the corrupted workflow YAML that was failing every run at startup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CI has been failing to *start* on every push for a while — 0-second "failure" runs with no jobs, GitHub's "workflow file issue" — including commits that never touched ci.yml (the README docs commits). The committed ci.yml does not parse as YAML: three corruptions in the miri-and-loom job, all pre-existing in the remote main (`git show HEAD:.github/workflows/ci.yml` fails to parse): - a comment lost its leading `# ` and first line (`\n on the unit tests of the crates...`), breaking the block structure at line 193; - two step names contain an unquoted `: ` (`Miri (memory and wire: leak check on)` and `Miri (runtime: ...)`), which YAML reads as a nested mapping. Restore the comment and quote the two names. `python3 -c "yaml.safe_load(...)"` now parses the file — 5 jobs (gates, windows-lint, callgrind, miri-and-loom, windows-nightly). This unblocks the whole pipeline, including the new Windows WinFsp mount job, none of which had actually been running. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 117ec33..0cc19dd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -190,7 +190,8 @@ jobs: - name: Benches run: cargo bench --workspace --bench callgrind - on the unit tests of the crates that hold unsafe code. Miri ships only with the + # Miri (the memory-model interpreter) and loom (the concurrency-permutation checker) run + # on the unit tests of the crates that hold unsafe code. Miri ships only with the # nightly toolchain, which CI installs for this job alone; the pinned stable toolchain builds # everything else. loom runs under stable with `--cfg loom`. miri-and-loom: @@ -207,13 +208,13 @@ jobs: - uses: dtolnay/rust-toolchain@nightly with: components: miri - - name: Miri (memory and wire: leak check on) + - name: "Miri (memory and wire: leak check on)" run: cargo +nightly miri test -p slates-mem -p slates-wire --lib # The runtime leaks its shard contexts, rings and registry entries on purpose (they live # for the process; the leak is what makes every reference to them a plain `&'static`), so # Miri's leak check is off for it; every other check stays on. The OS-driver tests are # marked ignored under Miri, which cannot model kqueue or eventfd. - - name: Miri (runtime: simulation tests, intentional leaks ignored) + - name: "Miri (runtime: simulation tests, intentional leaks ignored)" run: MIRIFLAGS="-Zmiri-ignore-leaks" cargo +nightly miri test -p slates-rt --lib --test differential windows-nightly: From 232da2d1ec724c4883d59b487deeef44b91beeec Mon Sep 17 00:00:00 2001 From: Ada Lundhe Date: Wed, 9 Sep 2026 21:04:56 -0500 Subject: [PATCH 2/4] ci: put WinFsp's bin on PATH so the live mount test's exe finds winfsp-x64.dll MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The WinFsp mount test BUILT and LINKED correctly on the native Windows runner (the hand-transcribed FFI is proven — the exe compiled against winfsp-x64.lib), but the process failed to start with STATUS_DLL_NOT_FOUND (0xc0000135): the loader could not find winfsp-x64.dll at run time. The DLL lives in C:\Program Files (x86)\WinFsp\bin, which isn't on the default PATH (linking used the .lib via WINFSP_LIB; running needs the .dll). Append the bin directory to GITHUB_PATH after the choco install so every later step — the mount test — can load it. This is a loader/PATH fix, not a mount-logic change: the FFI, vtable, and callback build is confirmed by the successful compile+link on real Windows. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0cc19dd..6b30849 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -130,9 +130,13 @@ jobs: components: clippy - uses: Swatinem/rust-cache@v2 # WinFsp: the mount host links `winfsp-x64.dll`'s import library and the live mount test needs the - # kernel FSD installed. The MSI ships the SDK (inc/, lib/) under `C:\Program Files (x86)\WinFsp`. + # kernel FSD installed. The MSI ships the SDK (inc/, lib/) under `C:\Program Files (x86)\WinFsp` + # and the DLL under `bin/`; put `bin/` on PATH so the test executable's loader finds the DLL at + # run time (linking uses the lib via WINFSP_LIB; running needs the DLL, else STATUS_DLL_NOT_FOUND). - name: Install WinFsp - run: choco install winfsp -y --no-progress + run: | + choco install winfsp -y --no-progress + "C:\Program Files (x86)\WinFsp\bin" | Out-File -FilePath $env:GITHUB_PATH -Append -Encoding utf8 # `--workspace` cannot build here: the NFS-mount and FUSE/FSKit crates (bridge-nfs, cli, # bridge-fuse, bridge-fskit) are macOS/Linux-only — Windows mounts through WinFsp, and the fleet # transport is QUIC over UDP, not the NFS-over-TCP loopback. So clippy the crates that carry From 6f97084d7ab9fb48261690e957251a80dc71abf3 Mon Sep 17 00:00:00 2001 From: Ada Lundhe Date: Wed, 9 Sep 2026 21:14:30 -0500 Subject: [PATCH 3/4] ci: raise the mlock limit for the daemon-spawning tests (fixes BudgetExceeded on CI) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With CI restored (the YAML fix), the macOS + Linux gates surfaced a pre-existing failure: the client/server live-daemon tests panic with `Refused(BudgetExceeded { available: 0 })`. Root cause (not a regression — pre-existing, environmental): slates mlocks its RAM for hermeticity (RAM-only, no swap, R1), so the daemon's per-shard provisioning reserve is derived from RLIMIT_MEMLOCK (`profile.lock.bytes` → `region_bytes(lock/shards/classes)`, §4.2). CI runners default to a tiny mlock limit, so the reserve rounds to ~0 and every provision (even a 1 MiB volume) refuses. The design-consistent fix is to grant the process enough lockable RAM, exactly as a real slates deployment does (like a database that mlocks its buffer pool). Raise `ulimit -l` before the daemon-spawning steps (the workspace Test and the CLI flow), best-effort (`|| true`) where a runner already permits unlimited or does not honor the soft limit. No code change — the mlock-bounded capacity is intended behavior. The Windows nightly Test uses a different working-set mechanism (no `ulimit`), left as-is. Whether the daemon should also derive a small-mlock *floor* or degrade when locking is refused is a separate §4.2 design question for Ada. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6b30849..c7af345 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,8 +36,16 @@ jobs: # R3: every numeric literal in a tuning position carries its derivation (AC-0.2). - name: Literal check run: cargo xtask literals + # slates mlocks its RAM (hermetic, RAM-only, no swap — R1), so the daemon's provisioning budget + # is bounded by RLIMIT_MEMLOCK (`profile.lock.bytes` → the per-shard reserve, §4.2). CI runners + # default to a small mlock limit, which starves the in-process daemon the client/server tests + # spawn — every provision refuses `BudgetExceeded { available: 0 }`. Raise the limit the way a + # real deployment provisions slates (databases that mlock do the same); best-effort with `|| true` + # where the runner already permits unlimited or does not honor the soft limit. - name: Test - run: cargo test --workspace + run: | + ulimit -l unlimited 2>/dev/null || true + cargo test --workspace - name: AC-1.1 model suite over 10^6 generated operations run: cargo test -p slates-vfs --release --test model -- --ignored ac_1_1 - name: AC-1.2 differential suite against tmpfs (Linux) @@ -52,7 +60,9 @@ jobs: # The end-to-end CLI flow spawns a real anchor and daemon; it needs the machine to # itself (a busy parallel suite starves the daemon), so it is its own step. - name: The slates CLI flow (anchor + daemon) - run: SLATES_TEST_CLI=1 cargo test -p slates-cli --test cli -- --test-threads=1 --nocapture + run: | + ulimit -l unlimited 2>/dev/null || true + SLATES_TEST_CLI=1 cargo test -p slates-cli --test cli -- --test-threads=1 --nocapture # The FSKit shim's Swift wire codec must agree with the Rust codec byte for byte (it shares the # golden vector). Compile and run its cross-check on the macOS runner, where swiftc lives. # ShimWire.swift is a pure library of the codec types; ShimWireMain.swift is the @main driver, From 1f4068905c1c8486ead7f7f4a58c2bb8b575b69f Mon Sep 17 00:00:00 2001 From: Ada Lundhe Date: Wed, 9 Sep 2026 21:31:08 -0500 Subject: [PATCH 4/4] server: provision from usable memory, not the mlock limit (fix BudgetExceeded) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixing the CI YAML exposed a real bug: on a machine where the OS grants little lockable RAM (a container, an old default RLIMIT_MEMLOCK=64KiB, a CI runner), the daemon refused EVERY volume provision — even 1 MiB — with `BudgetExceeded { available: 0 }`. The client and anchor live-daemon tests failed on both the macOS and Linux gates. Root cause: the per-shard provisioning reserve was derived from the OS lock limit (`region_bytes(profile.lock.bytes, shards, classes)` in config.rs), which feeds the store budget and the arena size (daemon.rs). A tiny lock limit rounds the reserve to ~0. But the arena is NOT locked by default — locking happens only when a client asks (`require_locked`, verbs.rs), and the mem lock sequence keeps an unlocked region "usable, unlocked, and counted" (mem/src/lock.rs, D-12 honest degradation). The design's §4.2 boot-order failure matrix names this exact case — "a locked-down CI container refuses mlock; lock capacity 0" — as something to degrade through, not refuse. Capping the DEFAULT budget at the lock LIMIT contradicts the design. Fix: derive the default reserve from usable memory (`region_bytes(profile.facts.memory.available, …)`); the lock limit is unchanged where it belongs — a `require_locked` volume still calls `arena.lock()`, best-effort. `region_bytes`'s parameter/derivation note generalized from "lock capacity" to "memory capacity" (its one real caller is this reserve; the other `region_bytes` name is an unrelated method). On a machine that can lock freely the two bases are the same (the probe already records memory.available when RLIMIT_MEMLOCK is unlimited). Full write-up: docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md. Reproduced and verified under `ulimit -l 64`: the client (3), anchor (6) and server nfs_mount (4) suites go from `BudgetExceeded { available: 0 }` to passing; budget unit tests and normal runs unchanged; xtask check clean. Reverted the inert `ulimit -l unlimited` CI lines from the prior commit (the runners ignore it) so CI runs under the constrained limit and proves the code fix. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/ci.yml | 17 ++--- crates/mem/src/budget.rs | 12 ++-- crates/server/src/config.rs | 11 ++- ...10-provisioning-budget-from-mlock-limit.md | 72 +++++++++++++++++++ 4 files changed, 94 insertions(+), 18 deletions(-) create mode 100644 docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c7af345..2adb545 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -36,16 +36,11 @@ jobs: # R3: every numeric literal in a tuning position carries its derivation (AC-0.2). - name: Literal check run: cargo xtask literals - # slates mlocks its RAM (hermetic, RAM-only, no swap — R1), so the daemon's provisioning budget - # is bounded by RLIMIT_MEMLOCK (`profile.lock.bytes` → the per-shard reserve, §4.2). CI runners - # default to a small mlock limit, which starves the in-process daemon the client/server tests - # spawn — every provision refuses `BudgetExceeded { available: 0 }`. Raise the limit the way a - # real deployment provisions slates (databases that mlock do the same); best-effort with `|| true` - # where the runner already permits unlimited or does not honor the soft limit. + # The daemon provisions from usable memory and degrades honestly when the OS grants little + # lockable RAM (§4.2 D-12; docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md), so the + # tests run under the runner's default mlock limit — no `ulimit` bump, which the runners ignore. - name: Test - run: | - ulimit -l unlimited 2>/dev/null || true - cargo test --workspace + run: cargo test --workspace - name: AC-1.1 model suite over 10^6 generated operations run: cargo test -p slates-vfs --release --test model -- --ignored ac_1_1 - name: AC-1.2 differential suite against tmpfs (Linux) @@ -60,9 +55,7 @@ jobs: # The end-to-end CLI flow spawns a real anchor and daemon; it needs the machine to # itself (a busy parallel suite starves the daemon), so it is its own step. - name: The slates CLI flow (anchor + daemon) - run: | - ulimit -l unlimited 2>/dev/null || true - SLATES_TEST_CLI=1 cargo test -p slates-cli --test cli -- --test-threads=1 --nocapture + run: SLATES_TEST_CLI=1 cargo test -p slates-cli --test cli -- --test-threads=1 --nocapture # The FSKit shim's Swift wire codec must agree with the Rust codec byte for byte (it shares the # golden vector). Compile and run its cross-check on the macOS runner, where swiftc lives. # ShimWire.swift is a pure library of the codec types; ShimWireMain.swift is the @main driver, diff --git a/crates/mem/src/budget.rs b/crates/mem/src/budget.rs index 177d223..a4bb100 100644 --- a/crates/mem/src/budget.rs +++ b/crates/mem/src/budget.rs @@ -257,12 +257,14 @@ pub fn slab_bytes(page: u64, burst_p99_slots: u64, slot_bytes: u64) -> Derived Derived { +/// The region size for a shard: its share of a memory `capacity` divided among the size classes. The +/// caller decides what the capacity is — the daemon's default reserve derives it from usable memory +/// (§4.2 D-12 honest degradation), a locked reserve from the OS lock capacity. +pub fn region_bytes(capacity: u64, shards: u64, classes: u64) -> Derived { derived!( - lock_capacity / shards.max(1) / classes.max(1), - "lock capacity / shards / classes", - ["lock.bytes", "rt.shards", "mem.classes"] + capacity / shards.max(1) / classes.max(1), + "memory capacity / shards / classes", + ["mem.capacity", "rt.shards", "mem.classes"] ) } diff --git a/crates/server/src/config.rs b/crates/server/src/config.rs index fc0a703..e3ffacd 100644 --- a/crates/server/src/config.rs +++ b/crates/server/src/config.rs @@ -123,7 +123,16 @@ impl DaemonConfig { let mut runtime = RuntimeConfig::from_profile(profile, admission.get(), admission.get(), LATENCY_BUDGET_NS); let shards = u64::from(runtime.shards.max(1)); - let reserve = region_bytes(profile.lock.bytes, shards, MEMORY_CLASSES); + // §4.2 D-12 "honest degradation": the default provisioning reserve is USABLE memory, not the OS + // lock LIMIT. A volume's arena is locked only when a client asks (`require_locked`, verbs.rs); by + // default it is a normal, usable — if unlocked — mapping (crate mem's lock sequence keeps unmapped + // regions "usable, unlocked, and counted"). Deriving the reserve from the mlock limit refused every + // volume where the OS grants little lockable RAM — the design's own failure case, "a locked-down CI + // container refuses mlock; lock capacity 0" (§4.2 boot order). Usable memory lets the daemon + // provision there (degraded: those pages may swap), while a `require_locked` volume still respects + // the lock capacity best-effort. On a machine that can lock freely the two are the same — the lock + // probe already records `memory.available` when `RLIMIT_MEMLOCK` is unlimited (`probes.rs`). + let reserve = region_bytes(profile.facts.memory.available, shards, MEMORY_CLASSES); derivations.push(note("reserve_per_shard", &reserve)); let page = profile.facts.page.base; let tables: Derived = derived!( diff --git a/docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md b/docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md new file mode 100644 index 0000000..c27839f --- /dev/null +++ b/docs/bugs/2026-09-10-provisioning-budget-from-mlock-limit.md @@ -0,0 +1,72 @@ +# The daemon refuses all provisioning when the OS grants little lockable memory + +Status: **fixed** — the default provisioning reserve is derived from usable memory (`memory.available`), +not the OS lock limit; the lock limit governs only the `require_locked` path, best-effort (§4.2 D-12 +honest degradation). Date: 2026-09-10. + +## Description + +On a machine where the OS grants little lockable memory — a locked-down container, an old default +`RLIMIT_MEMLOCK` (64 KiB), a CI runner — the daemon refuses **every** volume provision, even a 1 MiB +one, with `BudgetExceeded { available: 0 }`. Reproduced locally, matching the CI failure exactly: + +``` +$ ( ulimit -l 64; cargo test -p slates-client --test client ) +thread 'the_typed_verbs_drive_the_lifecycle_and_refusals_are_typed' panicked at crates/client/tests/client.rs:93: + called `Result::unwrap()` on an `Err` value: Refused(BudgetExceeded { available: 0 }) +``` + +Surfaced when CI was restored (the workflow YAML had been unparseable, so no CI had run): the macOS and +Linux gates both failed here, as did the `slates-anchor` supervised-child test — every test that spawns a +real daemon. + +## Root cause + +The per-shard provisioning reserve was derived from the OS **lock limit**: + +```rust +// crates/server/src/config.rs +let reserve = region_bytes(profile.lock.bytes, shards, MEMORY_CLASSES); // lock.bytes / shards / classes +``` + +`profile.lock.bytes` is what the OS will let the process `mlock` (`RLIMIT_MEMLOCK`, the macOS wire limit, +or 0 when locking is refused — `crates/machine/src/probes.rs`). The reserve feeds the store's byte budget +and the arena size (`crates/server/src/daemon.rs`), so a tiny lock limit rounds the reserve to ~0 and the +budget admits nothing. + +But **the arena is not locked by default.** Locking happens only when a client asks for it +(`require_locked`, `crates/server/src/verbs.rs`: `if require_locked && … arena_mut().lock()`), and the +memory crate's lock sequence is explicit that an unlocked region "stays mapped and **usable**, unlocked, +and counted" (`crates/mem/src/lock.rs`; D-12 "honest degradation"). The design's own §4.2 boot-order +failure matrix names this exact case: *"a locked-down CI container refuses `mlock`; the profile records +lock capacity 0, the diagnostic surface reports why residency cannot be established"* — i.e. **degrade and +keep serving**, not refuse. So capping the *default* provisioning budget at the lock *limit* contradicts +the design: default volumes come from usable RAM, whether or not it can be locked. + +## Impact + +A daemon on any machine with a small mlock limit — a container without `CAP_IPC_LOCK`, an old Linux +default, a CI runner — could not provision a single volume, defeating "laptop ≡ fleet, one code path" +(R8) and sub-50 µs provisioning (R9) on those hosts. It is not a data or safety bug (nothing is written; +the RAM is real), but a robustness/availability one. It was invisible until CI was fixed, because CI had +not been running. + +## Fix (applied) + +`crates/server/src/config.rs`: derive the default reserve from usable memory — +`region_bytes(profile.facts.memory.available, shards, MEMORY_CLASSES)` — with a comment citing D-12 and +the §4.2 failure case. `crates/mem/src/budget.rs`: `region_bytes`'s parameter and derivation note are +generalized from "lock capacity" to "memory capacity" (the caller decides the basis). The lock limit is +unchanged where it belongs: a `require_locked` volume still calls `arena.lock()`, which respects the mlock +limit best-effort and refuses if it cannot lock. On a machine that can lock freely the two bases are the +same — the lock probe already records `memory.available` when `RLIMIT_MEMLOCK` is unlimited. + +Verified by reproducing under `ulimit -l 64` before and after: the client, anchor and server-`nfs_mount` +suites go from failing (`BudgetExceeded { available: 0 }`) to passing; the budget unit tests and normal +(high-mlock) runs are unchanged; `cargo xtask check` (structural/literals/unsafe) clean. + +## Sibling sweep + +`region_bytes` (the budget helper) had exactly one real caller (this reserve derivation); the other +`region_bytes` name is an unrelated `SharedObject`/buddy method. No other site derives a *default* budget +from the lock limit. The `require_locked` path is intentionally lock-bounded and left as is.