ci(cuda): gate nvcc peak memory on a budget, and record why -j2 is not headroom

[balbasty]

Follow-up to #87 (which merged at 23:43:58 while this was being written, so it could not go in there). Answers the two questions raised on that PR: is -j2 safe or luck, and does the FFMEM readout survive usefully or does it need a threshold.

MODULES, -O1 and -j2 are unchanged. The split is right; it was the recorded rationale that was wrong, and #87 fixed the number. This fixes the reasoning and makes the measurement a gate.

1. -j2 is luck, not headroom

#87 established the per-module peaks. Feeding the measured wall times through
make's -j2 dispatch order (which is MODULES order) shows the heavy modules
do not take turns — they overlap:

concurrent pair	overlap	peaks sum to
reg_field + reg_field_rls	~760 s	15.59 GB
reg_field + reg_flow	~50 s	21.90 GB
reg_flow + reg_flow_rls	~150 s	14.88 GB

On a 16 GB runner the second is over the limit outright and the first clears it
by ~0.4 GB (2.6%). The build is green anyway because a module's peak RSS is a
brief spike late in its own ptxas phase, and so far two spikes have not landed
in the same instant.

Nothing enforces that. Anything that shifts the relative phase of two heavy
compiles — reordering MODULES, a different nvcc, ccache, a bound/spline policy
change, adding -arch/-gencode targets — reshuffles it, and the failure mode
is an OOM kill that reads as a random infrastructure flake rather than a build
decision. So: my answer to "is it safe?" is no, it is luck, and that is now
written down next to MODULES rather than left as an inference.

Model check: the simulated schedule totals 35.1 min against an observed compile
step of 33.2 min (23:07:15→23:40:29 on run 32311771480), i.e. the real schedule
packs slightly tighter than the model — so if anything there is more overlap,
not less.

2. The readout is now a gate, not a log line

build-cuda fails if any module's peak RSS exceeds FF_MEM_BUDGET_KB (14 GiB),
against a measured maximum of 12.98 GB.

Measuring and only logging would repeat the exact mistake #87 exists to fix: a
number nobody reads is not a gate, in the same way a link without
-Wl,--no-undefined is not a check. The band is ~1 GB; the heavy modules
reproduce to better than 0.1% across the two runs measured, so it will not flap.
If it fires, the intended response is to find out what got heavier — the error
message says so, and points at the measured table.

Logic verified locally against the real FFMEM data before pushing: passes at
14 GiB with the measured table, fails and names the offender when a module
exceeds, and exits 0 gracefully when there is no table at all (a compile that
died before any nvcc finished — an apt stall — is the compile step's failure to
report, not this one's).

What it deliberately does not claim: a per-module budget cannot catch two
modules peaking at once. That risk is scheduling, and the comment says so rather
than implying the gate covers it.

On multi-arch (-gencode), since it was asked

Not touched here — another agent owns it — but the measurement bears on whether
that matrix is affordable, so, flagged rather than assumed:

• The build currently passes no -arch/-gencode at all, so these numbers
  are for one default architecture.
• nvcc compiles per-arch sequentially by default (--threads 1), so I would
  expect additional targets to multiply wall time roughly linearly while
  leaving peak RSS near its current level — the peak is one ptxas on one
  arch. On that expectation reg_flow stays ~13 GB but its 1097 s becomes
  ~2200 s (2 arch) or ~3300 s (3 arch), and the ~33 min compile step becomes
  ~65–100 min against a 120 min job timeout. Time is the binding constraint,
  not memory — but only while nvcc stays single-threaded per arch; --threads
  or a parallel -gencode path would multiply memory instead and reg_flow has
  no room for that.
• This is reasoning from the measurement, not measured — there is no nvcc in
  my container and no multi-arch build has been run. Worth one probe build
  before committing to a cu11/cu12/cu13 matrix.

Validation

CI on this branch. Note the new gate only exercises on a CUDA-triggering change,
which this is (src/lib-cuda/Makefile, .github/workflows/).

As on #87: a green build-cuda means compile + link only. There is no GPU in
CI and nothing here claims anything about GPU runtime behaviour.

---

Generated by Claude Code