From bb90041410e26120a089c9f7c8bf369116b431b9 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 19 Aug 2026 23:44:49 +0000 Subject: [PATCH] ci(cuda): gate nvcc peak memory on a budget, and record why -j2 is not headroom Two follow-ups to the measurement added earlier, both prompted by reading the numbers rather than just printing them. 1. The readout is now a gate. build-cuda fails if any module's peak RSS exceeds FF_MEM_BUDGET_KB (14 GiB), against a measured maximum of 12.98 GB for reg_flow on a 16 GB runner. Measuring and only logging would have repeated the exact mistake this branch exists to fix: a number nobody reads is not a gate, in the same way a link without --no-undefined is not a check. The band is ~1 GB and the heavy modules reproduce to better than 0.1% across runs, so it will not flap; if it fires, the answer is to find out what got heavier, not to raise the number. 2. The -j2 rationale is corrected, not just the per-module figure. Feeding the measured wall times through make's -j2 dispatch order shows the heavy modules do not take turns: reg_field + reg_field_rls overlap ~760 s peaks sum to 15.59 GB reg_field + reg_flow overlap ~50 s peaks sum to 21.90 GB reg_flow + reg_flow_rls overlap ~150 s peaks sum to 14.88 GB The second is over a 16 GB runner outright; the first clears it by ~0.4 GB. The build is green because each module's peak is a brief spike late in its own ptxas phase and two spikes have not yet coincided -- not because two regularisers fit. Nothing enforces that, and anything shifting relative phase (reordering MODULES, a different nvcc, ccache, a bound/spline policy change, adding -arch targets) reshuffles it. That is now stated in src/lib-cuda/Makefile so the next person reads it before raising -j. The per-module budget cannot catch two peaks coinciding; the comment says so rather than implying the gate covers it. MODULES, -O1 and -j2 are unchanged. The split is right; it was the recorded rationale that was wrong. --- .github/workflows/ci.yml | 34 ++++++++++++++++++++++++++++--- src/lib-cuda/Makefile | 43 ++++++++++++++++++++++++++++++---------- 2 files changed, 64 insertions(+), 13 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 08cb2e3..26e11e7 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -584,10 +584,25 @@ jobs: # couple of hundred lines of template warnings, so the table was no # longer anywhere near the end of the log where anyone would look. # - # One line per nvcc invocation, biggest first. Read against the - # ~3.8 GB per-module figure recorded in src/lib-cuda/Makefile: with - # -j2, two concurrent jobs must fit a 16 GB runner. + # One line per nvcc invocation, biggest first, plus a hard budget. + # + # The budget is the point. Measuring and only *logging* would repeat + # the mistake this whole PR is about: a number nobody reads is not a + # gate. The measured maximum is reg_flow at 12.98 GB on a 16 GB runner + # (see the table above MODULES in src/lib-cuda/Makefile); 14 GB leaves + # ~1 GB of warning band, and the measurement reproduces to better than + # 0.1% across runs, so this will not flap. If it fires, something made + # a module materially heavier and the right response is to look at + # why -- NOT to raise the number to make the red go away. + # + # What it cannot do: catch two modules peaking at once. Under -j2 + # reg_field and reg_field_rls overlap for ~760 s and their peaks sum to + # 15.59 GB; that risk is scheduling, not any single module, and no + # per-module budget sees it. if: always() + env: + # 14 GiB, in kB as /usr/bin/time reports it. + FF_MEM_BUDGET_KB: "14680064" run: | table=$(grep -h '^FFMEM ' /tmp/build-cuda.log \ | sed -E 's#^FFMEM ([0-9]+) kB +([0-9.]+) s .*-o ([^ ]+).*#\1 kB \2 s \3#' \ @@ -596,6 +611,19 @@ jobs: { echo '### Peak nvcc RSS per module'; echo; echo '```'; \ echo "$table"; echo '```'; } >> "$GITHUB_STEP_SUMMARY" + # No table at all means the compile died before any nvcc finished + # (an apt stall, a checkout failure); that is the compile step's to + # report, not this one's. + [ -n "$table" ] || { echo "no measurements to check"; exit 0; } + + over=$(printf '%s\n' "$table" | awk -v b="$FF_MEM_BUDGET_KB" '$1 > b') + if [ -n "$over" ]; then + printf '%s\n' "$over" + echo "::error::nvcc peak RSS exceeded FF_MEM_BUDGET_KB=$FF_MEM_BUDGET_KB kB -- see the measured table above MODULES in src/lib-cuda/Makefile before changing this budget" + exit 1 + fi + echo "peak nvcc RSS within budget ($FF_MEM_BUDGET_KB kB)" + # The CUDA impl layer's compile-only probe: tests/impl-cuda/*.cu are # compiled (not run) so that launchers with no other caller -- the mesh # distance ones in particular -- are still type-checked by nvcc. diff --git a/src/lib-cuda/Makefile b/src/lib-cuda/Makefile index 5fc84ba..de2608c 100644 --- a/src/lib-cuda/Makefile +++ b/src/lib-cuda/Makefile @@ -87,21 +87,44 @@ SPLINEFLAGS ?= -DFF_STATIC_SPLINES=0 \ # posdef 0.37 GB 34 s # # (Peak RSS of the largest single process in the nvcc tree -- cicc or ptxas -- -# not a sum: `time` reports ru_maxrss, which is a maximum. Reproducible to -# better than 0.1% across runs.) +# not a sum: `time` reports ru_maxrss, which is a maximum. Across two runs the +# heavy modules reproduce to better than 0.1%; the small ones wander a few +# percent, which is why the budget below is set per module and with room.) # -# What that means, and what it does not: +# (Re-derive with: the `build-cuda` job in .github/workflows/ci.yml, step +# "Peak nvcc memory per module". It wraps nvcc in /usr/bin/time on every run.) +# +# ~~~ AND -j2 IS LUCK, NOT HEADROOM ~~~ +# Worth being blunt about, because the old numbers made it look settled. Feed +# the measured wall times above through make's -j2 dispatch order (which is +# MODULES order) and the heavy modules do NOT take turns -- they overlap: +# +# reg_field + reg_field_rls overlap ~760 s peaks sum to 15.59 GB +# reg_field + reg_flow overlap ~50 s peaks sum to 21.90 GB +# reg_flow + reg_flow_rls overlap ~150 s peaks sum to 14.88 GB +# +# On a 16 GB runner the second of those is over the limit outright and the +# first clears it by ~0.4 GB. The build is green anyway because a module's peak +# RSS is a brief spike late in its own ptxas phase, and so far two spikes have +# not landed in the same instant. Nothing enforces that. Anything that shifts +# the relative phase of two heavy compiles -- reordering MODULES, a different +# nvcc, ccache, a bound/spline policy change, adding -arch/-gencode targets -- +# reshuffles it, and the failure mode is an OOM kill that looks like a random +# infrastructure flake rather than a build decision. +# +# So: do NOT raise -j, do NOT drop -O1, and do NOT read a green build as proof +# that two regularisers fit side by side -- they do not. The `build-cuda` job +# fails if any single module exceeds FF_MEM_BUDGET_KB (14 GB), which turns the +# approach to the ceiling into a diagnosable error instead of a mystery OOM; +# it does not and cannot catch two peaks coinciding. +# +# What is NOT in question: # * The SPLIT IS STILL LOAD-BEARING and must not be undone -- reg_field and # reg_flow are the two heaviest things here even after being halved. -# * The -j2 CEILING IS NOT AS SAFE AS THE OLD NUMBERS SUGGESTED. reg_flow -# alone takes 13 of the runner's 16 GB. -j2 survives because make does not -# happen to overlap the two heaviest peaks, not because two jobs are -# known to fit. Do NOT raise -j on the strength of the old figures, and -# treat anything that makes a regulariser heavier as a real OOM risk. # * Nothing here is caused by the four modules added below; the regularisers # were always the hogs and are untouched by that change. -# Deciding what to do about the reg_flow headroom is out of scope for #80 -- -# it is a pre-existing condition, now measured instead of guessed. +# Fixing the headroom is out of scope for #80 -- it is a pre-existing +# condition, now measured instead of guessed. # # MODULES must list every .cpp in this directory. It did not until # fastfields-lib#80: posdef, resize, restrict and splinc were present as