perf(cuda): split reg_flow into per-(family, ndim) translation units

[balbasty]

Brought up to current main (247714e, through #153, #154 and #155) by merging rather than rebasing, so the push stays a fast-forward. src/lib-cuda/ had not been touched on main since this branch last merged, so the merge was clean, and the only code change since the last review is the one the review asked for.

What the review asked for

"ok but should use pragma once and the namespace macros"

#pragma once — already satisfied: both new headers (reg_flow_slice.h, reg_flow_slice.inl) open with it on line 1 and carry no #ifndef guard.

The namespace macros — was not satisfied, in exactly one file. reg_flow_slice.h spelled its namespaces by hand:

namespaceff {
namespacecuda {
namespaceflow_slice {

It is now FF_NAMESPACE_BEGIN(FF_NS) / (FF_DEVICE) / (flow_slice), pulling in core/cuda_switch.h for FF_DEVICE. The .inl and the twelve slice TUs already used the macros, so this was the odd file out rather than a pattern.

All four checkers pass --check: normalise-header-guards.py, normalise-include-delimiters.py, rename-macros.py (#145, #146, #91) and check-cuda-launches.py — and the lint (source conventions) job #156 has since added is green on this branch. I also confirmed the guard checker genuinely covers these two files rather than passing vacuously: injecting a whole-file guard into each makes it fail and name the file. check-cuda-launches.py matters here specifically because #154 landed the one-launch-site rule while this branch was open — the slice TUs launch nothing directly, they call the FF_CUHOST launchers in impl/cuda/reg_flow.h, and the lint still reports one <<< and 39 FF_CUDA_LAUNCH call sites.

Re-measured against #155 — and this is the part worth reading

#155 is why every number below was re-taken. It fixed nvcc replacing the body of every impl/kernels/utils.h helper in the host object with ::exit(1), with -O1 then deleting everything after the call. The first measurement of this split was taken against a library in that state.

The honest question is whether the split's case survives the fix. It does — and the finding is that the fix did not move the compile at all:

reg_flow, one TU, index32	peak RSS	nvcc elapsed
pre-#155 (run 32365351378, 85fdac7)	12.98 GiB	1149.89 s
post-#155 (run 32397332659, 0d40731)	12.98 GiB	1069.30 s

The peak is identical. That is not luck: the code #155 restored is host code, and the peak belongs to a device-side process (cicc/ptxas), so deleting host statements never touched the number that pins this build. A correctness catastrophe and a compile-cost non-event.

Every figure below is nonetheless from post-#155 runs on both sides — BEFORE 0d40731 (run 32397332659), AFTER 2c4842b (run 32418602148) — same image, same nvcc 12.0.140, same -O1, same bound/spline policy, -j2.

index32 — the default, and what ships

	before	after
TUs for reg_flow	1	13
heaviest reg_flow TU	12.98 GiB	1.94 GiB	−85.0%
reg_flow elapsed, summed	1069.30 s	803.32 s	−24.9%
make cuda -j2 wall	2145 s	1861 s	−13.2%
max peak over any module	12.98 (reg_flow)	8.35 (reg_field)

Per slice, from the job's own FFMEM table:

reg_flow_matvec_3d 1.94 GiB 177.13 s reg_flow_matvec_1d 0.61 48.84
reg_flow_matvec_2d 1.22 GiB 98.53 s reg_flow_relax_2d 0.60 48.01
reg_flow_diag_3d 1.03 GiB 64.72 s reg_flow_kernel_2d 0.59 49.71
reg_flow_relax_3d 0.96 GiB 89.84 s reg_flow_kernel_1d 0.48 38.77
reg_flow_kernel_3d 0.89 GiB 83.19 s reg_flow_diag_1d 0.44 35.04
reg_flow_diag_2d 0.67 GiB 46.24 s reg_flow_relax_1d 0.26 21.66
reg_flow (front) 0.20 1.64

index64

	before	after
heaviest reg_flow TU	5.69 GiB	1.05 GiB	−81.5%
reg_flow elapsed, summed	482.78 s	455.23 s	−5.7%
make cuda -j2 wall	966 s	918 s	−5.0%
max peak over any module	5.69 (reg_flow)	3.76 (reg_field)

The elapsed saving is far smaller on this leg, which is the expected shape: collapsing the offset axis has already removed half the instantiations, so there is less left for the split to divide. Memory still falls by four fifths.

Two corrections to the earlier numbers

I am flagging these rather than quietly restating them.

1. The time savings are smaller than first reported — −24.9% and −13.2%, not −34.6% and −25.5%. Same change, different runs.
2. Wall-clock on this job is noisy and peak RSS is not.reg_field, which this PR does not touch, moved 8.23 → 8.35 GiB and 848.87 → 861.05 s between the same two runs. So read −13.2% as "clearly faster", not as a figure good to three digits. Peak RSS, by contrast, is solid: all thirteen per-slice peaks reproduce the earlier pre-fix(kernels): utils.h helpers must be host+device, not device-only (#150) #155 run of the same split to within 0.01 GiB, and reproduce off-runner — measured locally on a different machine at the same flags, the front TU came out 205532 kB against CI's 205332 (0.1%) and reg_flow_diag_1d 456904 against 456160 (0.2%).

What is here

file	role
reg_flow.cpp	all 15 exported entry points, every check, every message — and zero instantiations
reg_flow_slice.h	the seam: 30 hidden-visibility declarations, one per (entry point, ndim)
reg_flow_slice.inl	the five wrapper templates + the dtype × offset × bound dispatch, moved verbatim; bodies selected by FF_FLOW_SLICE_*
reg_flow_{matvec,diag,kernel,relax}_{1,2,3}d.cpp	twelve four-line slice TUs

The cut is (operation family) × (ndim) because that is how the dispatch already factors: each entry point picks one internal wrapper template and switch (ndim)es it, and no two arms share an instantiation.

Why a forwarding seam, and not extern template or -D

Compiling one source N times under different -D — what BOUNDFLAGS does — gives N objects each defining ff::cuda::flow_matvec, and the link fails. So the entry point lives in one TU and the slices export something else.

extern template is not usable here: it must name every leaf, and with FF_STATIC_BOUNDS=0 six of the eight FF_BOUND_* selectors collapse onto bound::type::Dynamic, so the enumeration would name the same specialization six times — ill-formed under [temp.explicit]/5 — and which leaves collapse is a build-flag decision. A switch has no such problem: duplicate implicit instantiations across its arms are the same instantiation.

ABI — unchanged, and checkable per object

The slice functions are declared __attribute__((visibility("hidden"))), so they never reach .dynsym:

$ readelf -sW reg_flow_diag_1d.o | grep flow_slice
3 GLOBAL HIDDEN ff::cuda::flow_slice::{diag,adddiag,subdiag}_1d
$ readelf -sW reg_flow_relax_1d.o | grep flow_slice
1 GLOBAL HIDDEN ff::cuda::flow_slice::relax_1d

GLOBAL HIDDEN is the whole ABI argument: external linkage so the call in reg_flow.cpp resolves at link time, hidden visibility so the dynamic symbol table of libfastfields-cuda.so is what it was. The exported ff::cuda::flow_* entry points, their signatures, their argument checks and the order those fire in are untouched — the bodies changed file, not shape. (The demangling above is incidentally also what confirms FF_DEVICE still expands to cuda after the namespace-macro change.)

The counter-force, measured rather than assumed

Every slice re-parses the same template headers, so total elapsed could have risen. reg_flow.cppis that measurement — it parses impl/cuda/reg_flow.h plus the kernels and instantiates nothing, at 0.20 GiB and 1.64 s. Twelve extra TUs of re-parsing is therefore ~20 s against a 1069 s baseline, under 2%, an order of magnitude below the instantiation work that now divides.

It is a ratio, not a law: a module whose per-TU instantiation work is comparable to ~1.6 s should not be split.

What this does not buy: a higher -j

The ceiling is max(peak) over every module. reg_field (8.35 GiB) and reg_field_rls (7.07 GiB) are untouched, so reg_flow stops being the constraint and reg_field becomes it — and their 15.42 GiB overlap against a 16 GB runner is the same -j2 hazard the Makefile already documents, neither improved nor worsened here. Do not raise -j on the strength of this change. Splitting reg_field and reg_field_rls the same way — structurally identical files, the port is a prefix rename of this diff — is what would take max(peak) low enough to make -j4 CPU-bound rather than RAM-bound.

Gate

src/lib-cuda/ is the entire diff — sixteen files, none outside that directory — and nothing on the CPU side compiles or includes any of them: src/lib-cpu and src/lib build their own sources against include/, which this branch does not touch. So the 13-suite / 59,886-check gate cannot move, and CI's path filter skips those legs on this PR for exactly that reason — they show as skipped, not green, which is worth not misreading.

Because CI skips them, I ran the gate here by hand: tools/test-baseline.sh --legs default,lib on this branch, diffed row for row against tools/test-baseline.expected — 15 rows, 0 differ; 13 suites / 59,886 checks / 0 failures on the default leg, plus the 2 hub suites (14 checks). (--check itself refuses a two-leg run, since it compares whole reports and the recording covers six legs, so the rows were diffed against the recording directly.)

Both build-cuda legs are green, including the --no-undefined hub link, tools/check-cuda-host-stubs.sh and the ldd -r load-time resolution check; compile-probe-cuda is green.

No GPU in CI, so nothing here is a claim about runtime behaviour — the CUDA side is compile + link only.

lint (clang-format, changed lines) is red, as it is on every PR; per #89 it cannot print its findings, so it is not chased here. #156 has since made it informational.

[balbasty]

ok but should use pragma once and the namespace macros

[balbasty]

CI note, so the state is legible to whoever picks this up.

The run on the final commit (3366779, run 32394048973) is stalled in Actions
infrastructure, not on anything in this branch: compile-probe-cuda, which
takes ~21 s, has been in_progress for over 30 minutes, and the previous run
(32392090091) had its build-cuda (index32) leg sit past the 120-minute timeout
while the same commit'sindex64 leg finished normally in 12 minutes. Worth
re-running rather than reading as a result.

Between them, the two earlier runs already cover both legs of the matrix:

run	commit	leg	result
32365351378	8f08c7d	build-cuda (pre-#143, single leg = index32)	success, 1672 s compile
32392090091	7b20ab9	build-cuda (index64)	success, 723 s compile

8f08c7d and 7b20ab9 carry an identical split; the difference between them is
the rebase onto b0463bc (off32_t, #pragma once, <fastfields/…>), which
changes no instantiation. So the split is known-good at index32, and the rebased
tree is known-good at index64 — the two together cover the matrix, and
3366779 on top of 7b20ab9 is a Makefile-comment-only commit.

Both FFMEM tables are transcribed into src/lib-cuda/Makefile and the PR body.

lint (clang-format, changed lines) is red and is continue-on-error; per #89
it cannot print its findings, so I have not chased it.

---

Generated by Claude Code

[balbasty]

Correcting my previous comment.

I said the run on 3366779 was "stalled in Actions infrastructure". That was
wrong, or at least unsupported, and I would rather fix it than leave it standing.

What actually happened is that the Actions API was serving me a cached view of
run 32394048973 — its updated_at sat at 16:49:03 across half an hour of
polling while the individual step timestamps did advance underneath. The two
steps I cited as evidence both completed normally once the cache caught up:

• compile-probe-cuda — 16:49:02 → 16:50:54, 112 s, success;
• Install CUDA toolkit and clang on the index64 leg — 16:49:04 → 16:51:03,
  119 s, success.

So there is no reason to think anything is wrong with this run; it was simply
still compiling, and I was reading a stale snapshot as if it were a live one.

The one observation that does still stand is separate and from the earlier
run: 32392090091's build-cuda (index32) leg had a real step start of
16:29:20 and had not completed by 18:27, past the job's own 120-minute
timeout, while the same commit's index64 leg finished in 12 minutes. That
remains anomalous against the 27.9 minutes the identical split took at index32
in run 32365351378, and is worth a re-run rather than a diagnosis from me.

Net effect on the PR: none of the measurements change — they come from
32365351378 (index32) and 32392090091 (index64), both of which completed and
both of whose FFMEM tables are transcribed into src/lib-cuda/Makefile. The
only thing I got wrong was the status report, not the data.

---

Generated by Claude Code