Uh oh!
There was an error while loading. Please reload this page.
tools: add a re-runnable test-baseline gate for the repo consolidation - #76
Merged
Conversation
The six C++/CUDA repos are about to be consolidated into one, which rewrites git history and relocates every file. The only mechanical proof that such a migration changed nothing is that the test suite produces identical results before and after -- so that evidence needs to be a re-runnable artifact rather than a number recorded once by hand. tools/test-baseline.sh builds fastfields-cpu-lib's suite once per configuration leg and emits a sorted, machine-comparable report: suite <TAB> config <TAB> checks <TAB> failures The legs mirror .github/workflows/test.yaml exactly -- the three-way BOUNDFLAGS/SPLINEFLAGS matrix (static, dynamic, cuda-default) plus the separate ASan+UBSan job -- and add two more: `default` (a bare `make test`, so a change to the Makefile's target-specific defaults shows up rather than silently weakening the gate) and `lib` (fastfields-lib's own two standalone argument-validation tests). tools/test-baseline.expected records the measured result for all six legs at the commits named in its header: 13 cpu-lib suites totalling 53988 checks in each of the five cpu-lib configs, and 14 checks across fastfields-lib's two. Zero failures everywhere, including under the sanitizers. `--check` compares a fresh run against that file and exits non-zero on any failure, any suite that did not run, or any check-count difference, so the migration gate is a single command with a single pass/fail condition. Three details in here are load-bearing and are documented at length in the script rather than left to be rediscovered: * Every leg goes through `make test`. The Makefile sets BOUNDFLAGS and SPLINEFLAGS with target-specific plain `=` assignments on `test:` (they cannot be `?=` -- the global `?=` defaults already count as set at parse time, so a target-specific `?=` would never fire). Building a test binary by its own path does not enter that context and would silently measure the fully-static policy instead of the requested leg. * -DFF_TEST_SPARSE is hard-coded into TESTCPPFLAGS, so it is not a configuration axis -- it is on for every leg here and in CI. * The Makefiles are clang-only by default (CXXFLAGS picks up -ferror-limit and -ftemplate-backtrace-limit), so `make CXX=g++` fails on the flags rather than the source. The script detects a non-clang compiler and replaces CXXFLAGS wholesale, which is what makes --cxx g++ work. Measurement only -- no source, test or Makefile is touched.
balbasty added a commit
that referenced
this pull request
Aug 18, 2026
…it (#77) Moves cpu dbd3274 -> 1fb2b37 and cuda 6ae52bd -> 2b2ad55, the current mains of fastfields-cpu-lib and fastfields-cuda-lib. This is the top of a bottom-up cascade that closes a live pin skew. Before it, the two backends resolved to different kernels commits: CPU path : lib -> cpu-lib -> cpu-impl -> kernels be7be08 CUDA path: lib -> cuda-lib -> cuda-impl -> kernels b09b284 (three behind) The CUDA side was missing kernels#74 (heap over-read from the past-the-end FaceIterator, a memory-safety fix), #76 (make the host BVH/normal builders visible to nvcc's host pass -- a CUDA-specific fix the CUDA path lacked) and #78 (drop the pointless virtual destructors). After this commit every path resolves to kernels 1df9fd3: lib -> cpu-lib 1fb2b37 -> cpu-impl 5e2c78e -> kernels 1df9fd3 lib -> cuda-lib 2b2ad55 -> cuda-impl 83fa026 -> kernels 1df9fd3 The cascade also carries the diag_bending/diag_all corner cross-term correctness fix (kernels#81) to both backends, its regression test (cpu-lib#89), the reg_flow Lame dispatch routing (cpu-lib#87) and the cuda-impl include guards (cuda-impl#48). Claude-Session: https://claude.ai/code/session_016AjQcY78NgbagPSbPJRr6Z Co-authored-by: Claude <noreply@anthropic.com>
The first recording in this branch was measured against cpu-lib dbd3274 / kernels be7be08. While it was being taken, three things landed upstream: kernels 1df9fd3 fix(regularisers): correct diag_bending/diag_all corner cross-term cpu-impl 5e2c78e deps: bump kernels pin to that fix cpu-lib 1fb2b37 test(reg): cover the diag_bending/diag_all corner cross-term bug (+ db3c3b5 perf(reg_flow) routing) The old recording was internally consistent -- every leg was measured against one frozen set of clones that was never re-fetched -- but it described a tree that main has since moved past, so it would have failed as a gate for the wrong reason. Re-recorded against the current pin chain, which is now fully aligned (each recorded pin equals the pinned repo's own main tip). The only rows that move are reg_field and reg_flow, in all five cpu-lib legs, which is what the new corner cross-term coverage should do and nothing else: reg_field 18284 -> 19250 (+966) reg_flow 11415 -> 16347 (+4932) per leg 53988 -> 59886 (+5898) The other eleven suites are unchanged to the check, and all five legs still agree with each other exactly. Still zero failures everywhere, including under ASan+UBSan. The header now also records that the pin chain is aligned, and why that is worth stating: when a pin lags its repo's main (as the kernels pin did for the first recording), "clone main and follow the pins" and "check out main everywhere" are different trees, and a baseline has to say which one it means.
--check compares whole reports, so running a subset of the recorded legs diffed as 'every recorded row vanished' -- which reads like catastrophic breakage rather than the operator error it is. Compare the config sets first and name both sides.
Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The six C++/CUDA repos are about to be consolidated into one, which rewrites git history and relocates every file. The only thing that will prove the migration was mechanically correct is that the test suite produces identical results before and after — so that evidence has to be a re-runnable artifact, not a number written down once.
A previous investigation reported "11 suites, 27,925 checks". That was measured on a stale dev branch (11 test files) under the default sparse configuration only, and does not describe
origin/main. The real figures are 13 suites and 59,886 checks per configuration leg, across five cpu-lib legs plus this repo's own two tests.What this adds
tools/test-baseline.sh— buildsfastfields-cpu-lib's suite once per configuration leg and emits a sorted, machine-comparable report:Two runs compare with plain
diff.--checkcompares against the recorded file and exits non-zero on any failure, any suite that did not run, or any check-count difference.tools/test-baseline.expected— the recorded result for all six legs, with the exact commits it was measured at in its header.The measured baseline
Measured at
cpu-lib 1fb2b37→impl 5e2c78e→kernels 1df9fd3(the pin chain is currently fully aligned with each repo's main tip). All five cpu-lib legs produce identical counts — 13 suites, 59,886 checks, 0 failures:fastfields-lib's two tests add 14 checks (device_check5,splinc_bound9).The sanitizer leg (ASan + UBSan,
-fno-sanitize-recover=all) is clean — same 59,886 checks, zero failures, no runtime errors.The legs
The three
BOUNDFLAGS/SPLINEFLAGSlegs and the sanitizer job mirror.github/workflows/test.yamlexactly. Two more are added:default— a baremake test. Identical tocuda-defaultby construction; recorded separately so a change to the Makefile's target-specific defaults shows up here rather than silently weakening the other legs.lib— this repo's two standalone argument-validation tests, gated by nothing else.FF_TEST_SPARSEis not a configuration axis: it is hard-coded intoTESTCPPFLAGS, so it is on for every leg here and in CI.Load-bearing details, documented in the script
make test, nevermake build/test_<x>. The Makefile setsBOUNDFLAGS/SPLINEFLAGSwith target-specific plain=assignments ontest:— they cannot be?=, because the global?=defaults already count as "set" at parse time, so a target-specific?=would never fire. Those values propagate to the prerequisitesmake testbuilds, but building a test binary by its own path does not enter that context and would silently compile the fully-static policy instead of the requested leg.CXXFLAGSpicks up-ferror-limitand-ftemplate-backtrace-limit, somake CXX=g++fails on the flags before reaching any source. The script detects a non-clang compiler and replacesCXXFLAGSwholesale, which is what makes--cxx g++work.checks: N, failures: M;P/N checks passed;ok:/FAIL:lines). A fourth is reported asUNPARSEDand fails the run rather than being skipped.Determinism
The gate is only sound if check counts are reproducible. Verified:
#ifdefat all, so counts are fixed by loop structure rather than compile policy — which is why all five legs agree exactly.std::mt19937seeded from literal loop counters (for seed = 1..5), never time or entropy.FF_NUM_THREADSto 1, 2, 8 and 16 gives byte-identical output, includingsolve_field's CG iteration counts; repeat runs are identical too.straceshows zeroclonesyscalls: every test workload is belowGRAIN_SIZE(32768), soparallel_fornever takes its parallel branch and the thread pool is never instantiated. Machine core count is therefore irrelevant, andpushpull's atomic-scatter path never runs concurrently during tests.static,dynamic,cuda-default,default,lib) — 54 rows, identical md5. The sanitizer leg is clang-only, as in CI.Usage
Exit 0 means the tree behaves identically to the recording. All paths are resolved up front, so it runs from any working directory.
Scope
Measurement plus this new tooling only — no source, test, or Makefile is modified.
🤖 Generated with Claude Code
https://claude.ai/code/session_016AjQcY78NgbagPSbPJRr6Z