tools: add consolidate.sh, the frozen rule set for the six-repo merge

[balbasty]

Phase 1 of the six-repo consolidation. This PR adds only the re-runnable rule set — no consolidated history is merged anywhere by it.

tools/consolidate.sh takes six pristine clones and produces the finished consolidated tree in about 8 seconds, so if a container is reset the work costs minutes rather than a day. (That contingency was not hypothetical — see "Phase 2 results" below.)

The frozen rule

The file carries the warning in capitals at the top. git filter-repo runs exactly once per source repo, over main and teeny together in a single pass (--refs main teeny).

Identical rules applied to both refs rewrite their shared ancestors to identical SHAs, so the merge-base survives and teeny stays mergeable. Changing a single path rule between runs produces zero shared commits and an empty merge-base — teeny permanently un-mergeable. Rule drift is the fatal failure mode, not branch structure.

Stage 2 therefore asserts, per repo, that a main/teeny merge-base still exists after the rewrite, and refuses to continue if one is gone:

ok fastfields-lib pre=b59a436a post=f7f8c262
ok fastfields-cpu-lib pre=67149398 post=52cf1f1c
ok fastfields-cuda-lib pre=2c894eee post=c882b586
ok fastfields-cpu-impl pre=fbe7e096 post=c8ab28b8
ok fastfields-cuda-impl pre=4ec198c1 post=03e8b558
ok fastfields-kernels pre=dc9f7f18 post=676b9925

Structure

stage	what
1	pristine clones
2	path moves only, one filter-repo pass per repo over both refs
3	merge the six rewritten mains (--allow-unrelated-histories)
4	core/ dedupe
5	include rewrite
6	build system

Stage 2 is deliberately path-moves-only. Content edits live in stages 4–6 as ordinary commits on main, because they could not be replayed identically on teeny, whose content differs. Phase 2 does the teeny side.

The six repos' output path sets are disjoint by construction, so stage 3's five merges cannot conflict — that is what lets the duplicate dlpack.h/autocast.h/defines.h copies arrive intact and be deduplicated in a visible, reviewable commit in stage 4 rather than resolved as a merge conflict.

Notes on stages 4–6

• core/dlpack.h — three byte-identical copies collapse to one; the upstream DLPACK_DLPACK_H_ guard is kept verbatim and no #pragma once is added, so a consumer that also has a system dlpack can include both harmlessly. The script asserts both.
• core/defines.h — the two colliding defines.h merge into one file with one guard. This is what makes the cuda_switch.h hazard impossible by construction: it ended in a bare #include "defines.h" that resolved to whichever copy was nearest, and with both on the -I path that choice becomes silent and wrong (FF_DEVICE undefined, FF_CPU/FF_CUDA leaking into kernel namespaces — a miscompile, not an error). The include is also made fully qualified.
• core/autocast.h — not a duplicate pair. The CUDA copy stages through pinned host memory (cudaMallocHost/cudaFreeHost) for async H2D copies; the CPU copy uses new[]/delete[]. That is the only functional difference between them, so this is a refactor to one header with the host allocator injected behind FF_AUTOCAST_PINNED_HOST — not two files under tidier names.
• Include rewrite is longest-prefix-first, and the ordering is load-bearing twice: the cuda_switch.h rules must fire before the "impl/kernels/ and "kernels/ prefix rules, and "impl/kernels/ before "impl/. "impl/ resolves to impl/cpu/ or impl/cuda/ depending on which library is including it — the one thing the old symlink layout encoded implicitly. Includes that kept same-directory adjacency are deliberately left alone. The script also checks for include-guard collisions once the tree is flattened.
• Build system — make/common.mk plus four makefiles. DIAGFLAGS is compiler-detected (clang -ferror-limit=1, gcc -fmax-errors=1, both -ftemplate-backtrace-limit=0), so make CXX=g++ works natively for the first time. src/lib/ and src/lib-cuda/ gain -MMD -MP. The Windows block is stated once instead of three times. build/ and build/lib/ output paths, all as default target, the BOUNDFLAGS/SPLINEFLAGS/FF_TEST_SPARSE semantics (including the target-specific plain-= assignments and the comment explaining why they cannot be ?=), and the split CUDA MODULES list are all preserved.

---

Fix in this PR: don't abort when a clone has only main and teeny

The ref-prune before git filter-repo deletes every ref that is not main or teeny. When the source clone carries nothing but those two — the normal case for a clone staged for this script — grep -v matches no lines and exits 1. Under set -euo pipefail that aborts the entire run at the first repository, before anything is rewritten, and with no diagnostic because filter-repo's stdout goes to /dev/null:

=== stage 2: path rewrite (one filter-repo pass per repo, --refs main teeny)
fastfields-lib
$ echo $?
1

PIPESTATUS is 0 1 0 — git succeeded, grep found nothing to delete, the delete loop ran cleanly. Deleting no refs is the correct outcome, so the grep now swallows the no-match status.

No path rule was touched. No filename callback, no --refs argument and no rule ordering changed, and the prune still deletes exactly the refs it did before. Determinism was re-verified after the fix, since that is the property the whole migration rests on: two runs produce identical SHAs for all 12 rewritten refs and an identical consolidated tree hash (828c9164…).

Phase 2 results (consolidated teeny)

Built with this script, unmodified apart from the fix above.

• merge-base(main, teeny) == consolidated main tip exactly (d4f90e68…), 159 ahead / 0 behind — teeny is a strict descendant, one merge base, no criss-cross.
• teeny's suite: 12 suites, 33,534 checks, 0 failures, byte-identical before and after consolidation. It was measured at teeny's own HEAD first, so the comparison is against a branch known to be healthy.
• main untouched: tree hash unchanged and the gate re-run gives BASELINE MATCH: 67 rows.
• external/teeny (submodule, pinned 5c46bd4) exists on teeny only; main has no external/ and no .gitmodules.

Both consolidated histories are preserved on claude/consolidated-main and claude/consolidated-teeny — backup branches, not a publication step, and trivially deletable.

Three hazards Phase 2 surfaced

Recorded in the script's header so they travel with it. Anyone repeating this operation should expect all three.

1. The clean merges are more dangerous than the conflicts. Each per-repo merge is three-way with the fork point as base, so git silently auto-merges main's divergence into teeny's files wherever the two touched different regions — producing hybrids neither branch ever compiled, with no conflict and no warning. Concretely, kernels' spline.h came out with main's runtime SplineVec work spliced into teeny's copy, which teeny's pushpull/utils.h cannot compile against. Re-checking-out every path the owning branch owns, then re-running the stage 4 dedupe and stage 5 include rewrite, is mandatory, not a tidy-up.
2. git checkout --theirs silently keeps our copy on modify/delete.teeny deletes pushpull/{2d,3d,nd}.h and regularisers/{field,flow}/{1,2,3}d.h while main modifies them; with no "theirs" blob, --theirs fails and leaves main's content in the tree, which a following git add stages — resurrecting nine deleted files. Concrete evidence for the caveat that resolving with --theirs answers the ancestry question and says nothing about correctness.
3. Include guards that were unique per-repo can collide here. Six repos with six include paths tolerate duplicate guards; one tree behind one -I does not — the second header silently expands to nothing. teeny's api/cpu/reg_dispatch.h and api/cuda/reg_dispatch.h both used FF_REG_DISPATCH_H. Stage 5's guard check is what catches this, and it is a hard failure on purpose.

🤖 Generated with Claude Code

https://claude.ai/code/session_016AjQcY78NgbagPSbPJRr6Z

---

Generated by Claude Code

[github-actions]

Submodule staleness (non-blocking -- fastfields-lib#15)

submodule	pinned commit	tracks	status	behind by
cpu	1fb2b37	fastfields/fastfields-cpu-lib@main	up to date	0
cuda	2b2ad55	fastfields/fastfields-cuda-lib@main	up to date	0

[balbasty]

Closing as superseded — the base branch this PR targets no longer exists.

main has been rewritten to the consolidated six-repo tree (d4f90e6). This PR's base was the pre-consolidation main (85f987e), whose history is disjoint from the new one, so this PR is no longer mergeable in principle — not merely conflicted.

The script itself is not lost. tools/consolidate.sh is restored onto the new main in #79, byte-identical to this branch's head c55e791 (blob 4255943, mode 100755), hazards header included.

It was deliberately not merged here first. This branch's script is what built the consolidated tree; merging it and rebuilding would have changed claude/consolidated-main's SHA, and the 159 irreproducible teeny commits sit directly on top of that SHA. Restoring the file as an ordinary commit on the rewritten main avoids rebasing work that cannot be regenerated.

The pre-rewrite state remains reachable at the branch pre-consolidation-main (85f987e) if this PR's original context is ever needed.

---

Generated by Claude Code