From bc4ea462d6f6e466148108413134c8eaf1157083 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 20 Aug 2026 21:44:38 +0000 Subject: [PATCH 1/2] refactor(lib-cpu): state the regulariser dispatch pyramid once The regulariser API surface -- `reg_field` (multi-channel fields) and `reg_flow` (vector flows) -- carried its ndim x bound x dtype x index-width dispatch pyramid as per-file macros. Measured across the six regulariser translation units on main: * BND1 / BND2 / BND3 and BOUND_SWITCH are byte-identical in all six. * NDIM_SWITCH differs in exactly one word: the noun in its diagnostic, "field" versus "flow". * The dtype x index-width fan is copied once per ENTRY POINT rather than once per module -- thirteen times in reg_field.cpp alone, fifty-two across the six, out of seventy-one in the whole tree. Each copy differs only in the leaf's name and, for nine of them, one extra `char` template argument. This moves that pyramid into include/fastfields/api/cpu/reg_dispatch.h, the same shape and location as the existing api/{cpu,cuda}/pushpull_dispatch.h, and converts the two CPU regulariser translation units to it. The fan is now written twice (once per leaf shape: with and without the compile-time op char) instead of fifty-two times; the boundary and rank switches once each. Behaviour is unchanged by construction. The FF_BOUND_ selectors, the switch labels, the boundary-pack lengths and every diagnostic string are the same tokens they were; only their location moved. Instantiation-neutral, which is the constraint that matters here -- the leaf set is what ptxas has to process on the CUDA side. clang++ -O3 -fPIC with the CUDA shipping bound policy, counting defined text/weak symbols: reg_field main 10,140 this 10,140 (identical; object 19,533,368 -> 19,533,360 bytes) Argument lists use ISO `...` / `__VA_ARGS__` rather than the GNU named variadic `args...` the other dispatch macros still use. That is a portability down payment and not a fix: MSVC's traditional preprocessor also mis-forwards __VA_ARGS__ into a nested macro, so an MSVC build would need /Zc:preprocessor regardless. Nothing here builds under MSVC and that is untested. Scope is deliberately the CPU side only. The four CUDA regulariser TUs want the mirror header, but src/lib-cuda/reg_flow.cpp is being rewritten by the per-(family, ndim) TU split, so that half should follow rather than collide with it. C++11. Clean under clang++ and g++ at -std=c++11. --- include/fastfields/api/cpu/reg_dispatch.h | 176 +++++++++++ src/lib-cpu/reg_field.cpp | 350 ++++------------------ src/lib-cpu/reg_flow.cpp | 347 ++++----------------- 3 files changed, 288 insertions(+), 585 deletions(-) create mode 100644 include/fastfields/api/cpu/reg_dispatch.h diff --git a/include/fastfields/api/cpu/reg_dispatch.h b/include/fastfields/api/cpu/reg_dispatch.h new file mode 100644 index 0000000..c920560 --- /dev/null +++ b/include/fastfields/api/cpu/reg_dispatch.h @@ -0,0 +1,176 @@ +#pragma once +/** + * Private (not installed) header: the ndim x bound x dtype x index-width + * dispatch pyramid shared by the regulariser API surface -- `reg_field.cpp` + * (multi-channel fields) and `reg_flow.cpp` (vector flows). + * + * Why one header for two modules + * -------------------------------------------------------------------------- + * These two are one *surface*, not two: their pyramids were byte-identical. + * Before this header, `BND1`/`BND2`/`BND3` and `BOUND_SWITCH` were literally + * the same text in all six regulariser translation units (the two here, plus + * `src/lib-cuda/reg_{field,flow}{,_rls}.cpp`), and `NDIM_SWITCH` differed in + * exactly one word -- the noun in its diagnostic, "field" versus "flow". That + * word is now the `NOUN` parameter, and it is the only thing the two modules + * did not already share. + * + * The dtype x index-width fan is the part that had really multiplied: it was + * copied once per *entry point*, not once per module -- thirteen times in + * `reg_field.cpp` alone, fifty-two across the six regulariser TUs, out of + * seventy-one in the whole tree. Every copy differed only in the leaf's name + * and, for nine of them, one extra `char` template argument. Both variations + * are parameters here, so the fan is written twice (once per leaf shape) + * instead of fifty-two times. + * + * This follows `api/{cpu,cuda}/pushpull_dispatch.h`, which did the same for + * the pushpull surface, and it is deliberately NOT a dispatcher generic over + * all modules: the seven API surfaces have eight different leaf + * template-argument orders and do not even agree on the order the axes are + * resolved in (`splinc` dispatches dtype *outermost*; every surface here + * dispatches it innermost). See fastfields-lib#94 for the measurement. + * + * Not in `core/` + * -------------------------------------------------------------------------- + * Per the definition fastfields-lib#149 settled on, `core/` holds what more + * than one layer needs and what is not the computation of a named fastfields + * operation. This is tied to one named operation family's template signatures, + * so it belongs in `api/`, beside the entry points whose leaves it names. + * + * Variadic macro syntax + * -------------------------------------------------------------------------- + * The argument lists use ISO `...` / `__VA_ARGS__`, not the GNU named-variadic + * `args...` extension the rest of the dispatch macros still use. That is a + * portability down payment, not a fix: MSVC's *traditional* preprocessor also + * mis-forwards `__VA_ARGS__` to a nested macro (it arrives as one argument), + * so an MSVC build needs `/Zc:preprocessor` regardless. Nothing in this tree + * builds under MSVC today and that claim is untested here. + */ +#include +#include +#include +#include +#include +#include + +FF_NAMESPACE_BEGIN(FF_NS) +FF_NAMESPACE_BEGIN(FF_DEVICE) + +/*********************************************************************** + * THE BOUNDARY PACK * + ***********************************************************************/ + +// The regulariser leaves take the boundary condition as a pack of `ndim` +// copies, and the length is load-bearing: `bound::getutils` / `` / +// `` detect isotropy from it. These three expand one condition to the +// pack the leaf expects. They are macro *arguments* at every call site, so +// the commas they introduce are protected until they land in the template +// argument list itself. +#define FF_REG_BND1(B) B +#define FF_REG_BND2(B) B, B +#define FF_REG_BND3(B) B, B, B + +/*********************************************************************** + * LEVEL 1 -- dtype x index width * + ***********************************************************************/ + +// Two leaf shapes exist in this surface and there is no way to write them as +// one: nine entry points thread a compile-time `char` op ('=', '+', '-') as +// the leaf's second template argument, and seven (the `relax` and `_rls` +// families) have no op concept at all. Both macros take `OP` so that the +// levels above can forward one argument list to either; FF_REG_DT ignores it. +// +// `off32_t` is `int32_t` or, under `FF_INDEX32=0`, `int64_t` -- which is how +// the index axis collapses onto one instantiation. See core/dispatch.h. + +// Leaf shape A: FN +#define FF_REG_DT(FN, NDIM, OP, BNDS, ...) \ + switch (code) { \ + case kDLFloat: switch (bits) { \ + case 32: return use_32bits \ + ? FN(__VA_ARGS__) \ + : FN(__VA_ARGS__); \ + case 64: return use_32bits \ + ? FN(__VA_ARGS__) \ + : FN(__VA_ARGS__); \ + default: break; \ + } break; \ + default: break; \ + } \ + throw std::invalid_argument("only floating point data types are supported"); + +// Leaf shape B: FN +#define FF_REG_DT_OP(FN, NDIM, OP, BNDS, ...) \ + switch (code) { \ + case kDLFloat: switch (bits) { \ + case 32: return use_32bits \ + ? FN(__VA_ARGS__) \ + : FN(__VA_ARGS__); \ + case 64: return use_32bits \ + ? FN(__VA_ARGS__) \ + : FN(__VA_ARGS__); \ + default: break; \ + } break; \ + default: break; \ + } \ + throw std::invalid_argument("only floating point data types are supported"); + +/*********************************************************************** + * LEVEL 2 -- boundary condition * + ***********************************************************************/ + +// Which of these conditions gets a dedicated (static) instantiation and which +// shares the single Dynamic (runtime) one is a build-time choice -- see +// FF_STATIC_BOUND_* in impl/kernels/bounds.h. The switch labels stay +// exhaustive on the *runtime* value either way; only the instantiated +// template argument collapses onto the shared Dynamic path. `bvec`, which the +// call site passes through, carries the runtime condition for the ones that +// do. +#define FF_REG_BOUND(DT, FN, NDIM, OP, BND, ...) \ + switch (bnd) { \ + case bound::type::Zero: DT(FN,NDIM,OP,BND(FF_BOUND_ZERO), __VA_ARGS__); break; \ + case bound::type::Replicate: DT(FN,NDIM,OP,BND(FF_BOUND_REPLICATE),__VA_ARGS__); break; \ + case bound::type::DCT1: DT(FN,NDIM,OP,BND(FF_BOUND_DCT1), __VA_ARGS__); break; \ + case bound::type::DCT2: DT(FN,NDIM,OP,BND(FF_BOUND_DCT2), __VA_ARGS__); break; \ + case bound::type::DST1: DT(FN,NDIM,OP,BND(FF_BOUND_DST1), __VA_ARGS__); break; \ + case bound::type::DST2: DT(FN,NDIM,OP,BND(FF_BOUND_DST2), __VA_ARGS__); break; \ + case bound::type::DFT: DT(FN,NDIM,OP,BND(FF_BOUND_DFT), __VA_ARGS__); break; \ + case bound::type::NoCheck: DT(FN,NDIM,OP,BND(FF_BOUND_NOCHECK), __VA_ARGS__); break; \ + default: throw std::invalid_argument("Unsupported boundary condition"); \ + } + +/*********************************************************************** + * LEVEL 3 -- ndim * + ***********************************************************************/ + +// NOUN is the word the rank diagnostic uses ("field" / "flow"). It is the +// only text that was ever module-specific in this pyramid. +#define FF_REG_NDIM(DT, FN, OP, NOUN, ...) \ + switch (ndim) { \ + case 1: FF_REG_BOUND(DT,FN,1,OP,FF_REG_BND1,__VA_ARGS__); break; \ + case 2: FF_REG_BOUND(DT,FN,2,OP,FF_REG_BND2,__VA_ARGS__); break; \ + case 3: FF_REG_BOUND(DT,FN,3,OP,FF_REG_BND3,__VA_ARGS__); break; \ + default: throw std::invalid_argument("Only 1D, 2D and 3D " NOUN " are supported"); \ + } + +/*********************************************************************** + * THE TWO CALL FORMS * + ***********************************************************************/ + +/** + * Dispatch `FN` over ndim x bound x dtype x index width. + * + * The call site must already have `ndim`, `bnd`, `code`, `bits` and + * `use_32bits` in scope -- the same five locals every regulariser entry point + * computed before this header existed. + * + * FF_DISPATCH_REG (_field_relax, "field", args...) FN + * FF_DISPATCH_REG_OP(_field_matvec_acc, '+', "field", args...) FN + */ +#define FF_DISPATCH_REG(FN, NOUN, ...) \ + FF_REG_NDIM(FF_REG_DT, FN, '=', NOUN, __VA_ARGS__) + +#define FF_DISPATCH_REG_OP(FN, OP, NOUN, ...) \ + FF_REG_NDIM(FF_REG_DT_OP, FN, OP, NOUN, __VA_ARGS__) + +FF_NAMESPACE_END(FF_DEVICE) +FF_NAMESPACE_END(FF_NS) diff --git a/src/lib-cpu/reg_field.cpp b/src/lib-cpu/reg_field.cpp index 457041a..1b0d4e3 100644 --- a/src/lib-cpu/reg_field.cpp +++ b/src/lib-cpu/reg_field.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -479,230 +480,6 @@ inline void _field_relax_rls( * DISPATCH * ***********************************************************************/ -#define BND1(B) B -#define BND2(B) B, B -#define BND3(B) B, B, B - -#define MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_matvec(MV_ARGS) \ - : _field_matvec(MV_ARGS); \ - case 64: return use_32bits \ - ? _field_matvec(MV_ARGS) \ - : _field_matvec(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_matvec_acc(MV_ARGS) \ - : _field_matvec_acc(MV_ARGS); \ - case 64: return use_32bits \ - ? _field_matvec_acc(MV_ARGS) \ - : _field_matvec_acc(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_matvec_acc(MV_ARGS) \ - : _field_matvec_acc(MV_ARGS); \ - case 64: return use_32bits \ - ? _field_matvec_acc(MV_ARGS) \ - : _field_matvec_acc(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _field_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_diag(DG_ARGS) \ - : _field_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _field_diag(DG_ARGS) \ - : _field_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_diag(DG_ARGS) \ - : _field_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _field_diag(DG_ARGS) \ - : _field_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _field_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_kernel(KN_ARGS) \ - : _field_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _field_kernel(KN_ARGS) \ - : _field_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_kernel(KN_ARGS) \ - : _field_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _field_kernel(KN_ARGS) \ - : _field_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RX_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_relax(RX_ARGS) \ - : _field_relax(RX_ARGS); \ - case 64: return use_32bits \ - ? _field_relax(RX_ARGS) \ - : _field_relax(RX_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_matvec_rls(RLS_MV_ARGS) \ - : _field_matvec_rls(RLS_MV_ARGS); \ - case 64: return use_32bits \ - ? _field_matvec_rls(RLS_MV_ARGS) \ - : _field_matvec_rls(RLS_MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_diag_rls(RLS_DG_ARGS) \ - : _field_diag_rls(RLS_DG_ARGS); \ - case 64: return use_32bits \ - ? _field_diag_rls(RLS_DG_ARGS) \ - : _field_diag_rls(RLS_DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_RX_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _field_relax_rls(RLS_RX_ARGS) \ - : _field_relax_rls(RLS_RX_ARGS); \ - case 64: return use_32bits \ - ? _field_relax_rls(RLS_RX_ARGS) \ - : _field_relax_rls(RLS_RX_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -// Which of these boundary conditions gets a dedicated (static) instantiation -// and which shares the single Dynamic (runtime) one is a build-time choice -- -// see FF_STATIC_BOUND_* in kernels/bounds.h. `bvec` carries the runtime -// condition for whichever ones fall back to Dynamic. -#define BOUND_SWITCH(DT, NDIM, BND) \ - switch (bnd) { \ - case bound::type::Zero: DT(NDIM, BND(FF_BOUND_ZERO)); break; \ - case bound::type::Replicate: DT(NDIM, BND(FF_BOUND_REPLICATE)); break; \ - case bound::type::DCT1: DT(NDIM, BND(FF_BOUND_DCT1)); break; \ - case bound::type::DCT2: DT(NDIM, BND(FF_BOUND_DCT2)); break; \ - case bound::type::DST1: DT(NDIM, BND(FF_BOUND_DST1)); break; \ - case bound::type::DST2: DT(NDIM, BND(FF_BOUND_DST2)); break; \ - case bound::type::DFT: DT(NDIM, BND(FF_BOUND_DFT)); break; \ - case bound::type::NoCheck: DT(NDIM, BND(FF_BOUND_NOCHECK)); break; \ - default: throw std::invalid_argument("Unsupported boundary condition"); \ - } - -#define NDIM_SWITCH(DT) \ - switch (ndim) { \ - case 1: BOUND_SWITCH(DT, 1, BND1); break; \ - case 2: BOUND_SWITCH(DT, 2, BND2); break; \ - case 3: BOUND_SWITCH(DT, 3, BND3); break; \ - default: throw std::invalid_argument("Only 1D, 2D and 3D field are supported"); \ - } - void field_matvec( DLTensor & out_ , const DLTensor & inp_ , @@ -735,11 +512,10 @@ void field_matvec( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG(_field_matvec, "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, + out.shape, out.strides, inp.strides) } /** @@ -777,11 +553,10 @@ void field_addmatvec_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(ADD_MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG_OP(_field_matvec_acc, '+', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, + out.shape, out.strides, inp.strides) } /** @@ -819,11 +594,10 @@ void field_submatvec_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(SUB_MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG_OP(_field_matvec_acc, '-', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, + out.shape, out.strides, inp.strides) } void field_diag( @@ -853,11 +627,10 @@ void field_diag( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_field_diag, '=', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } /** @@ -894,11 +667,10 @@ void field_adddiag_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(ADD_DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_field_diag, '+', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } /** @@ -931,11 +703,10 @@ void field_subdiag_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(SUB_DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_field_diag, '-', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } void field_kernel( @@ -965,11 +736,10 @@ void field_kernel( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_field_kernel, '=', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } /** @@ -1003,11 +773,10 @@ void field_addkernel_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(ADD_KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_field_kernel, '+', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } /** @@ -1041,11 +810,10 @@ void field_subkernel_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides - NDIM_SWITCH(SUB_KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_field_kernel, '-', "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, + out.shape, out.strides) } void field_relax( @@ -1080,11 +848,10 @@ void field_relax( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RX_ARGS bvec, static_cast(nbatch), nc, FF_VOIDPTR(sol), FF_CVOIDPTR(hes), \ - FF_CVOIDPTR(grd), voxel_size, absolute, membrane, bending, \ - nb_iter, sol.shape, sol.strides, hes.strides, grd.strides - NDIM_SWITCH(RX_DT) -#undef RX_ARGS + FF_DISPATCH_REG(_field_relax, "field", + bvec, static_cast(nbatch), nc, FF_VOIDPTR(sol), FF_CVOIDPTR(hes), + FF_CVOIDPTR(grd), voxel_size, absolute, membrane, bending, + nb_iter, sol.shape, sol.strides, hes.strides, grd.strides) } void field_forward( @@ -1233,12 +1000,11 @@ void field_matvec_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_MV_ARGS bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(out), \ - FF_CVOIDPTR(inp), FF_CVOIDPTR(wgt), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides, inp.strides, wgt.strides - NDIM_SWITCH(RLS_MV_DT) -#undef RLS_MV_ARGS + FF_DISPATCH_REG(_field_matvec_rls, "field", + bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(out), + FF_CVOIDPTR(inp), FF_CVOIDPTR(wgt), + voxel_size, absolute, membrane, bending, + out.shape, out.strides, inp.strides, wgt.strides) } void field_diag_rls( @@ -1274,12 +1040,11 @@ void field_diag_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_DG_ARGS bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(out), \ - FF_CVOIDPTR(wgt), \ - voxel_size, absolute, membrane, bending, \ - out.shape, out.strides, wgt.strides - NDIM_SWITCH(RLS_DG_DT) -#undef RLS_DG_ARGS + FF_DISPATCH_REG(_field_diag_rls, "field", + bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(out), + FF_CVOIDPTR(wgt), + voxel_size, absolute, membrane, bending, + out.shape, out.strides, wgt.strides) } void field_relax_rls( @@ -1323,13 +1088,12 @@ void field_relax_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_RX_ARGS bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(sol), \ - FF_CVOIDPTR(hes), FF_CVOIDPTR(grd), FF_CVOIDPTR(wgt), \ - voxel_size, absolute, membrane, bending, \ - nb_iter, sol.shape, sol.strides, hes.strides, grd.strides, \ - wgt.strides - NDIM_SWITCH(RLS_RX_DT) -#undef RLS_RX_ARGS + FF_DISPATCH_REG(_field_relax_rls, "field", + bvec, static_cast(nbatch), nc, is_jrls, FF_VOIDPTR(sol), + FF_CVOIDPTR(hes), FF_CVOIDPTR(grd), FF_CVOIDPTR(wgt), + voxel_size, absolute, membrane, bending, + nb_iter, sol.shape, sol.strides, hes.strides, grd.strides, + wgt.strides) } FF_NAMESPACE_END(FF_DEVICE) diff --git a/src/lib-cpu/reg_flow.cpp b/src/lib-cpu/reg_flow.cpp index 7cb2163..67b1211 100644 --- a/src/lib-cpu/reg_flow.cpp +++ b/src/lib-cpu/reg_flow.cpp @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -460,231 +461,6 @@ inline void _flow_relax_rls( * DISPATCH * ***********************************************************************/ -#define BND1(B) B -#define BND2(B) B, B -#define BND3(B) B, B, B - -// matvec dtype x offset dispatch, given ndim and the (repeated) bound pack. -#define MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_matvec(MV_ARGS) \ - : _flow_matvec(MV_ARGS); \ - case 64: return use_32bits \ - ? _flow_matvec(MV_ARGS) \ - : _flow_matvec(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_matvec_acc(MV_ARGS) \ - : _flow_matvec_acc(MV_ARGS); \ - case 64: return use_32bits \ - ? _flow_matvec_acc(MV_ARGS) \ - : _flow_matvec_acc(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_matvec_acc(MV_ARGS) \ - : _flow_matvec_acc(MV_ARGS); \ - case 64: return use_32bits \ - ? _flow_matvec_acc(MV_ARGS) \ - : _flow_matvec_acc(MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _flow_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_diag(DG_ARGS) \ - : _flow_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _flow_diag(DG_ARGS) \ - : _flow_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_diag(DG_ARGS) \ - : _flow_diag(DG_ARGS); \ - case 64: return use_32bits \ - ? _flow_diag(DG_ARGS) \ - : _flow_diag(DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _flow_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define ADD_KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_kernel(KN_ARGS) \ - : _flow_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _flow_kernel(KN_ARGS) \ - : _flow_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define SUB_KN_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_kernel(KN_ARGS) \ - : _flow_kernel(KN_ARGS); \ - case 64: return use_32bits \ - ? _flow_kernel(KN_ARGS) \ - : _flow_kernel(KN_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RX_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_relax(RX_ARGS) \ - : _flow_relax(RX_ARGS); \ - case 64: return use_32bits \ - ? _flow_relax(RX_ARGS) \ - : _flow_relax(RX_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_MV_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_matvec_rls(RLS_MV_ARGS) \ - : _flow_matvec_rls(RLS_MV_ARGS); \ - case 64: return use_32bits \ - ? _flow_matvec_rls(RLS_MV_ARGS) \ - : _flow_matvec_rls(RLS_MV_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_DG_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_diag_rls(RLS_DG_ARGS) \ - : _flow_diag_rls(RLS_DG_ARGS); \ - case 64: return use_32bits \ - ? _flow_diag_rls(RLS_DG_ARGS) \ - : _flow_diag_rls(RLS_DG_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -#define RLS_RX_DT(NDIM, BNDS...) \ - switch (code) { \ - case kDLFloat: switch (bits) { \ - case 32: return use_32bits \ - ? _flow_relax_rls(RLS_RX_ARGS) \ - : _flow_relax_rls(RLS_RX_ARGS); \ - case 64: return use_32bits \ - ? _flow_relax_rls(RLS_RX_ARGS) \ - : _flow_relax_rls(RLS_RX_ARGS); \ - default: break; \ - } break; \ - default: break; \ - } \ - throw std::invalid_argument("only floating point data types are supported"); - -// Which of these boundary conditions gets a dedicated (static) instantiation -// and which shares the single Dynamic (runtime) one is a build-time choice -- -// see FF_STATIC_BOUND_* in kernels/bounds.h. `bvec` carries the runtime -// condition for whichever ones fall back to Dynamic. -#define BOUND_SWITCH(DT, NDIM, BND) \ - switch (bnd) { \ - case bound::type::Zero: DT(NDIM, BND(FF_BOUND_ZERO)); break; \ - case bound::type::Replicate: DT(NDIM, BND(FF_BOUND_REPLICATE)); break; \ - case bound::type::DCT1: DT(NDIM, BND(FF_BOUND_DCT1)); break; \ - case bound::type::DCT2: DT(NDIM, BND(FF_BOUND_DCT2)); break; \ - case bound::type::DST1: DT(NDIM, BND(FF_BOUND_DST1)); break; \ - case bound::type::DST2: DT(NDIM, BND(FF_BOUND_DST2)); break; \ - case bound::type::DFT: DT(NDIM, BND(FF_BOUND_DFT)); break; \ - case bound::type::NoCheck: DT(NDIM, BND(FF_BOUND_NOCHECK)); break; \ - default: throw std::invalid_argument("Unsupported boundary condition"); \ - } - -#define NDIM_SWITCH(DT) \ - switch (ndim) { \ - case 1: BOUND_SWITCH(DT, 1, BND1); break; \ - case 2: BOUND_SWITCH(DT, 2, BND2); break; \ - case 3: BOUND_SWITCH(DT, 3, BND3); break; \ - default: throw std::invalid_argument("Only 1D, 2D and 3D flow are supported"); \ - } - void flow_matvec( DLTensor & out_ , const DLTensor & inp_ , @@ -719,11 +495,10 @@ void flow_matvec( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG(_flow_matvec, "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, inp.strides) } /** @@ -764,11 +539,10 @@ void flow_addmatvec_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(ADD_MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG_OP(_flow_matvec_acc, '+', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, inp.strides) } /** @@ -809,11 +583,10 @@ void flow_submatvec_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define MV_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, inp.strides - NDIM_SWITCH(SUB_MV_DT) -#undef MV_ARGS + FF_DISPATCH_REG_OP(_flow_matvec_acc, '-', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, inp.strides) } void flow_diag( @@ -845,11 +618,10 @@ void flow_diag( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides - NDIM_SWITCH(DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_flow_diag, '=', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides) } /** @@ -888,11 +660,10 @@ void flow_adddiag_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides - NDIM_SWITCH(ADD_DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_flow_diag, '+', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides) } /** @@ -927,11 +698,10 @@ void flow_subdiag_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define DG_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides - NDIM_SWITCH(SUB_DG_DT) -#undef DG_ARGS + FF_DISPATCH_REG_OP(_flow_diag, '-', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides) } void flow_kernel( @@ -972,11 +742,10 @@ void flow_kernel( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, static_cast(out.ndim) - NDIM_SWITCH(KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_flow_kernel, '=', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, static_cast(out.ndim)) } /** @@ -1021,11 +790,10 @@ void flow_addkernel_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, static_cast(out.ndim) - NDIM_SWITCH(ADD_KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_flow_kernel, '+', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, static_cast(out.ndim)) } /** @@ -1070,11 +838,10 @@ void flow_subkernel_( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define KN_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), \ - voxel_size, absolute, membrane, bending, shears, div, \ - out.shape, out.strides, static_cast(out.ndim) - NDIM_SWITCH(SUB_KN_DT) -#undef KN_ARGS + FF_DISPATCH_REG_OP(_flow_kernel, '-', "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), + voxel_size, absolute, membrane, bending, shears, div, + out.shape, out.strides, static_cast(out.ndim)) } void flow_relax( @@ -1112,12 +879,11 @@ void flow_relax( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RX_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(sol), FF_CVOIDPTR(hes), \ - FF_CVOIDPTR(grd), voxel_size, absolute, membrane, bending, \ - shears, div, nb_iter, sol.shape, sol.strides, hes.strides, \ - grd.strides - NDIM_SWITCH(RX_DT) -#undef RX_ARGS + FF_DISPATCH_REG(_flow_relax, "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(sol), FF_CVOIDPTR(hes), + FF_CVOIDPTR(grd), voxel_size, absolute, membrane, bending, + shears, div, nb_iter, sol.shape, sol.strides, hes.strides, + grd.strides) } void flow_forward( @@ -1272,11 +1038,10 @@ void flow_matvec_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_MV_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), \ - FF_CVOIDPTR(wgt), voxel_size, absolute, membrane, shears, div, \ - out.shape, out.strides, inp.strides, wgt.strides - NDIM_SWITCH(RLS_MV_DT) -#undef RLS_MV_ARGS + FF_DISPATCH_REG(_flow_matvec_rls, "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(inp), + FF_CVOIDPTR(wgt), voxel_size, absolute, membrane, shears, div, + out.shape, out.strides, inp.strides, wgt.strides) } void flow_diag_rls( @@ -1316,11 +1081,10 @@ void flow_diag_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_DG_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(wgt), \ - voxel_size, absolute, membrane, shears, div, \ - out.shape, out.strides, wgt.strides - NDIM_SWITCH(RLS_DG_DT) -#undef RLS_DG_ARGS + FF_DISPATCH_REG(_flow_diag_rls, "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(out), FF_CVOIDPTR(wgt), + voxel_size, absolute, membrane, shears, div, + out.shape, out.strides, wgt.strides) } void flow_relax_rls( @@ -1369,12 +1133,11 @@ void flow_relax_rls( const bound::type bnd = static_cast(bound); const bound::BoundVec bvec(bnd); -#define RLS_RX_ARGS bvec, static_cast(nbatch), FF_VOIDPTR(sol), FF_CVOIDPTR(hes), \ - FF_CVOIDPTR(grd), FF_CVOIDPTR(wgt), voxel_size, absolute, membrane, \ - shears, div, nb_iter, sol.shape, sol.strides, hes.strides, \ - grd.strides, wgt.strides - NDIM_SWITCH(RLS_RX_DT) -#undef RLS_RX_ARGS + FF_DISPATCH_REG(_flow_relax_rls, "flow", + bvec, static_cast(nbatch), FF_VOIDPTR(sol), FF_CVOIDPTR(hes), + FF_CVOIDPTR(grd), FF_CVOIDPTR(wgt), voxel_size, absolute, membrane, + shears, div, nb_iter, sol.shape, sol.strides, hes.strides, + grd.strides, wgt.strides) } FF_NAMESPACE_END(FF_DEVICE) From 65429fabd586075b46c94e8c062213cee859331f Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 20 Aug 2026 22:15:50 +0000 Subject: [PATCH 2/2] docs(reg_dispatch): correct the op/no-op split -- 8 and 5, not 9 and 7 The header said nine dispatch sites thread a compile-time `char` op and seven do not. Counted properly it is eight and five, of thirteen per file: with op addmatvec_, submatvec_, diag, adddiag_, subdiag_, kernel, addkernel_, subkernel_ without op matvec, relax, matvec_rls, diag_rls, relax_rls Comment only; no code change. --- include/fastfields/api/cpu/reg_dispatch.h | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/include/fastfields/api/cpu/reg_dispatch.h b/include/fastfields/api/cpu/reg_dispatch.h index c920560..6725ba7 100644 --- a/include/fastfields/api/cpu/reg_dispatch.h +++ b/include/fastfields/api/cpu/reg_dispatch.h @@ -18,9 +18,9 @@ * copied once per *entry point*, not once per module -- thirteen times in * `reg_field.cpp` alone, fifty-two across the six regulariser TUs, out of * seventy-one in the whole tree. Every copy differed only in the leaf's name - * and, for nine of them, one extra `char` template argument. Both variations - * are parameters here, so the fan is written twice (once per leaf shape) - * instead of fifty-two times. + * and, for eight of the thirteen per file, one extra `char` template + * argument. Both variations are parameters here, so the fan is written twice + * (once per leaf shape) instead of fifty-two times. * * This follows `api/{cpu,cuda}/pushpull_dispatch.h`, which did the same for * the pushpull surface, and it is deliberately NOT a dispatcher generic over @@ -74,10 +74,12 @@ FF_NAMESPACE_BEGIN(FF_DEVICE) ***********************************************************************/ // Two leaf shapes exist in this surface and there is no way to write them as -// one: nine entry points thread a compile-time `char` op ('=', '+', '-') as -// the leaf's second template argument, and seven (the `relax` and `_rls` -// families) have no op concept at all. Both macros take `OP` so that the -// levels above can forward one argument list to either; FF_REG_DT ignores it. +// one. Of the thirteen dispatch sites in each file, eight thread a +// compile-time `char` op ('=', '+', '-') as the leaf's second template +// argument -- the add/sub matvec pair, and diag and kernel in all three +// variants -- while five (plain matvec, relax, and the three `_rls` ops) have +// no op concept at all. Both macros take `OP` so that the levels above can +// forward one argument list to either; FF_REG_DT ignores it. // // `off32_t` is `int32_t` or, under `FF_INDEX32=0`, `int64_t` -- which is how // the index axis collapses onto one instantiation. See core/dispatch.h.