[Interop #478] Benchmark Python→Flow package overhead, zero-copy paths and break-even batch sizes

[godofecht]

Parent: #478

flow-scikit also exposes Python interop, so Python callers can erase native wins through conversion, module loading or per-call FFI overhead. Treat that boundary as a separate performance surface.

Measure import/startup, estimator construction, ndarray/buffer handoff, copies/conversions, fit/predict call overhead and return conversion. Add zero-copy contiguous paths where ABI-safe, explicit behavior for non-contiguous/dtype-mismatched arrays, and batch-size crossover curves.

Acceptance: same-process Python wrapper vs direct sklearn and direct Flow measurements; copy bytes/counts reported; no hidden full-array copies on supported contiguous inputs; documented break-even sizes; representative wrapper workloads remain faster end-to-end where direct Flow has enough headroom.

[godofecht]

I worked the Python→Flow boundary as a concrete buffer-contract problem and archived the full derivation + Flow experiment in OpenProjects.

Two actionable findings:

1. The current wrapper's zero-copy admission rule is incomplete. _matrix() uses np.asarray(..., dtype=np.float32, order="C") and then passes .ctypes.data to C as float *, but it never checks array.flags.aligned. NumPy permits C-contiguous, float32, unaligned ndarrays; I reproduced one with np.ndarray(..., buffer=bytearray(...), offset=1), and np.asarray(..., dtype=np.float32, order="C") preserves it. NumPy's C API treats alignment as an independent requirement (NPY_ARRAY_IN_ARRAY = C-contiguous + aligned). So alignment should be part of the native borrow contract for both inputs and outputs.

2. The current f32-only ABI makes common C-contiguous float64 input inherently O(N) at the Python boundary. For a 1797×64 Digits-shaped matrix, the required f32 materialization is 460,032 bytes; a one-read/one-write traffic lower bound for f64→f32 is 1,380,096 bytes. In a local NumPy-only isolation run, the same-shape np.asarray path was ~0.126 µs for already-compatible C/f32 versus ~50.8 µs for C/f64→f32. These are not production benchmark numbers, but they show why bridge time must be reported separately.

The exact crossover condition is useful for #478/#496. If native Flow is F_F + P_F N, sklearn is F_S + P_S N, and bridge cost is F_B + P_B N, then Python-visible Flow has any asymptotic win only if P_S - P_F - P_B > 0. Otherwise conversion has consumed the entire native throughput advantage and no batch size fixes it.

Suggested production bridge: validate/borrow NumPy arrays in the C extension using the NumPy C API, with an explicit Borrow / Repair-layout-alignment / Convert-dtype path, and benchmark those paths separately. Consider an f64 ABI only for estimators where float64 compatibility matters enough to justify maintaining it.

OpenProjects: ml/zero-copy-python-flow-buffer-contract.md and adjacent .flow experiment. No hosted workflow changes/runs were added.