Uh oh!
There was an error while loading. Please reload this page.
Point SparseDiffEngine submodule at current engine main - #21
Merged
Conversation
Engine main now contains the Swedish sparsity-fill algorithm, the Accelerate header pragma (#106), and the param_source mark-refresh fix (#107); the previous pointer (7e7678e) predates all three, so builds from main carried the stale parametric re-solve bug fixed by #107. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Uh oh!
There was an error while loading. Please reload this page.
Transurgeon added a commit
that referenced
this pull request
Sep 7, 2026
The audit in 2e19844 was performed against engine d8cb9b8, but the submodule pointer was left at 4172c5e from #21, nine commits behind. The two revisions differ in exactly the way the audit depended on. Engine 6e64401 ("Make peak-memory tracking a compile-time option (SP_TRACK_MEMORY)", merged one day before this branch) puts the g_allocated_bytes / g_peak_bytes counters behind an option that defaults to OFF, precisely so the library can be called from several threads at once. At 4172c5e those counters are unconditional, and every sp_malloc / sp_free updates them non-atomically. So the audit's "no writable data/bss symbols" finding was true of the engine it inspected and false of the engine this branch ships: nm on the cp313t extension built from 4172c5e lists _g_allocated_bytes and _g_peak_bytes as its only external writable data. That race is reachable from the usage the README blesses, since two threads building or evaluating *distinct* problems both allocate. Bumping the pointer removes it: nm on the rebuilt extension shows no mutable globals at all, and the engine's own ctest suite passes at d8cb9b8. Verified on cpython-3.13.5+freethreaded, 8 threads: * independent problem per thread, 300 evaluations each, every result bit-identical to a single-threaded reference: 10/10 runs clean. * sharing one expression capsule across threads, which the README forbids: 10/10 runs die with SIGSEGV / SIGBUS / SIGABRT / SIGTRAP. The second number is why the README and the bindings.c comment no longer describe that contract as "the same contract as with the GIL". expr::refcount is a plain int updated non-atomically by expr_retain() / free_expr(), and capsule destructors run on whichever thread drops the last Python reference, so breaking the rule now corrupts the heap instead of interleaving calls. The same misuse is harmless under the GIL. Making that count atomic upstream would turn it back into ordinary unsupported usage; until then the docs say plainly how sharp the edge is. Both documents also note that SP_TRACK_MEMORY must stay off in a wheel build, since turning it on silently reintroduces the global-counter race. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Transurgeon added a commit
that referenced
this pull request
Sep 8, 2026
… support (#24) * Build free-threaded (cp313t/cp314t) wheels and declare GIL-free support Closes#18. The extension now calls PyUnstable_Module_SetGIL(Py_MOD_GIL_NOT_USED) on free-threaded CPython, so importing it no longer re-enables the GIL. The build matrix gains an `abi` axis ("" / "t") for 3.13 and 3.14, opts in via CIBW_ENABLE=cpython-freethreading, and the wheel test asserts the GIL stays disabled after import on the t builds. Audit backing the declaration (engine d8cb9b8 + these bindings): - nm on libdnlp_diff.a shows no writable data/bss symbols; the only globals in the engine are the SP_TRACK_MEMORY byte counters, which the wheel build does not enable. No function-local statics, no non-reentrant libc. - All version counters (values_version, *_seen) are per-matrix/per-expr. - Every problem/expr owns its buffers; forward, update_params and the problem constructor memcpy their inputs, and every wrapper copies results into fresh NumPy arrays, so no Python memory is aliased across calls. - The only static state in the bindings is the NumPy-init flag and the module def, both written once at import under the import lock. - expr refcounts are plain ints, so a single expr/problem capsule must not be used from two threads at once -- the same contract as with the GIL. Verified locally on Homebrew python3.14t: import keeps the GIL disabled and 16 threads x 200 evaluations on distinct problems are bit-identical to a single-threaded reference (forward, jacobian, hessian). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013aQs1u37EG6XTHFrcaBRkA * Drop 3.13t from the wheel matrix; run twine via python -m NumPy >= 2.5 ships no cp313t wheels on any platform, so the 3.13t builds had to compile NumPy from source: macOS failed outright, Windows took six minutes for the NumPy build alone, and Linux aarch64 under QEMU was still running after an hour. Only 3.14t is built now. The macOS 3.14t job built and tested both wheels fine (GIL stays disabled on x86_64 and arm64) but then failed in the twine step with "pip: command not found". Call pip and twine via python -m so the step does not depend on a pip launcher being on PATH. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013aQs1u37EG6XTHFrcaBRkA * Bump engine to d8cb9b8 so the GIL-free declaration actually holds The audit in 2e19844 was performed against engine d8cb9b8, but the submodule pointer was left at 4172c5e from #21, nine commits behind. The two revisions differ in exactly the way the audit depended on. Engine 6e64401 ("Make peak-memory tracking a compile-time option (SP_TRACK_MEMORY)", merged one day before this branch) puts the g_allocated_bytes / g_peak_bytes counters behind an option that defaults to OFF, precisely so the library can be called from several threads at once. At 4172c5e those counters are unconditional, and every sp_malloc / sp_free updates them non-atomically. So the audit's "no writable data/bss symbols" finding was true of the engine it inspected and false of the engine this branch ships: nm on the cp313t extension built from 4172c5e lists _g_allocated_bytes and _g_peak_bytes as its only external writable data. That race is reachable from the usage the README blesses, since two threads building or evaluating *distinct* problems both allocate. Bumping the pointer removes it: nm on the rebuilt extension shows no mutable globals at all, and the engine's own ctest suite passes at d8cb9b8. Verified on cpython-3.13.5+freethreaded, 8 threads: * independent problem per thread, 300 evaluations each, every result bit-identical to a single-threaded reference: 10/10 runs clean. * sharing one expression capsule across threads, which the README forbids: 10/10 runs die with SIGSEGV / SIGBUS / SIGABRT / SIGTRAP. The second number is why the README and the bindings.c comment no longer describe that contract as "the same contract as with the GIL". expr::refcount is a plain int updated non-atomically by expr_retain() / free_expr(), and capsule destructors run on whichever thread drops the last Python reference, so breaking the rule now corrupts the heap instead of interleaving calls. The same misuse is harmless under the GIL. Making that count atomic upstream would turn it back into ordinary unsupported usage; until then the docs say plainly how sharp the edge is. Both documents also note that SP_TRACK_MEMORY must stay off in a wheel build, since turning it on silently reintroduces the global-counter race. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Tidy the free-threaded wheel build configuration No functional change to the default builds. CIBW_ENABLE=cpython-freethreading was a no-op here. cibuildwheel 3.3.0 gates only cp313t behind that group (selector.py checks fnmatch(build_id, "cp313t-*") before consulting the enable set); cp314t is selectable by default, and this matrix builds no cp313t. The option is also deprecated in 3.4.1 and removed in 4.0, so leaving it set would break a later cibuildwheel bump for no benefit. The matrix excludes now say why each one is there. 3.11t and 3.12t are not build identifiers at all, since free threading starts at 3.13, so without the exclude cibuildwheel selects nothing and the job fails on an empty wheelhouse. 3.13t is a deliberate choice about NumPy instead: no cp313t wheels are published from 2.5 onwards, 2.4.6 being the last, which was confirmed by installing numpy on a free-threaded 3.13 and watching 2.5.3 build from source. CIBW_TEST_COMMAND uses bool(...) rather than == 1 on the Py_GIL_DISABLED config var. The var is always an int on 3.13+, on Windows too, so this is not a fix, just the idiom the free-threading porting guide uses. It also prints the flag so the log shows which ABI the job exercised. cmake.version moves to >=3.30.3. CMake 3.30 is the first whose FindPython3 understands the free-threaded "t" ABI, and scikit-build-core only sends the Python3_FIND_ABI hint when the configured specifier admits 3.30+, so at >=3.15 a 3.15-3.29 system CMake would silently skip it. 3.30.3 adds the Python3_DEFINITIONS that a Windows free-threaded build needs, since PC/pyconfig.h relies on Py_GIL_DISABLED being defined to auto-link python3XXt.lib. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Trim the comments added in the previous two commits The rationale belongs in these commit messages, not inline. Cuts the Py_MOD_GIL_NOT_USED block from 23 lines to 5, the cmake.version note from 4 lines to 1, and the free-threading README section back to roughly its original length, keeping the two facts a reader needs: SP_TRACK_MEMORY must stay off, and expr::refcount being a plain int makes a shared capsule fatal rather than merely unsupported. No functional change. Rebuilt on cpython-3.13.5+freethreaded: the import still leaves the GIL disabled and 8 threads on independent problems stay bit-identical to a single-threaded reference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Collapse the wheel matrix back to one python-version axis Drop the separate abi axis and its three excludes; "3.14t" goes straight into python-version instead. The existing subversion step already handles it, since cut -c 3- of "3.14t" is "14t" and CIBW_BUILD becomes cp314t-*. Artifact names stay unique without the suffix, and the job count is unchanged at 15. Two excludes only existed because free threading starts at 3.13, so cp311t and cp312t were dead combinations the axis generated; listing the ABIs that do exist avoids generating them in the first place. The host interpreter for the 3.14t job is now free-threaded as well. That only affects the twine step, since pypa/cibuildwheel runs its own setup-python pinned to "3.11 - 3.13" with update-environment false, and actions/python-versions publishes free-threaded 3.14 builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: William Zijie Zhang <william@gridmatic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the submodule pointer
7e7678e→4172c5e(current engine main).Engine main now contains the Swedish sparsity-fill algorithm, the Accelerate header pragma (SparseDiffEngine#106), and the param_source mark-refresh fix (SparseDiffEngine#107). The previous pointer predates all three, so source builds from SparseDiffPy main carried the stale parametric re-solve bug fixed by #107 (see cvxpy#3455).
All three commits are already shipping in the
v0.6.1release cut fromrelease/0.6.x; this just bringsmainin line.🤖 Generated with Claude Code