BLD: build with numpy instead of oldest-supported-numpy by neutrinoceros · Pull Request #478 · pydata/numexpr

neutrinoceros · 2024-04-01T12:00:45Z

I noticed while testing a downstream package that the following ImportError was raised from NumPy 2.0.0rc1 pointing to numexpr:

Traceback

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.0.dev0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
Traceback (most recent call last): File "/opt/hostedtoolcache/Python/3.12.2/x64/bin/pytest", line 8, in <module>
sys.exit(console_main())
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 197, in console_main
code = main()
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/config/__init__.py", line 174, in main
ret: Union[ExitCode, int] = config.hook.pytest_cmdline_main(
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__returnself._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexecreturnself._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 332, in pytest_cmdline_mainreturn wrap_session(config, _main)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 285, in wrap_session
session.exitstatus = doit(config, session) or0
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 338, in _main
config.hook.pytest_collection(session=session)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__returnself._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexecreturnself._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 349, in pytest_collection
session.perform_collect()
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 813, in perform_collectself.items.extend(self.genitems(node))
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 981, in genitemsyield fromself.genitems(subnode)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 981, in genitemsyield fromself.genitems(subnode)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 976, in genitems
rep, duplicate =self._collect_one_node(node, handle_dupes)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/main.py", line 839, in _collect_one_node
rep = collect_one_node(node)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 565, in collect_one_node
rep: CollectReport = ihook.pytest_make_collect_report(collector=collector)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_hooks.py", line 501, in __call__returnself._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_manager.py", line 119, in _hookexecreturnself._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/pluggy/_callers.py", line 102, in _multicall
res = hook_impl.function(*args)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 390, in pytest_make_collect_report
call = CallInfo.from_call(collect, "collect")
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 340, in from_call
result: Optional[TResult] = func()
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/runner.py", line 388, in collectreturnlist(collector.collect())
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 576, in collectself._register_setup_module_fixture()
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 589, in _register_setup_module_fixtureself.obj, ("setUpModule", "setup_module")
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 315, in objself._obj = obj =self._getobj()
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 573, in _getobjreturn importtestmodule(self.path, self.config)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/python.py", line 520, in importtestmodule
mod = import_path(
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/pathlib.py", line 584, in import_path
importlib.import_module(module_name)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/importlib/__init__.py", line 90, in import_modulereturn _bootstrap._gcd_import(name[level:], package, level)
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/_pytest/assertion/rewrite.py", line 178, in exec_moduleexec(co, module.__dict__)
File "/home/runner/work/nonos/nonos/tests/test_plotting.py", line 4, in <module>import numexpr as ne
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/numexpr/__init__.py", line 24, in <module>from numexpr.interpreter importMAX_THREADS, use_vml, __BLOCK_SIZE1__
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.12.2/x64/lib/python3.12/site-packages/numpy/core/_multiarray_umath.py", line 44, in __getattr__raiseImportError(msg)
ImportError: A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.1.0.dev0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.

For more context:

oldest-supported-numpy is superseded by NumPy itself since version 1.25.0
NumPy 2 just reached ABI stability with the release of version 2.0.0rc1 a couple days ago, and forward compatibility with it requires building against this newest version instead of oldest ones.

It is recommended by NumPy developers to plan releases of packages with Python extensions a some point between 2.0.0rc1 and 2.0.0 (final), which from the looks of it should be a couple months from now.

neutrinoceros · 2024-04-01T12:02:32Z

I also updated the minimal requirement for numpy to sync it with python-requires: NumPy 1.19.3 was the first version to support Python 3.9, which is the current minimal requirement.

FrancescAlted · 2024-04-01T12:15:27Z

Your suggestion sounds good to me. However, there are still some issues in building the wheels in CI:

 numexpr/interpreter.cpp:1206:28: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘elsize’
1206 | dtypes[0]->elsize = (int)self->memsizes[1];
| ^~~~~~
numexpr/interpreter.cpp:1452:44: error: ‘PyArray_Descr’ {aka ‘struct _PyArray_Descr’} has no member named ‘elsize’
1452 | self->memsizes[i] = dtypes_tmp[i]->elsize;
| ^~~~~~

neutrinoceros · 2024-04-01T12:32:31Z

Oh, I didn't see this when I tested locally, but luckilly I already had to deal with this exact change in a different package downstream of NumPy ! I should be able to fix this quickly.

neutrinoceros · 2024-04-01T12:45:25Z

I was able to reproduce the crash locally when I switched from Python 3.12 to 3.9, and I just pushed a fix.
For reference, this corresponds to an intentional change in NumPy 2.0's C API, xref numpy/numpy#25943

FrancescAlted · 2024-04-01T15:12:25Z

LGTM. Thanks @neutrinoceros !

maxnoe · 2024-04-18T08:30:16Z

This was included in numexpr 2.10, right? However I see an import failure of numexpr claiming it wasn't compiled against numpy 2.0 here:

https://github.com/PyTables/PyTables/actions/runs/8734526298/job/23965425971?pr=1160#step:7:13

Any idea? Or am I misreadig this and it's not actually numexpr?

neutrinoceros · 2024-04-18T10:38:13Z

It was indeed included in 2.10
Grepping your example logs for numexpr, I see that version 2.8.7 is used

 Requirement already satisfied: numexpr>=2.6.2 in /usr/share/miniconda/envs/test/lib/python3.12/site-packages (from tables==3.9.3.dev0) (2.8.7)

so I think a different dependency may be adding constraints which prevent 2.10 from being installed.

I would recommend running pip list or some variation of it in CI between installation and tests, so it's easier to figure this stuff out :)

maxnoe · 2024-04-18T10:52:37Z

Thanks! I saw 2.10 but that was only in the isolated wheel build environment, indeed in the actual runtime environment, the old version was installed.

jakirkham

Thanks Clément and Francesc! 🙏

It is nice to see the NumPy 2 upgrade in numexpr was relatively straightforward 🥳

Have noted a couple things we might be able to smooth out, which should also make maintenance easier going forward

cc @seberg (for awareness)

jakirkham · 2024-04-30T21:40:37Z

+def_macros = [
+ # keep in sync with minimal runtime requirement (requirements.txt) 
+ ('NPY_TARGET_VERSION', 'NPY_1_19_API_VERSION')
+]


AIUI this shouldn't be needed unless numexpr is doing something more particular with the NumPy API
During the conda-forge NumPy 2 bringup discussion we talked to @rgommers about this, Ralf wrote up a nice explanation here ( conda-forge/conda-forge.github.io#1997 (comment) ). Quoting that below:
numpy's pyproject.toml contains and enforces the lowest-supported version, e.g.: requires-python = ">=3.9"
the first numpy version to support that python version (e.g., 1.19.0 for py39) determined the maximum version that NPY_FEATURE_VERSION may be set to
when that >=3.9 is bumped to >=3.10, then that max-allowed version for NPY_FEATURE_VERSION moves up.
So would suggest dropping
Suggested change
def_macros= [
# keep in sync with minimal runtime requirement (requirements.txt)
('NPY_TARGET_VERSION', 'NPY_1_19_API_VERSION')
]
def_macros= []

jakirkham · 2024-04-30T21:42:08Z

@@ -1 +1 @@
-numpy >= 1.13.3
+numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py)


Similarly the version constraint here could be dropped if we just want to support the oldest NumPy available on Python 3.9. Unless of course there are other reasons than NumPy C API to set a NumPy minimum
Suggested change
numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py)
numpy

jakirkham · 2024-04-30T21:48:13Z

 /* Get the sizes of all the operands */
 dtypes_tmp = NpyIter_GetDescrArray(iter);
 for (i = 0; i < n_inputs+1; ++i) {
- self->memsizes[i] = dtypes_tmp[i]->elsize;
+ self->memsizes[i] = PyDataType_ELSIZE(dtypes_tmp[i]);
 }


Wonder if it is possible to rewrite this with PyArray_ITEMSIZE as that works on NumPy 1 & 2 based on this release note

jakirkham · 2024-04-30T21:51:30Z

 } else { // constant, like in '"foo"'
 dtypes[0] = PyArray_DescrNewFromType(NPY_STRING);
- dtypes[0]->elsize = (int)self->memsizes[1];
+PyDataType_SET_ELSIZE(dtypes[0], (npy_intp)self->memsizes[1]);


Wonder if there is a way to write this so it branches on NPY_ABI_VERSION. That way there could be NumPy 1 & 2 compilation paths

The easiest way is to define the macro if it isn't defined. Without that users can't do a local build with build isolation on old NumPy (which won't allow to compile with numpy 1 and run in 2, that would be harder).
Clearer instructions: https://numpy.org/devdocs/numpy_2_0_migration_guide.html#the-pyarray-descr-struct-has-been-changed

BLD: build with numpy instead of oldest-supported-numpy
49b8e33

neutrinoceros force-pushed the bld/build_with_numpy2.0.0rc1 branch from 6d5a94e to 49b8e33Compare April 1, 2024 12:03

BLD: fix compiling against numpy 2.0
beedecb

FrancescAlted merged commit 23d1a0e into pydata:masterApr 1, 2024

FrancescAlted mentioned this pull request Apr 1, 2024
MNT: use compat functions for accessing dtype state #475
Closed

neutrinoceros deleted the bld/build_with_numpy2.0.0rc1 branch April 1, 2024 15:17

rgommers mentioned this pull request Apr 2, 2024
Ecosystem compatibility with numpy 2.0 numpy/numpy#26191
Closed

This was referenced Apr 30, 2024
NumPy 2.0 support dask/dask#11066
Closed
numexpr v2.10.0 conda-forge/numexpr-feedstock#62
Closed

jakirkham reviewed Apr 30, 2024
View reviewed changes

jakirkham mentioned this pull request May 2, 2024
numpy 2.0.0rc migration conda-forge/conda-forge-pinning-feedstock#5790
Merged

		@@ -1 +1 @@
		numpy >= 1.13.3
		numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py)

	numpy >= 1.19.3 # keep in sync with NPY_TARGET_VERSION (setup.py)
	numpy

Uh oh!

Uh oh!

Conversation

neutrinoceros commented Apr 1, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

neutrinoceros commented Apr 1, 2024

Uh oh!

FrancescAlted commented Apr 1, 2024

Uh oh!

neutrinoceros commented Apr 1, 2024

Uh oh!

neutrinoceros commented Apr 1, 2024

Uh oh!

FrancescAlted commented Apr 1, 2024

Uh oh!

maxnoe commented Apr 18, 2024

Uh oh!

neutrinoceros commented Apr 18, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxnoe commented Apr 18, 2024

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkhamApr 30, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkhamApr 30, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkhamApr 30, 2024

Choose a reason for hiding this comment

Uh oh!

jakirkhamApr 30, 2024

Choose a reason for hiding this comment

Uh oh!

sebergMay 1, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

neutrinoceros commented Apr 1, 2024•
edited
Loading

neutrinoceros commented Apr 18, 2024•
edited
Loading