[Python] Adopt new features and bugfixes from Cython 3.1

[AlenkaF]

Describe the enhancement requested

#47396 will bump minimum required Cython version to 3.1 and as a follow-up we should evaluate and adopt newly available features and bugfixes in PyArrow's Cython codebase. See https://cython.readthedocs.io/en/latest/src/changes.html#beta-1-2025-04-03

Some are already mentioned in the issue #47370:

• The cython command has a new option --cache to cache generated files. (#6091)
• std::span declarations were added to libcpp. (#6539)
• std::string_view declarations were added to libcpp. (#6651)

Component(s)

Python

[raulcd]

@AlenkaF @pitrou I was going to create a different issue for the Cython cache issue in order to split here from other improvements, but I''ve been trying locally and I've hit an issue that looks like a bug on Cython when extracting the artifacts from the cache. I've been able to solve it locally by modifying how Cython extracts the artifacts from the cache:

• [BUG] Cython cache seems to extracts files to wrong folder when finding zip cache files cython/cython#7559

This might require us to wait to enable Cython cache until Cython fixes it. See the issue above for more details.

[raulcd]

As a note for future work, building with:

time python -m pip install -C cmake.build-type=Debug -C build.verbose=true --no-deps --no-build-isolation -vv -e .

The first PyArrow build took:

real	1m7.339s
user	3m30.576s
sys	0m13.188s

Once cached, if the bug mentioned above is locally patched:

real	0m58.111s
user	2m56.534s
sys	0m10.796s

Cache seems to improve just by ~15% as it only caches the generation of the CPP files but the C++ compilation of those is still required. We might have to investigate pairing it with ccache or sccache but scikit-build-core rebuilds in a fresh /tmp/ directory each time that might require some extra configuration.

[jorisvandenbossche]

@raulcd you can try with --config-settings=build-dir=build to reuse the same build directory. With that I got rebuilds of around 10s.

(with conda, I am actually using --config-settings=build-dir=build/$CONDA_DEFAULT_ENV such that if I would have multiple conda envs where I am installing a dev version of pyarrow, it uses separate build dirs)

[raulcd]

Welp, that's nice! Thanks @jorisvandenbossche . We could add a default build-dir configuration to our pyproject.toml.
Using:

time python -m pip install -C cmake.build-type=Debug -C build.verbose=true --no-deps --no-build-isolation -vv --config-settings=build-dir=build -e .

I got on a rebuild:

real	0m11.357s
user	0m15.001s
sys	0m1.548s

That's indeed an amazing improvement.

I've validated that changing a pxd file or a PyArrow C++ file only compiles the required, example:

real	0m40.480s
user	1m54.172s
sys	0m6.618s

[pitrou]

You should be able to use hash_dir = false in your ccache configuration.

[raulcd]

My fix to solve the issue with Cython cache has been merged upstream at the Cython repository. It'll be released as part of 3.2.5, we will be able to add a check on our CMake to use the cache if version is > than that.

[pitrou]

My fix to solve the issue with Cython cache has been merged upstream at the Cython repository. It'll be released as part of 3.2.5, we will be able to add a check on our CMake to use the cache if version is > than that.

FTR, latest Cython release is 3.2.8.

[pitrou]

See also #51139