GH-41910: [Python] Add support for Pyodide by joemarshall · Pull Request #37822 · apache/arrow

joemarshall · 2023-09-21T14:51:37Z

pyarrow knows about ARROW_ENABLE_THREADING and doesn't use threads if they are not enabled in libarrow.

Split from #37696

GitHub Issue: [Python] Add support for Emscripten Python (Pyodide) #41910

joemarshall · 2023-09-21T14:53:30Z

@kou And these are the python changes

jorisvandenbossche · 2023-09-25T12:33:41Z

@joemarshall thanks for the PR!

We might want to expose is_threading_enabled() in pyarrow publicly (it might be useful for downstream packages as well?), in __init__.py

2. pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide).

This part is not actually included here? (or I don't understand the sentence)
(and it might also make sense to leave that for a third PR since it is changing the build process beyond emscriptem?)

joemarshall · 2023-09-28T11:27:48Z

@joemarshall thanks for the PR!
We might want to expose is_threading_enabled() in pyarrow publicly (it might be useful for downstream packages as well?), in __init__.py
pyarrow sets defaults for inclusion of submodules based on their inclusion in the arrow build. e.g. pyarrow.parquet is built only if ARROW_PARQUET is set. This makes it possible to build in situations where you don't have access to set the build environment variables (e.g. in cross compiling situations like pyodide).
This part is not actually included here? (or I don't understand the sentence) (and it might also make sense to leave that for a third PR since it is changing the build process beyond emscriptem?)

Sorry, I missed out putting in the setup.py changes. They're in now.

About is_threading_enabled(), it is currently in pyarrow.lib.is_threading_enabled(). Does it need to be top-level?

joemarshall · 2023-09-28T11:28:23Z

Oh and for now I have blocked the auto-setting of PYARROW_* to happen only on emscripten - I don't know if that makes sense or not, but it isn't possible to build for emscripten without that change or something similar right now.

kou · 2024-07-03T08:48:54Z

@jorisvandenbossche @pitrou Do you want to review the Python part?

ianmcook · 2024-07-03T15:36:42Z

@kou I believe Joris and Antoine are mostly away this week. Joris reviewed this earlier and communicated that he would like this to be merged in time for 17.0.0 if possible. Let's please not block waiting for additional reviews of the Python parts, unless you have concerns.

kou

OK. I've reviewed the Python part too.

kou · 2024-07-04T01:24:23Z

+
+ print("Load pyodide in browser")
+ driver.load_pyodide(dist_dir)
+ print("Done\n")


Is \n needed?

@joemarshall I see the print("Done") is removed entirely now. Is that what you intended?

ianmcook · 2024-07-04T18:11:31Z

@joemarshall if you're able to react to Kou's review in the next day or so, then I think we can get this merged in time for the 17.0.0 release

joemarshall · 2024-07-04T18:59:15Z

@joemarshall if you're able to react to Kou's review in the next day or so, then I think we can get this merged in time for the 17.0.0 release

Brilliant, I'll get onto it either later on today or first thing tomorrow

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

joemarshall · 2024-07-05T09:54:32Z

@ianmcook changes merged in, running tests locally right now, could probably fire off the emscripten tests here to be sure also

joemarshall · 2024-07-05T10:42:24Z

Tests pass here, so hopefully we're ready to merge.

ianmcook · 2024-07-05T12:02:11Z

@github-actions crossbow submit test-conda-python-emscripten

github-actions · 2024-07-05T12:05:34Z

Revision: fa0e497

Submitted crossbow builds: ursacomputing/crossbow @ actions-e5764a79dd

Task	Status
test-conda-python-emscripten

ianmcook · 2024-07-05T12:45:23Z

@github-actions crossbow submit -g wheel

github-actions · 2024-07-05T12:48:26Z

Revision: fa0e497

Submitted crossbow builds: ursacomputing/crossbow @ actions-a1c5cd96da

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

ianmcook · 2024-07-05T14:19:18Z

@joemarshall could you please sync your fork with upstream main? I think that will resolve the CI failures.

joemarshall · 2024-07-05T15:19:36Z

@joemarshall could you please sync your fork with upstream main? I think that will resolve the CI failures.

Try now

raulcd · 2024-07-05T15:23:00Z

@github-actions crossbow submit -g wheel

github-actions · 2024-07-05T15:25:50Z

Revision: 71a2f6a

Submitted crossbow builds: ursacomputing/crossbow @ actions-8f6301a1d5

Task	Status
wheel-macos-big-sur-cp310-arm64
wheel-macos-big-sur-cp311-arm64
wheel-macos-big-sur-cp312-arm64
wheel-macos-big-sur-cp38-arm64
wheel-macos-big-sur-cp39-arm64
wheel-macos-catalina-cp310-amd64
wheel-macos-catalina-cp311-amd64
wheel-macos-catalina-cp312-amd64
wheel-macos-catalina-cp38-amd64
wheel-macos-catalina-cp39-amd64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

ianmcook · 2024-07-05T21:15:57Z

Checks look good to me. The CI failures are unrelated.

@kou do you think this is good to merge now? If so please go ahead.

kou

+1

conbench-apache-arrow · 2024-07-06T04:53:11Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 2de8008.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 31 possible false positives for unstable benchmarks that are known to sometimes produce them.

jorisvandenbossche · 2024-07-25T16:38:31Z

Thanks @joemarshall for the amazing effort and everyone for the reviews!

python changes for pyodide build
eda86ae

joemarshall mentioned this pull request Sep 21, 2023
GH-23221: [C++] webassembly / pyodide / emscripten build support #37696
Closed

github-actionsBot added Component: Python awaiting review Awaiting review labels Sep 21, 2023

jorisvandenbossche changed the title ~~GH23221 - python changes for pyodide build~~GH-23221: [Python] python changes for pyodide buildSep 25, 2023

apache deleted a comment from github-actionsBotSep 25, 2023

joemarshall added 2 commits September 28, 2023 12:23

included setup.py changes
1fe1845

setup fixes for pyodide
7579749

joemarshall added 6 commits September 28, 2023 12:29

fix so pyodide build works as default
1bda7a9

python formatting
ef1ce8e

fix emscripten checks
7166f46

emscripten check fixes
a6e78f5

removed cmakelists.txt from python pr
61d95ad

removed unnecessary change
bef91cb

kou mentioned this pull request Oct 1, 2023
GH-23221: [C++] Add support for building with Emscripten #37821
Merged

moved python cmake changes into python pr
7b806c4