GH-23221: [C++] Add support for building with Emscripten by joemarshall · Pull Request #37821 · apache/arrow

joemarshall · 2023-09-21T14:42:36Z

Split from #37696

This is just the cmake changes to enable building on emscripten.

Changes are:

Support for target system "emscripten"
Cmake preset for building libarrow ninja-release-python-emscripten (same as ninja-release-python, but with emscripten support)
Override file for cmake on emscripten, to set various build parameters that need setting to make it build there.
Changes in pyarrow cmake so it works if you are building libarrow as shared library, and also an option to enable the cmake file there to just dump the current arrow configuration, which is useful for cross-compile builds.

Closes: [C++] Add/document configuration to build with WebAssembly #23221

github-actions · 2023-09-21T14:43:03Z

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

In the case of PARQUET issues on JIRA the title also supports:

PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

See also:

joemarshall · 2023-09-21T14:53:19Z

@kou here are the cmake changes for building libarrow in emscripten.

kou

(Sorry. My review isn't completed yet. I'll continue later.)

kou

(My first review is completed.)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

…changes

jorisvandenbossche · 2024-03-29T09:11:06Z

The failing integration test build seems unrelated (it's failing in a rust<->rust test)

bitsondatadev · 2024-04-02T18:02:33Z

The failing integration test build seems unrelated (it's failing in a rust<->rust test)

@jorisvandenbossche and @kou, are there any changes you're waiting for or is this ready for another review?

kou · 2024-04-03T21:05:48Z

Sorry... I should have said #include "arrow/util/config.h" not #include "arrow/util/future.h" for #37821 (comment) and so on...

Could you use arrow/util/config.h not arrow/util/future.h instead...? (Can I push a commit to this branch?)

joemarshall · 2024-04-04T14:00:27Z

Sorry... I should have said #include "arrow/util/config.h" not #include "arrow/util/future.h" for #37821 (comment) and so on...
Could you use arrow/util/config.h not arrow/util/future.h instead...? (Can I push a commit to this branch?)

Yes pushing a commit is fine.

kou · 2024-04-04T14:12:56Z

@github-actions crossbow submit test-ubuntu-22.04-cpp-emscripten

github-actions · 2024-04-04T14:15:18Z

Revision: d3bc3b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-b7535ddb15

Task	Status
test-ubuntu-22.04-cpp-emscripten

kou · 2024-04-04T15:33:16Z

@github-actions crossbow submit -g cpp

github-actions · 2024-04-04T15:35:58Z

Revision: d3bc3b4

Submitted crossbow builds: ursacomputing/crossbow @ actions-fd564c7b19

Task	Status
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-cuda-cpp
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-fedora-39-cpp
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-14

kou

+1

bitsondatadev · 2024-04-05T03:38:20Z

Thanks for your reviews @kou and thanks @joemarshall for all the hard work!!

joemarshall · 2024-04-05T06:36:39Z

Brilliant, thanks everyone

conbench-apache-arrow · 2024-04-05T10:16:51Z

After merging your PR, Conbench analyzed the 7 benchmarking runs that have been run so far on merge-commit 1a1d2c8.

There was 1 benchmark result indicating a performance regression:

Commit Run on ursa-i9-9960x at 2024-04-05 03:34:48Z
- dataset-selectivity (Python) with dataset=nyctaxi_multi_ipc_s3, selectivity=100%

The full Conbench report has more details. It also includes information about 5 possible false positives for unstable benchmarks that are known to sometimes produce them.

raulcd · 2024-04-10T09:37:44Z

@github-actions crossbow submit test-build-vcpkg-win

raulcd · 2024-04-10T09:39:58Z

I am trying to understand if this PR was the one causing the test-build-vcpkg-win job failure.

github-actions · 2024-04-10T09:40:05Z

Revision: 1307910

Submitted crossbow builds: ursacomputing/crossbow @ actions-5c4dbcd949

Task	Status
test-build-vcpkg-win

pitrou · 2024-04-11T12:06:36Z


+# install emscripten using EMSDK
+ARG emscripten_version="3.1.45"
+RUN cd ~ && git clone https://github.com/emscripten-core/emsdk.git && \


Hmm, two things:
it would have been better for resource usage to make a separate dockerfile for this, perhaps
could this be a script to avoid having tons of hand-coded bash code in dockerfiles?

pitrou · 2024-04-11T12:07:08Z

 fi

+# make sure zlib is cached in the EMSDK folder
+RUN source ~/emsdk/emsdk_env.sh && embuilder --pic build zlib


Any reason why this isn't together with the install step above?

…41178) ### Rationale for this change #37821 changed to use `add_test()` usage from old style to new style: 1a1d2c8?diff=unified&w=1#diff-1ce47eec54afaee769086e1a720c5ed65bc347cd8fc60a233de67fd895dda329L763-R764 MSVC generators multi-config generators. With old style, all tests are run without specifying `--build-config` explicitly. With new style, we need to specify `--build-config` explicitly. See also: https://cmake.org/cmake/help/latest/command/add_test.html ### What changes are included in this PR? Specify `--build-config` explicitly. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41169 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

cmake changes for emscripten
5503984

github-actionsBot added Component: C++ Component: Python Component: Documentation awaiting review Awaiting review labels Sep 21, 2023

joemarshall mentioned this pull request Sep 21, 2023
GH-23221: [C++] webassembly / pyodide / emscripten build support #37696
Closed

reverted cmakelists from python
331aaa3

github-actionsBot removed the Component: Python label Sep 21, 2023

kou changed the title ~~GH23221 [C++] cmake changes for emscripten build support~~GH-23221: [C++] Add suppor for building with EmscriptenSep 22, 2023

kou reviewed Sep 23, 2023
View reviewed changes

github-actionsBot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 23, 2023

kou reviewed Sep 25, 2023
View reviewed changes

jorisvandenbossche reviewed Sep 25, 2023
View reviewed changes

Comment threaddocs/source/developers/cpp/emscripten.rst Outdated

jorisvandenbossche reviewed Sep 25, 2023
View reviewed changes

Comment threadcpp/CMakePresets.json Outdated

jorisvandenbossche changed the title ~~GH-23221: [C++] Add suppor for building with Emscripten~~GH-23221: [C++] Add support for building with EmscriptenSep 25, 2023

documentation tidying after review
3e4156e
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

github-actionsBot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 25, 2023

joemarshall added 5 commits September 25, 2023 20:42

preset fixes
80b21d1

updates to simplify emscripten build process
908abf9

Merge branch 'main' of github.com:apache/arrow into emscripten_cmake_…
6effae5
…changes

build fixes
31ca7f9

better comments
c26d68c

github-actionsBot added Component: Python awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Sep 25, 2023

cmake fixes for python emscripten build to work
f5ec444

Use arrow/util/config.h
d3bc3b4

Remove trailing spaces
1307910

kou approved these changes Apr 5, 2024
View reviewed changes

kou merged commit 1a1d2c8 into apache:mainApr 5, 2024

kou removed the awaiting change review Awaiting change review label Apr 5, 2024

github-actionsBot added the awaiting merge Awaiting merge label Apr 5, 2024

joemarshall mentioned this pull request Apr 5, 2024
pyarrow pyodide/pyodide#2933
Closed

bitsondatadev mentioned this pull request Jul 21, 2025
Add Daft Package pyodide/pyodide-recipes#257
Open

pitrou reviewed Apr 11, 2024
View reviewed changes

kou mentioned this pull request Apr 12, 2024
GH-41169: [CI][Release] Specify --build-config explicitly on Windows #41178
Merged

Uh oh!

Conversation

joemarshall commented Sep 21, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Sep 21, 2023

Uh oh!

joemarshall commented Sep 21, 2023

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jorisvandenbossche commented Mar 29, 2024

Uh oh!

bitsondatadev commented Apr 2, 2024

Uh oh!

kou commented Apr 3, 2024

Uh oh!

joemarshall commented Apr 4, 2024

Uh oh!

kou commented Apr 4, 2024

Uh oh!

github-actionsBot commented Apr 4, 2024

Uh oh!

kou commented Apr 4, 2024

Uh oh!

github-actionsBot commented Apr 4, 2024

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

bitsondatadev commented Apr 5, 2024

Uh oh!

joemarshall commented Apr 5, 2024

Uh oh!

conbench-apache-arrowBot commented Apr 5, 2024

Uh oh!

raulcd commented Apr 10, 2024

Uh oh!

raulcd commented Apr 10, 2024

Uh oh!

github-actionsBot commented Apr 10, 2024

Uh oh!

pitrouApr 11, 2024

Choose a reason for hiding this comment

Uh oh!

pitrouApr 11, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

joemarshall commented Sep 21, 2023•
edited
Loading