CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894

Merged
rgommers merged 3 commits into
numpy:mainfrom
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged

CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom
agriyakhetarpal:enable-emscripten-ci

Conversation

@agriyakhetarpal

@agriyakhetarpalagriyakhetarpal commented Feb 27, 2024

Copy link
Copy Markdown
Contributor

Description

This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:

  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:
    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on

Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246

@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal

agriyakhetarpal commented Feb 27, 2024

Copy link
Copy Markdown
ContributorAuthor

Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365

@agriyakhetarpalagriyakhetarpal changed the title CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal

agriyakhetarpal commented Feb 27, 2024

Copy link
Copy Markdown
ContributorAuthor

It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)

@rgommersrgommers mentioned this pull request Feb 27, 2024
Closed
@charris

charris commented Feb 27, 2024

Copy link
Copy Markdown
Member

RE:Blas Is this based on the current tip of NumPy main?

@agriyakhetarpal

agriyakhetarpal commented Feb 27, 2024

Copy link
Copy Markdown
ContributorAuthor

RE:Blas Is this based on the current tip of NumPy main?

Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.

@ngoldbaum

ngoldbaum commented Feb 28, 2024

Copy link
Copy Markdown
Member

The BLAS failures are happening on every PR and are unrelated.

ngoldbaum
ngoldbaum reviewed Feb 28, 2024
View reviewed changes

@ngoldbaumngoldbaum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)

Comment threadmeson_cpu/meson.build
hoodmane
hoodmane reviewed Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml Outdated
@agriyakhetarpal

agriyakhetarpal commented Feb 28, 2024

Copy link
Copy Markdown
ContributorAuthor

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.

I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.

ryanking13
ryanking13 reviewed Feb 28, 2024
View reviewed changes

@ryanking13ryanking13 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!

rgommers
rgommers reviewed Feb 28, 2024
View reviewed changes

@rgommersrgommers left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.

the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)

I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.

Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml Outdated
Comment threadmeson_cpu/meson.build Outdated
Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal mentioned this pull request Feb 28, 2024
Merged
@agriyakhetarpal

agriyakhetarpal commented Feb 28, 2024

Copy link
Copy Markdown
ContributorAuthor

I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.

@agriyakhetarpalagriyakhetarpal mentioned this pull request Feb 28, 2024
Closed
@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers approved these changes Feb 29, 2024
View reviewed changes

@rgommersrgommers left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.

@rgommers
rgommers merged commit 3c4ee6b into numpy:mainFeb 29, 2024
@rgommersrgommers added this to the 2.0.0 release milestone Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal deleted the enable-emscripten-ci branch February 29, 2024 08:46
This was referenced Feb 29, 2024
Closed
Closed
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal mentioned this pull request Mar 5, 2024
Closed
agriyakhetarpal added a commit to agriyakhetarpal/scikit-image that referenced this pull request Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
This was referenced Mar 18, 2024
Merged
Closed
Open
@lestevelesteve mentioned this pull request Apr 16, 2024
Closed
This was referenced May 17, 2024
Closed
Open
Closed
Open
@agriyakhetarpalagriyakhetarpal mentioned this pull request May 28, 2024
Merged
@agriyakhetarpalagriyakhetarpal mentioned this pull request Jun 6, 2024
Merged
4 tasks
@agriyakhetarpalagriyakhetarpal mentioned this pull request Oct 22, 2024
Open
This was referenced Nov 15, 2024
Open
Open
@agriyakhetarpalagriyakhetarpal mentioned this pull request Nov 26, 2024
Closed
This was referenced Jul 12, 2026
Merged
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes
+2 more reviewers
@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

component: CI

Projects

None yet

Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy - #25894
Merged
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Feb 29, 2024
Merged
CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy#25894
rgommers merged 3 commits into
numpy:mainfrom 
agriyakhetarpal:enable-emscripten-ci
Conversation
@agriyakhetarpal
@agriyakhetarpalagriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

Contributor
Description
This PR supersedes gh-24603. It adds a CI job to test NumPy v2.0.0.dev0 against a Pyodide (wasm32) runtime. Some of the key changes here are:
  1. A patch has been added to the ci/tools/emscripten/ directory based on an upstream change to Pyodide at Update numpy to 1.26.4 and don't set MESON env variable pyodide/pyodide#4502. This patch ensures that the correct Meson build system (i.e., vendored-meson) is found during the build process.
  2. A new requirements file has been created that contains pure-Python dependencies that can be installed inside a Pyodide virtual environment. It has been placed in the same directory as above.
  3. Some ancillary files have been added to ensure that the WASM wheel is compiled without specialised CPU instructions (SIMD, etc.) and that the build procedure does not need to build against BLAS or LAPACK, which are currently unavailable on WebAssembly.
  4. Various tests have been skipped, such as:

    • All f2py tests, since Fortran cannot run in WASM
    • Some tests for np.where() that require floating point exception support
    • Some tests related to the use of subprocesses to retrieve configurations for how NumPy has been built
    • A bug in Cython, i.e., [BUG] CythonCImports shadowing functionality can confuse pickle cython/cython#5411, has now been resolved. The import cython line was breaking the test discovery because Cython is not supported in Pyodide in-tree yet.
    • and so on
Thanks to @rgommers and the notes provided on gh-24603, all tests pass! Here's a workflow run from my fork where they can be observed: https://github.com/agriyakhetarpal/numpy/actions/runs/8065629246
@agriyakhetarpal
TST: Add test requirements file for Emscripten CI job
e12be3e
`ninja` is unavailable for WASM for now, I have added a new requirements file for this and used a `sys_platform != 'emscripten'` platform marker to ignore the `ninja` dependency on WASM-based platforms.
@agriyakhetarpal
TST: Skip irrelevant tests on WASM platforms
e1e0e84
This commit performs the following:
1. Skip `RuntimeWarnings` on exotic `np.where()` tests on WASM because of the lack of floating point exception support
2. Skip NumPy config tests that use subprocess module on WASM
3. Ignore threaded tests for PRNGs on WASM
4. Remove numpygh-5411 Cython `AttributeError` check. See cython/cython#5411, which is now resolved for Cython>3, and we are at Cython>=3.0.6.
4. For f2py, check compilers only if not on WASM
5. Skip pickle serialisation tests for `stringdtype` on WASM runtimes
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
Pyodide build is passing here as well: https://github.com/numpy/numpy/actions/runs/8066720365
@agriyakhetarpalagriyakhetarpal
changed the title
CI, TST: Re-enable Emscripten/Pyodide CI job for NumPyCI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPyFeb 27, 2024
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
It looks like the build with nightly OpenBLAS wheels is failing – is there something I can do to help here? (cc: @mattip)
@rgommersrgommers
mentioned this pull request
Feb 27, 2024
 Closed

@charris
charris
commented
Feb 27, 2024

Copy link


Copy Markdown

Member
RE:Blas Is this based on the current tip of NumPy main?
@agriyakhetarpal
agriyakhetarpal
commented
Feb 27, 2024

Copy link


Copy Markdown

ContributorAuthor
RE:Blas Is this based on the current tip of NumPy main?
Yes, this has been rebased on top of the main branch, IIUC and contains the latest changes.
@ngoldbaum
ngoldbaum
commented
Feb 28, 2024

Copy link


Copy Markdown

Member
The BLAS failures are happening on every PR and are unrelated.
ngoldbaum
ngoldbaum
reviewed
Feb 28, 2024
View reviewed changes
@ngoldbaumngoldbaum
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future. I didn’t review the github actions changes but the job passes so probably close enough :)
Comment threadmeson_cpu/meson.build
hoodmane
hoodmane
reviewed
Feb 28, 2024
View reviewed changes
Comment thread.github/workflows/emscripten.yml
Outdated

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
All the changes to tests make sense. I have a slight preference for marking tests with xfail(strict=True) for cases where we expect the test might pass in the future.
I just checked; I have been mostly conforming to the pytest skip markers that were already present. Out of the tests, Fortran (f2py) support should be unlikely to come up anytime soon and can be skipped entirely, but the floating-point exception support could be marked with @pytest.mark.xfail(strict=True). However, I am not aware of aware of the developments or the effort required to make them possible on the Pyodide side of things.
ryanking13
ryanking13
reviewed
Feb 28, 2024
View reviewed changes
@ryanking13ryanking13
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me for pyodide-build side. Thanks for working on this @agriyakhetarpal!
rgommers
rgommers
reviewed
Feb 28, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good, nice. A few more small comments/requests.
the floating-point exception support could be marked with @pytest.mark.xfail(strict=True)
I'd prefer not to do that. There are already lots of other identical skips for both floating-point errors and subprocesses, so keeping these the same is both nicer and less churn.
Comment thread.github/workflows/emscripten.yml
Comment thread.github/workflows/emscripten.yml
Outdated

Comment threadmeson_cpu/meson.build
Outdated

Comment threadmeson_cpu/meson.build
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Merged

@agriyakhetarpal
agriyakhetarpal
commented
Feb 28, 2024

Copy link


Copy Markdown

ContributorAuthor
I have resolved all the review comments that were posted; this should now be polished enough for another review, or for merging as and when needed.
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Feb 28, 2024
 Closed

@agriyakhetarpal@rgommers@hoodmane
CI: Re-enable Emscripten/Pyodide CI job for NumPy
d2d2c25
This commit performs the following actions:
1. Adds WASM builds to the CPU family for Meson configurations, but
without SSE or SIMD instructions.
2. Enables `IEEE_QUAD_LE` longdouble format for the wasm32 target
(cross-builds).
3. Enables run for Emscripten/Pyodide wheels by setting the `if:`
condition to `true`.
4. Uses recursive submodules to ensure that vendored-meson is received.
5. Moves the Meson cross file to `tools/ci/emscripten/` (i.e., creates a
separate Emscripten folder to store relevant files)
6. Adds a patch for vendored-meson detection for Pyodide and applies
this Pyodide-meson patch in the Emscripten CI jobs
7. Builds wasm32 wheels without BLAS and LAPACK support (see
numpy#24750 (comment))
8. Forces coloured and prettified outputs for test runs
Some of these changes have been copied with updates and suggestions
received from numpy#24603 on 23/02/2024 and authorship is preserved with this
commit.
[skip cirrus] [skip circle] [skip azp]
Co-Authored-By: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Hood Chatham <roberthoodchatham@gmail.com>
rgommers
rgommers
approved these changes
Feb 29, 2024
View reviewed changes
@rgommersrgommers
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks good now. I did a slight cleanup of the commits that reverted something. In it goes. Nice work @agriyakhetarpal, and thanks for the reviews @hoodmane, @ryanking13 and @ngoldbaum.
@rgommers
rgommers
merged commit 3c4ee6b
into
numpy:mainFeb 29, 2024
@rgommersrgommers
added this to the 2.0.0 release milestone
Feb 29, 2024
@agriyakhetarpal
agriyakhetarpal
deleted the

enable-emscripten-ci

branch
February 29, 2024 08:46

This was referenced Feb 29, 2024

Closed


Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
89c2095
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
e0e4141
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
50cf28b
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
2999f36
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Feb 29, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
1e04e3c
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Mar 5, 2024

Closed

agriyakhetarpal added a commit
to agriyakhetarpal/scikit-image
that referenced
this pull request
Mar 18, 2024
@agriyakhetarpal@lagru
Update Emscripten build workflow and actions
7a90748
Based on numpy/numpy#25894. This comment updates versions for Pyodide, Emscripten, and improves some names for job steps. Multiple reusable actions have also been updated in accordance with the latest versions available.
Co-Authored-By: Lars Grüter <20140352+lagru@users.noreply.github.com>

This was referenced Mar 18, 2024
 Merged


Closed


Open

@lestevelesteve
mentioned this pull request
Apr 16, 2024
 Closed


This was referenced May 17, 2024

Closed

 Open

 Closed

 Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
May 28, 2024
 Merged

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Jun 6, 2024
 Merged

4 tasks
@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Oct 22, 2024

Open


This was referenced Nov 15, 2024

Open


Open

@agriyakhetarpalagriyakhetarpal
mentioned this pull request
Nov 26, 2024

Closed


This was referenced Jul 12, 2026
 Merged

 Merged

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ngoldbaumngoldbaumngoldbaum left review comments
@rgommersrgommersrgommers approved these changes

+2 more reviewers

@hoodmanehoodmanehoodmane left review comments
@ryanking13ryanking13ryanking13 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

component: CI

Projects


None yet


Milestone

2.0.0 release

Development

Successfully merging this pull request may close these issues.

6 participants

@agriyakhetarpal@charris@ngoldbaum@rgommers@hoodmane@ryanking13