bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253)

[tiran]

https://bugs.python.org/issue40280

[hoodmane]

Maybe this should be called --enable-emscripten-dynamic-linking since it doesn't work with wasi or other wasm targets?

[tiran]

Maybe this should be called --enable-emscripten-dynamic-linking since it doesn't work with wasi or other wasm targets?

WASI does not support dynamic linking yet. It may get it in the future, https://helda.helsinki.fi/bitstream/handle/10138/337740/Dynamic_linking_in_WebAssembly_Architecture_and_Performance_Evaluation.pdf .

[hoodmane]

But probably stuff like -s SIDE_MODULE=1 will always be emscripten specific, so even when other wasm platforms gain dynamic linking it is not clear if this flag can work for both.

[hoodmane]

I still think we should discuss with Emscripten whether we could set some environment variable like EMSCRIPTEN_RESPECT_SHARED and then use the -shared flag. Then other wasm targets would just ignore the EMSCRIPTEN_RESPECT_SHARED variable, whereas they would probably get angry when passed nonsense like -s SIDE_MODULE=1 as a command line argument.

cf emscripten-core/emscripten#16481

@sbc100

[tiran]

I still think we should discuss with Emscripten whether we could set some environment variable like EMSCRIPTEN_RESPECT_SHARED and then use the -shared flag. Then other wasm targets would just ignore the EMSCRIPTEN_RESPECT_SHARED variable, whereas they would probably get angry when passed nonsense like -s SIDE_MODULE=1 as a command line argument.

WASI wouldn't see the Emscripten flags. We can easily extend our configure file later to do something like:

AS_VAR_IF([enable_wasm_dynamic_linking], [yes], [
AS_CASE([$ac_sys_system],
[Emscripten], [BLDSHARED='$(CC) -shared -s SIDE_MODULE=1 -s WASM=1'],
[WASI], [BLDSHARED='$(CC) --shared --wasi-magic-linking-flag']
)
])

[hoodmane]

Right, makes sense that autotools has ways of dealing with different compilers needing different flags.

[erlend-aasland]

AC changes look good to me!

[tiran]

The PR drops ac_cv_func_dlopen=no from our config.site override and will allow us to test dynamic linking in the future.

@hoodmane does the PR help you, or at least not cause any new problems?

[hoodmane]

I'm having trouble testing on tip of tree because of the changes to _sre and regex. I need a docker image like 3.11.0a6-slim-buster but with tip of tree. Or maybe I can just locate the changes to _sre and regex since alpha6 and revert them? I can still build locally against a locally built copy of tot Python but it's inconvenient not to have the CI working.

[hoodmane]

Or maybe I can cherry-pick this commit onto 3.11.0a6?

[tiran]

What's the problem with _sre module? GH-32177 converted the modules to a package. The change should be fully backwards compatible.

[hoodmane]

I get failures at
assert _sre.MAGIC == MAGIC, "SRE module mismatch" on line 17 of Lib/sre_compile.py

[tiran]

There is no line 17 in Lib/sre_compile.py any more. The file got moved and replaced with a shorter stub. Some files on your system are out of date.

[hoodmane]

Some files on your system are out of date.

Yeah hence I need a docker image with tip of tree Python because my v3.11.0a6 system is out of date

[tiran]

Are you familiar with out-of-tree builds? We compile a build Python interpreter to bootstrap Emscripten cross build from the same source check, https://github.com/ethanhs/python-wasm/blob/main/build-python-build.sh and https://github.com/ethanhs/python-wasm/blob/main/build-python-emscripten-browser.sh . The trick is ../../configure.

[hoodmane]

Are you familiar with out-of-tree builds?

Apparently not.

The trick is ../../configure

Is it important that there are two layers builddir/build?

[hoodmane]

Also, are there docs on out of tree build somewhere that I should read?

[tiran]

No, the paths are not important. Any directory structure will do. It's only important that you run make clean in the source directory first. And don't use build, it gets easily confused with distutils's build directory. ouf-of-tree builds (aka VPATH builds) are an autoconf feature. https://www.gnu.org/software/automake/manual/html_node/VPATH-Builds.html

[hoodmane]

It's only important that you run make clean in the source directory first.

Do I need to do this if I have cloned a fresh copy of python/cpython?

[tiran]

Do I need to do this if I have cloned a fresh copy of python/cpython?

No, a fresh clone is clean.

[hoodmane]

I tried to follow your directions for this:
https://github.com/hoodmane/pyodide/blob/python-tot/cpython/Makefile
But the build still fails with:

Traceback (mostrecentcalllast):
File"<frozen runpy>", line198, in_run_module_as_mainFile"<frozen runpy>", line88, in_run_codeFile"/src/cpython/installs/python-3.11.0dev0/lib/python3.11/lib2to3/pgen2/driver.py", line21, in<module>importlogging^^^^^^^^^^^^^^File"/src/cpython/installs/python-3.11.0dev0/lib/python3.11/logging/__init__.py", line26, in<module>importsys, os, time, io, re, traceback, warnings, weakref, collections.abc^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File"/src/cpython/installs/python-3.11.0dev0/lib/python3.11/re/__init__.py", line125, in<module>from . import_compiler, _parser^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^File"/src/cpython/installs/python-3.11.0dev0/lib/python3.11/re/_compiler.py", line17, in<module>assert_sre.MAGIC==MAGIC, "SRE module mismatch"^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^AssertionError: SREmodulemismatch

Maybe you can tell me what I'm doing wrong?
https://app.circleci.com/pipelines/github/hoodmane/pyodide/2654/workflows/282f60fd-a7e5-4d22-8688-09f2daddcf03/jobs/28765

[hoodmane]

Ah maybe the problem is just buggy make recipes.

[hoodmane]

I think the_sre.MAGIC crash is triggered while it is trying to log some other error, something about pybuilddir.txt.

[hoodmane]

Okay the problem is on Makefile.pre.in line 2032:

	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
-j0 -d $(LIBDEST) -f \
-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
$(DESTDIR)$(LIBDEST)

Note that this is the build Python (PYTHON_FOR_BUILD) running with the target Python's standard library (-PYTHONPATH=$(DESTDIR)$(LIBDEST)).

[hoodmane]

Or something. I tried manually removing PYTHONPATH from that and it still broke in the same way so I don't know.

[tiran]

Is the error coming from libinstall target? Our python-wasm PoC does not use make install. We are building WASM port in-place and then copy files around manually. I'll investigate after next alpha is out.

I'm merging the PR now to get it into the upcoming alpha. It's unlike to cause you any problems and allows us to test dynamic linking more easily.

[hoodmane]

Yeah I think make libinstall has a bug where when cross compiling it uses the Python standard library of the target Python with the Python executable of the build Python. As long as the target Python and the build Python are compatible for the small number of packages it needs, it works okay. But if there are changes to certain libraries, it breaks.

[tiran]

Congratulations, you have discovered an annoying issue with development during Python's alpha phase. :)

Code is moving fast and breaks often. You should rebuild your build Python interpreter every time you update your checkout or switch branches. It's going to stabilize during beta and won't be an issue once Python 3.11 reaches RC and final stages. I recommend config cache (./configure -C) and ccache (export EM_COMPILER_WRAPPER=ccache + export PATH=/usr/lib/ccache:$PATH) to speed up configure and build steps.

[hoodmane]

Makes sense. At least Python is fast to build. The main annoyance is that if I want to get the CI running I have to set up building Python either in my Dockerfile or in my pipeline.

[hoodmane]

Hmm, I'm still having trouble. Even building a x86 Linux Python and passing it as --with-build-python=, the build is generating invocations to /usr/local/bin/python3.11. The issue seems to be that ./configure is generating a slightly buggy Makefile. A bunch of variables are all on the same line:

PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) /src/cpython/build/Python-3.11.0dev0/builddir/build-host/python

I think this should be:

PYTHON_FOR_BUILD=/src/cpython/build/Python-3.11.0dev0/builddir/build-host/python
_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH)

[brettcannon]

@hoodmane at this point it's probably better to open an issue, else this is liable to get lost since the PR has already been merged.

[hoodmane]

Okay I opened an issue here: https://bugs.python.org/issue47232

[sbc100]

Is there some reason why you add -sWASM=1 here. Since its the default its unlikely you need it.

Also, you can simplify this command line (and the ones elsewhere in this file) by dropping with =1 at the end of all the settings.

[tiran]

Python shared extensions have a .so extension. emcc creates a .so JavaScript file and a .wasm file along the .so file. We want the .so file to contain the .wasm code.

[sbc100]

This is not what the -sWASM=1 option does though. The -sSIDE_MODULE option is enough to ensure that the output is just a wasm file.

If you are building a normal application without -sSIDE_MODULE then you always get a JS file and a wasm file side-by-side (even with -sWASM=1). The -sWASM=1 option (which is the default) simply means "don't convert the wasm file to JS using wasm2js".. which is what happens if you set -sWASM=0 (which is never the default).

-sSIDE_MODULE on its own should always build just a wasm file.

[brettcannon]

Please note this PR is merged already, so it would be best to open a new issue to track this discussion if there's something to change.