Skip to content

Bump emsdk 4.0.19 -> 5.0.7 (fixes the wasm worker hang) - #6850

Draft
Fedr wants to merge 13 commits into
masterfrom
bump-emsdk-6.0.9
Draft

Bump emsdk 4.0.19 -> 5.0.7 (fixes the wasm worker hang)#6850
Fedr wants to merge 13 commits into
masterfrom
bump-emsdk-6.0.9

Conversation

@Fedr

@Fedr Fedr commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Why

emscripten's emscripten_proxy_finish signalled the proxy context's condition variable after
releasing its mutex:

pthread_mutex_lock(&ctx->sync.mutex);
ctx->sync.state = DONE;
remove_active_ctx(ctx);
pthread_mutex_unlock(&ctx->sync.mutex);
/* preempted here */
pthread_cond_signal(&ctx->sync.cond);

In that window the waiting thread takes the mutex, sees DONE, skips pthread_cond_wait, returns
and destroys the context — which lives on its own stack. The finishing thread then signals a condvar
that is gone, and the thread that should have woken waits forever.

Under -pthread emscripten proxies every syscall synchronously to the main thread, so any
worker doing file I/O could hit it. That is the wasm STEP-import CI stall, and it explains why the
hang moved between unzip, std::filesystem::copy, compressZip and the scene load — the stage was
never the point.

Upstream fix: emscripten-core/emscripten#26582, merged 2026-04-01, shipped in emsdk 5.0.5. We
pin 4.0.19, which predates it.

Measured with a standalone reproducer (two threads doing MEMFS I/O, main thread in a normal event
loop, no MeshLib) at https://github.com/MeshInspector/wasm-memfs-stall — stalls per 480 s shard, 24
shards per cell:

emsdk 4.0.19 4.0.23 5.0.0 5.0.4 5.0.5 5.0.7 6.0.0 6.0.9
stalls ~15% 3 5 3 0 0 0 0 (72 shards)

Zero in 144 shards from 5.0.5 up, against roughly one shard in seven below it.

Why 5.0.7 and not 6.0.9

6.0.9 was the first target, and it kept uncovering work rather than being a bump: PIC needed across
all thirdparty, then emsdk's own zlib port is not PIC either and is not ours to rebuild, then fmt
stopped compiling because clang 24 no longer pulls <cstdlib> in transitively. 5.0.7 is the last of
the 5.x line, carries the fix, and is LLVM 23 rather than 24. Moving to 6.x is worth doing on its
own schedule.

What changed here

  • emscripten/emsdk:4.0.19-arm645.0.7-arm64, and EMSDK_VERSION with it.
  • Base stage ubuntu:22.04ubuntu:24.04, and the apt suites jammynoble, for the same
    glibc reason.
  • CMAKE_POSITION_INDEPENDENT_CODE=ON in the emscripten branch of build_thirdparty.sh. Several
    thirdparty libs are built shared here and link others statically; a wasm .so is a side module,
    so wasm-ld wants every object in it PIC. 4.0.19 tolerated non-PIC archives — libzip.so against
    libmbedcrypto.a, and libjpeg-turbo's sharedlib/ examples — and the newer toolchain does not.
    Every thirdparty script inherits MR_CMAKE_OPTIONS, so it is set once rather than per library,
    and only on the emscripten path.

Deliberately not touched

docker/emscripten-generate-c-bindingsDockerfile and the 4.0.19-arm64 row of
.github/workflows/matrix/docker-images-emscripten-c-bindings.json stay put. That matrix is
intentionally multi-version (it still carries 3.1.38), those images generate bindings rather than
build the shipped wasm, and moving the arm64 row to a noble base additionally needs
VERSION_CODENAME plumbed through as a build-arg in prepare-images.yml. Happy to fold that in if
you would rather they move together.

One temporary commit to undo before merge

The PGO toolchain fetch is commented out, so this builds on emsdk's stock clang. Restore it once
MeshInspector/toolchains publishes clang-emsdk-5.0.7-pgo (building now; emsdk 5.0.7 ships
llvm-project 7b58716d9, clang 23.0.0git).

🤖 Generated with Claude Code

4.0.19 carries the syscall-proxying race emscripten fixed in #26582 and shipped
in 5.0.5: emscripten_proxy_finish signalled the proxy context's condvar after
releasing its mutex, so the waiter could return and destroy the context off its
own stack before the signal landed. Under -pthread every syscall is proxied to
the main thread, so any worker doing file I/O could lose its wakeup and hang.

The base stage moves to noble with it - emsdk images are built on noble from the
5.x line on, and binaries linking glibc 2.39 do not run on jammy - which the apt
suites follow.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fedr and others added 4 commits September 12, 2026 23:04
clang-emsdk-6.0.9-pgo is still building in MeshInspector/toolchains, so the
download 404s. Comment it out to let this PR's CI exercise the emsdk bump now;
restore before merging or the image loses the PGO build-time win.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Its sharedlib/ directory adds example programs that link the .so unconditionally,
and on wasm a .so is a side module, so emcc links those executables as a main
module where every object has to be PIC. 4.0.19 let it through; 6.0.9's wasm-ld
fails them with R_WASM_TABLE_INDEX_SLEB against jpeg_std_error and friends.

This script runs only from the emscripten branch of build_thirdparty.sh, so no
other platform sees the flag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
libjpeg-turbo was not the only one: libzip is built shared here and links
libmbedcrypto.a, whose objects are not PIC either, so 6.0.9's wasm-ld fails that
link too. The thirdparty scripts all inherit MR_CMAKE_OPTIONS, so setting
CMAKE_POSITION_INDEPENDENT_CODE once in the emscripten block covers every one of
them instead of fixing them one at a time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
6.0.9 is LLVM 24 and was turning into a toolchain migration rather than a bump:
PIC across all thirdparty, then emsdk's own non-PIC zlib port, then fmt failing
to compile because clang 24 no longer pulls in <cstdlib> transitively.

5.0.5 is where the syscall-proxying fix landed and 5.0.7 is the last of that
line, at LLVM 23. The reproducer measured 0 stalls in 24 shards on it, same as
6.0.9. 6.x stays worth doing, separately.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Fedr Fedr changed the title Bump emsdk 4.0.19 -> 6.0.9 (fixes the wasm worker hang) Bump emsdk 4.0.19 -> 5.0.7 (fixes the wasm worker hang) Sep 12, 2026
Fedr and others added 8 commits September 13, 2026 00:14
fmt 11.x calls bare malloc/free in format.h and relied on a transitive <cstdlib>
that the libc++ in emsdk 5.x no longer provides. It is an unqualified name in a
non-dependent template body, so every TU including format.h fails to parse, not
just fmt's own os.cc.

fmt fixed this properly in 12.0.0 (std::malloc plus the include), but that pin
also drags spdlog, which is a decision for another PR. -include cstdlib is two
lines, costs nothing, is confined to the emscripten path, and comes out when fmt
moves. In build_thirdparty.sh it has to sit after the flag blocks above, which
rebuild CXXFLAGS from CFLAGS - and it cannot go in CFLAGS, being C++ only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CXXFLAGS export did not reach that configure at all - the failing compile
line carried only flags set in thirdparty/CMakeLists.txt and not one from the
environment. Set it beside the other MR_EMSCRIPTEN flags there instead, which is
also where the -w and -pthread ones live.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The file already says it means to use the deprecated API and wraps it in
MR_SUPPRESS_WARNING, but the newer libc++ attributes the diagnostic to its own
wstring_convert.h rather than to our call, so the pragma never applies and
-Werror fails the wasm build. A per-file flag reaches it; emscripten only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t now

Every non-PIC link error came from 6.0.9 / LLVM 24; none has appeared on 5.0.7.
Meanwhile both apps now fail at wasm-emscripten-finalize with a binaryen parse
exception, and feeding PIC thirdparty objects into a module that is not
relocatable is the most plausible thing in this branch to produce a binary
binaryen cannot read. If the link errors come back, this goes back in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
emscripten attaches findMatchingCatch's dependency on $setTempRet0 only under
!DISABLE_EXCEPTION_CATCHING, while the stub that actually calls setTempRet0 is
the DISABLE_EXCEPTION_CATCHING one - so with catching off, as we build, nothing
emits the symbol and --closure=1 fails with JSC_UNDEFINED_VARIABLE.

Worth reporting upstream; until then DEFAULT_LIBRARY_FUNCS_TO_INCLUDE pulls it in.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The thirdparty copy is proven - fmt's own os.cc will not compile without it. This
one covers MeshLib and MeshInspector sources, which reach format.h through
MRPch -> spdlog, and has never been tested on its own. If the app builds go red,
it goes straight back.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
clang-emsdk-5.0.7-pgo published (clang 23.0.0git, both arches). Restores the
toolchain swap this image has always done; the stock ThinLTO-only clang was only
ever a stand-in while the toolchain built.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant