Uh oh!
There was an error while loading. Please reload this page.
Enable real pthreads for emscripten-wasm32 builds - #154
Open
Tobias-Fischer wants to merge 8 commits into
Open
Conversation
…n builds Emscripten 4.0.9 (the SDK version emscripten-forge/recipes currently publishes as emscripten_emscripten-wasm32) has removed the DEMANGLE_SUPPORT link setting entirely: passing it now hard-errors with `emcc: error: invalid command line setting -sDEMANGLE_SUPPORT=1: No longer supported`, rather than being ignored or warned about. This broke every emscripten-wasm32 ament_cmake package build past the first one or two (confirmed hitting it on ros-humble's rcutils, the first real C library in its selected package set) -- the flag was unconditionally injected into CMAKE_SHARED_LIBRARY_CREATE_C_FLAGS, CMAKE_SHARED_LIBRARY_CREATE_CXX_FLAGS, and CMAKE_EXE_LINKER_FLAGS for every wasm32 shared-library build via this template, both in the live code path and in the dead, already-commented-out per-package branch above it (fixed for consistency in case it's ever re-enabled). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vinca_pinning.yaml's pinning_overrides are copied verbatim into the rendered conda_build_config.yaml by _overlay() -- but rattler-build's variant config loader does not evaluate v1-style `if: COND then: [...]` mappings as selectors the way recipe.yaml files do; it treats each one as an opaque literal value. Depending on how many if/then entries a key has, this surfaces as either "Could not parse version spec for variant key X: invalid channel" (single entry) or "multiple bracket sections not allowed" / a silent cartesian-product explosion across every branch (multiple entries), since rattler-build sees N distinct raw-dict "values" instead of one selector to resolve. The legacy `- VALUE # [selector]` comment-annotated list form, used throughout the rest of this file (c_compiler_version, cxx_compiler_version, etc.), *is* understood and evaluated lazily per --target-platform by rattler-build itself. Fixed by having _overlay() convert any v1-style if/then/else list entries in an override's value into that legacy form before writing it out, so overrides can be authored with the more readable if/then/else syntax (matching emscripten-forge/recipes' own variant.yaml style) while still producing a conda_build_config.yaml rattler-build actually parses as conditional. Also had to make sure existing `# [selector]` EOL comments on passthrough (non-if/then) list items survive the rebuild: ruamel keeps comments keyed by list position on the *source* CommentedSeq, not on the item value itself, so appending an item into a freshly created CommentedSeq silently drops its comment unless it's explicitly copied across -- confirmed by a first pass of this fix accidentally stripping the selectors off c_compiler/c_compiler_version's existing entries, which briefly turned every platform's compiler into a candidate for every other platform's build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Flips USE_PTHREADS=0 to 1 (compile and link) for every emscripten-wasm32 ament_cmake build, plus adds compile-time -s USE_PTHREADS=1 globally via CMAKE_PROJECT_INCLUDE so every translation unit gets the atomics/bulk-memory wasm features the linker requires for shared memory. Without real threads, libc++'s condition_variable timed-wait (used by rclcpp's executor / rmw wait-set) never wakes up on its own -- there is no OS thread to run a real blocking wait, so a synchronous C++ wait blocks the single JS thread forever with no way for wall-clock time to advance underneath it. This was blocking any ROS2 wasm demo from running more than one executor iteration. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… for emscripten rattler-build resolves "build:" dependencies against build_platform, not target_platform, so listing the target-arch-named rosidl_default_generators package under build: for emscripten-wasm32 cross builds only works if a native (non-emscripten) copy of that exact package has also been built and published -- which is not generally the case. The host-prefix copy is already sufficient for CMake's find_package() during configure (confirmed: packages using only rosidl_core_generators in host, with no matching build: entry, already build fine), so this extra requirement was both unsatisfiable and unnecessary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Hardcoding rmw_wasm_cpp as the default RMW for emscripten-wasm32 builds only works for repos that actually build that specific RMW. Read it from VINCA_EMSCRIPTEN_RMW_IMPLEMENTATION instead (falling back to rmw_wasm_cpp unchanged), so other emscripten-wasm32 experiments -- e.g. one building rmw_zenoh_pico instead -- can select their own default without forking this template again. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
STATIC_ROSIDL_TYPESUPPORT_C/CPP were hardcoded to rosidl_typesupport_introspection_c/cpp, which is what rmw_wasm_cpp expects. A micro-ROS-lineage RMW (e.g. rmw_zenoh_pico) instead requires messages' rosidl_typesupport_c dispatch table to register rosidl_typesupport_microxrcedds_c/cpp specifically -- confirmed via a live browser demo that publisher creation fails with "Type support not from this implementation" otherwise, since the dispatch table is filtered down to a single backend at build time and never even considers alternatives at runtime. Purely additive: defaults to the existing introspection backend when the new env vars are unset, so non-zenoh consumers of this template (e.g. ros-humble-emscripten's rmw_wasm_cpp pin) are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
4 tasks
pybind11_add_module() (used by rclpy's _rclpy_pybind11 extension, and any other pybind11-based Python C extension) creates a CMake MODULE-type library, not SHARED -- a distinct target type with its own CMAKE_SHARED_MODULE_CREATE_*_FLAGS variables. Setting only the SHARED ones left every MODULE .so linked without USE_PTHREADS=1: its object files were compiled with atomics support (add_compile_options applies globally) and looked fine individually, but the final linked module was non-shared-memory, causing a load-time "mismatch in shared state of memory" failure alongside the rest of a real-pthreads build. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…pthreads # Conflicts: # vinca/main.py
Tobias-Fischer added a commit
to Tobias-Fischer/ros-rolling
that referenced
this pull request
Sep 9, 2026
…shot vinca's PR (RoboStack/vinca#154) had a merge conflict against current master -- vinca/main.py has since been split into several modules (configuration.py, pipeline.py, recipes.py, sources.py, etc.) by an unrelated upstream refactor. Merged and verified: none of this branch's actual changes touch main.py in a way the refactor didn't already independently resolve (the one main.py hunk this branch touched -- dropping an unsatisfiable build-time rosidl_default_generators requirement for emscripten cross builds -- turned out to already be gone from the refactored file), and `pixi run generate-recipes-emscripten` against the merged vinca commit still produces the same 229-recipe closure with all of this repo's template customizations (real pthreads, configurable RMW_IMPLEMENTATION/typesupport backend) intact. The refactored vinca requires a newer rosdistro_snapshot.yaml schema (a per-package `dependencies:` list) that the snapshot committed here predates -- regenerated via the existing `create_snapshot` task. As a side effect this also re-syncs every package's pinned tag to current rolling (the prior snapshot was from 2026-08-24); spot-checked several packages and the version bumps are real upstream rolling releases, not the migration itself introducing drift. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flips
USE_PTHREADS0→1 (compile and link) for everyemscripten-wasm32ament_cmakebuild inbuild_ament_cmake.sh.in, plus several smaller emscripten-target fixes needed to get there and to support a second, independent emscripten-wasm32 pipeline (ROS 2 rolling +rmw_zenoh_pico, as opposed to the original ROS 2 Humble +rmw_wasm_cppone). All changes are confined to theif [[ $target_platform =~ emscripten.* ]]branch of the template (or are purely additive env-var opt-ins) — no effect on any other platform or on emscripten builds that don't set the new env vars.Built and validated against three full emscripten-wasm32 ROS 2 builds:
rmw_wasm_cpp, running a realrclcpptalker/listener continuously in the browser.rmw_zenoh_pico, running a realrclctalker in the browser that publishes to a nativezenohdrouter over WebSocket, verified end-to-end by an independent native process receiving the message.rmw_zenoh_picobuild running a realrclpytalker (CPython built with--enable-wasm-pthreads) in the browser, also verified end-to-end against a native subscriber — this is what surfaced theCMAKE_SHARED_MODULE_*gap below.Why real pthreads
Without real threads,
libc++'scondition_variabletimed-wait (used byrclcpp's executor and the RMW wait-set machinery) never wakes up on its own — there is no OS thread to run a genuine blocking wait, so a synchronous wait just blocks the single JS thread forever with no way for wall-clock time to advance underneath it. In practice this meant anyrclcpp::spin()-based program executed its first callback and then hung permanently.USE_PTHREADS=1has to be set at both compile and link time for every translation unit (not just the final link) — it bakes in the wasmatomics/bulk-memoryfeatureswasm-ldrequires when producing shared memory. This is applied globally viaadd_compile_options()in the generatedCMAKE_PROJECT_INCLUDEfile, so every package in the tree gets it consistently (mixing a pthread-enabled module with a non-pthread one is a hard ABI-level mismatch — a wasm module's shared-vs-non-shared linear memory is fixed at compile+link time).The other commits
-s DEMANGLE_SUPPORT=1emcc flag, which emscripten 4.0.9 (the SDK version currently published asemscripten_emscripten-wasm32) rejects outright ("No longer supported"), breaking everyemscripten-wasm32ament_cmakebuild past the first one or two._overlay()now converts v1if/then/elseselectors inpinning_overridesvalues into the legacy# [selector]comment form, since rattler-build's variant loader treats the raw if/then mapping as an opaque literal rather than evaluating it.build:-side requirement on a build-platform copy ofrosidl_default_generatorsfor emscripten cross builds — unsatisfiable (no such native package generally exists) and unnecessary (the host-prefix copy already suffices for CMake'sfind_package()).RMW_IMPLEMENTATIONfor emscripten builds configurable viaVINCA_EMSCRIPTEN_RMW_IMPLEMENTATION(falls back to the existingrmw_wasm_cppdefault), so other emscripten-wasm32 pipelines can select their own RMW (e.g.rmw_zenoh_pico) without forking this template.STATIC_ROSIDL_TYPESUPPORT_C/_CPPconfigurable the same way viaVINCA_EMSCRIPTEN_STATIC_TYPESUPPORT_C/_CPP(falls back to the existingrosidl_typesupport_introspection_c/_cppdefault). A micro-ROS-lineage RMW likermw_zenoh_picorequires messages' typesupport dispatch table to registerrosidl_typesupport_microxrcedds_c/_cppspecifically — the previous hardcoded default left that dispatch table with no matching entry, sorcl_publisher_initfailed with "Type support not from this implementation" for every message.CMAKE_SHARED_MODULE_CREATE_C_FLAGS/CMAKE_SHARED_MODULE_CREATE_CXX_FLAGS, alongside the existingCMAKE_SHARED_LIBRARY_CREATE_*_FLAGS. CMake'sMODULElibrary type (whatpybind11_add_module()uses for every Python C extension —rclpy's own_rclpy_pybind11, and every message package'srosidl_generator_pytypesupport accessor) is a distinct target type fromSHAREDand reads its own set of link-flag variables. Setting only theSHAREDones left everyMODULE-type.solinked withoutUSE_PTHREADS=1— its object files were compiled with atomics support fine (add_compile_optionsapplies globally regardless of target type), but the final linked module came out non-shared-memory, which fails to load next to the rest of a pthreads build with a "mismatch in shared state of memory" error. Invisible until something in the tree actually usedpybind11_add_module()— therclc/C-only pipelines never hit it.Testing
Companion changes:
rmw_wasm_cpppipeline this PR was built for).rmw_zenoh_picopipeline exercising the newer commits, including therclpyverification above.Full write-up
All the changes this required, across every repo, are documented together in Tobias-Fischer/ros2-emscripten-zenoh-demo — including a working
rclcandrclpybrowser demo verified end-to-end against a nativezenohdrouter.🤖 Generated with Claude Code