Uh oh!
There was an error while loading. Please reload this page.
FIX: run Debian ARM64 CI on preinstalled-python image to avoid QEMU segfault - #721
Open
Jahnvi Thakkar (jahnvi480) wants to merge 5 commits into
Open
FIX: run Debian ARM64 CI on preinstalled-python image to avoid QEMU segfault#721Jahnvi Thakkar (jahnvi480) wants to merge 5 commits into
Jahnvi Thakkar (jahnvi480) wants to merge 5 commits into
Conversation
The Linux ARM64 legs run under QEMU user-mode emulation on x86_64 agents. Older QEMU (multiarch/qemu-user-static) mishandles glibc restartable sequences (rseq), randomly SIGSEGV-ing (exit 139) the emulated arm64 Python during Debian's apt byte-compilation and forcing repeated job reruns (build 166395's Debian_ARM64 job needed 5 attempts). Two independent, root-cause guards scoped to the flaky Debian/Ubuntu ARM64 job: - Emulator level: pin a modern QEMU (tonistiigi/binfmt qemu-v9.2.2), which fixes rseq emulation (QEMU >= 7.2). - Guest level: disable rseq on the Debian container via GLIBC_TUNABLES=glibc.pthread.rseq=0. No retryCountOnTaskFailure, no skipped tests. RHEL9 ARM64 and Alpine legs are untouched. AB#47276
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the ARM64 Linux PR-validation CI job to eliminate intermittent Debian ARM64 segfaults under QEMU user-mode emulation by switching to a newer pinned QEMU binfmt image and disabling glibc rseq inside the Debian container.
Changes:
- Replace
multiarch/qemu-user-staticwith pinnedtonistiigi/binfmt:qemu-v9.2.2for ARM64 emulation. - Add Debian-only
GLIBC_TUNABLES=glibc.pthread.rseq=0during ARM64 test container creation to disable rseq.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Address PR review: bare `--install arm64` is a no-op when the hosted agent already has an arm64 handler registered (kernel returns EEXIST and tonistiigi/binfmt leaves the existing, possibly older QEMU in place, logging but not failing). That could silently keep the old buggy QEMU on the Ubuntu_ARM64 leg, which has no guest-side GLIBC_TUNABLES guard. tonistiigi/binfmt has no `--reset` flag (that was multiarch/qemu-user- static); the equivalent is `--uninstall arm64 --install arm64` (uninstall runs before install in run()). `--uninstall arm64` resolves through the arch config to the qemu-aarch64 handler via the -aarch64 suffix match and is a harmless no-op when nothing is registered. Also reword the container-creation comment to clarify that only the GLIBC_TUNABLES/rseq-disable guard is Debian-scoped; the ARM64 emulation is shared by both matrix legs. AB#47276
Drop the emulator-side QEMU swap (pinned tonistiigi/binfmt + uninstall/install) and revert the buildx step to the original multiarch/qemu-user-static registration. The crash mechanism is glibc rseq; disabling it in the guest via GLIBC_TUNABLES=glibc.pthread.rseq=0 neutralizes the root cause deterministically, without depending on binfmt registration order or agent pre-registration state. Apply the guard unconditionally to both ARM64 matrix legs (Debian 12 / glibc 2.36 and Ubuntu 22.04 / glibc 2.35) -- both share the same emulated-arm64 path and both glibc versions ship rseq -- which also removes the Ubuntu-leg exposure that motivated the emulator swap. Net change vs main is now a single env var; no new image dependency. AB#47276
…fault The Debian ARM64 leg SIGSEGVs (exit 139) during the apt 'Setting up python3' post-install byte-compilation under QEMU user-mode emulation (proven by build 167144, which ran the prior guest-side GLIBC_TUNABLES=glibc.pthread.rseq=0 guard and still crashed). Disabling rseq does not fix this emulator-level bug. Instead, avoid the crashing operation entirely: run the Debian leg on the python:3.11-bookworm image (Debian 12 + Python 3.11 already installed and byte-compiled natively at image-build time) and stop apt-installing the python3* packages. Python (pip, venv, dev headers) comes from /usr/local, pybind11 is installed via pip from requirements.txt, and only non-Python build tools (cmake, curl, wget, gnupg, build-essential) are pulled from apt -- none of which depend on the Debian python3 package. The ineffective rseq guard is removed. AB#47276
📊 Code Coverage Report
Diff CoverageDiff: main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. 📋 Files Needing Attention📉 Files with overall lowest coverage (click to expand)mssql_python.pybind.logger_bridge.cpp: 59.2%
mssql_python.pybind.ddbc_bindings.h: 59.9%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python.pybind.ddbc_bindings.cpp: 76.6%
mssql_python.__init__.py: 77.6%
mssql_python.row.py: 77.6%
mssql_python.ddbc_bindings.py: 79.6%
mssql_python.pybind.connection.connection_pool.cpp: 81.4%
mssql_python.pybind.connection.connection.cpp: 84.3%
mssql_python.logging.py: 85.5%🔗 Quick Links
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Debian ARM64 leg of the PR validation pipeline SIGSEGVs (exit 139) during
the apt
Setting up python3post-install byte-compilation under QEMU user-modeemulation on the x86_64 hosted agent. This is an emulator-level crash in the
Debian
python3package's post-install script, not something the guest cansuppress: build 167144 ran a
GLIBC_TUNABLES=glibc.pthread.rseq=0guard andstill crashed at exactly the same point, disproving the earlier rseq theory.
This change avoids the crashing operation entirely instead of trying to make
QEMU emulate it correctly:
python:3.11-bookworm(Debian 12 + CPython3.11 already installed and byte-compiled natively at image-build time)
instead of
debian:12.python3*packages (the trigger). Python — with pip, venv and dev headers —comes from
/usr/local;pybind11comes from pip (already inrequirements.txt); CMake auto-detects both from the active interpreter, sothe apt
python3-dev/pybind11-devpackages are redundant. Only non-Pythonbuild tools (
cmake curl wget gnupg build-essential) are apt-installed, noneof which depend on the Debian
python3package.software-properties-commonwas unused (the ODBC repo is added via
curl+dpkg -i, notadd-apt-repository).GLIBC_TUNABLESrseq guard is removed.The Ubuntu ARM64 leg is unchanged (it does not crash). Pytest subprocess
segfaults under QEMU are a separate concern and will be
skipif-guarded.AB#47276