feat(build): add glibc-static supervisor libc variant

[EmilienM]

Summary

Adds an opt-in glibc-static supervisor libc build variant, selectable via SUPERVISOR_LIBC alongside the default musl. Both variants produce a fully static supervisor binary, verified by a hardened verify-static-binary.sh wired into both local staging and CI. The default is unchanged, so image, release, and CI behavior is identical.

Related Issue

Related to #2053. musl's getaddrinfo returns zero addresses for external FQDNs under Kubernetes' default ndots:5, breaking supervisor DNS resolution. This PR adds the mechanism to build the supervisor against glibc (statically linked) as an alternative to musl. It keeps musl as the default and does not switch the shipped image, so it is an enabling step and does not close#2053 on its own.

Changes

• Add SUPERVISOR_LIBC (musl | glibc-static) to tasks/scripts/stage-prebuilt-binaries.sh; glibc-static builds the GNU target with -C target-feature=+crt-static.
• glibc-static has no cross-compile path (zig cc links glibc dynamically despite -static), so the staging script refuses cross-arch requests for that variant and requires a native per-architecture build.
• Add tasks/scripts/verify-static-binary.sh and run it after every supervisor build in both the staging script and CI. It fails closed on anything that is not a genuine, complete static executable — see "Verifier hardening" below.
• Add a supervisor-libc input to .github/workflows/rust-native-build.yml, partition the sccache/rust-cache keys per libc variant, and add a "Verify static linkage" step for the sandbox component.
• Add .github/workflows/supervisor-static-validate.yml — a dedicated CI workflow that builds the glibc-static variant on both amd64 and arm64 and runs the verifier (no image/release workflow builds it). Following the repo's self-hosted-runner convention (self-hosted runners reject pull_request-triggered jobs), it validates in the merge queue, on pushes to main touching the build inputs, nightly, and via workflow_dispatch.
• Update deploy/docker/Dockerfile.supervisor comment and architecture/build.md to document the variant and its constraints.

Verifier hardening

verify-static-binary.sh inspects via readelf (falling back to greadelf or llvm-readelf) and treats the tool's exit status as untrustworthy (readelf exits 0 on a truncated ELF while reporting the problem on stderr). For each input it requires:

• an executable ELF: ET_EXEC, or ET_DYN carrying DF_1_PIE (a static-PIE executable, not a shared library);
• a fully readable ELF header, program header table, and dynamic section — the inspector must exit 0 and emit no diagnostics;
• at least one PT_LOAD segment, with every PT_LOAD contained within the file (p_offset + p_filesz <= file size), which catches truncation that readelf does not flag when section headers are absent;
• the dynamic table's file offset to agree with the PT_DYNAMIC segment, so a SHT_DYNAMIC section pointed away from PT_DYNAMIC cannot hide DT_NEEDED entries;
• no PT_INTERP and no DT_NEEDED (the actual static-linkage properties).

Leading-dash paths use a ./ prefix (llvm-readelf rejects --). With no inspector present, Linux — and any host whose OS can't be determined — fails closed; a positively non-Linux host (e.g. macOS, which ships no binutils) skips with a warning, since CI enforces linkage on Linux.

Review feedback addressed

From the PR review discussion, all squashed into the single commit:

• Verifier accepted a truncated ELF — now rejects any input where readelf emits diagnostics or a PT_LOAD segment extends past EOF (validated against a static binary whose section headers were zeroed and body truncated: previously "statically linked" exit 0, now rejected).
• Shared object accepted as static — an ET_DYN object without DF_1_PIE is now rejected.
• Dash-prefixed paths — fixed with -- separators.
• CI reachability/coverage — added the dedicated validation workflow (amd64 + arm64) and broadened its triggers to manifests/crates/**/toolchain files plus a nightly schedule; added a concurrency group.
• Doc wording — architecture/build.md now says the variant "does not change the runtime layout or the supervisor image base".
• macOS/BSD portability — the PT_LOAD bounds check uses a portable wc -c (not GNU-only stat -c) and mktemp uses an explicit template. macOS ships no readelf, so the verifier now inspects via readelf/greadelf/llvm-readelf (whichever is present); a positively-non-Linux host with none skips with a documented warning, while Linux and any undeterminable host fail closed. Keeps the documented macOS musl staging path (cargo-zigbuild) working, with CI as the authoritative gate.
• Cross-tool hardening (from an adversarial break-it pass across GNU readelf and llvm-readelf): reject an ELF whose SHT_DYNAMIC section offset disagrees with PT_DYNAMIC — a genuinely dynamic binary that GNU readelf otherwise reports as static; use a ./-prefix for leading-dash paths since llvm-readelf rejects --.

Testing

• mise run lint passes (license/SPDX, rustfmt, clippy, python, helm, markdown); actionlint clean on the new/changed workflows; shellcheck clean on both scripts
• Adversarial verifier battery (built/crafted inputs, run through the script): rejects dynamic, truncated (including section-header-zeroed + PT_LOAD-past-EOF), SHT_DYNAMIC/PT_DYNAMIC desync, non-ELF, relocatable .o, and shared-object inputs; accepts classic-static, static-PIE, and the real musl and glibc-static supervisors — no false negatives or false positives
• Cross-tool validation: the full battery passes forced through GNU readelf, greadelf, and llvm-readelf; fail-closed when no inspector is present or the OS is undeterminable, skip (with warning) only on a positively non-Linux host
• Built both variants natively — SUPERVISOR_LIBC=glibc-static (static-PIE) and default musl (classic static) — each passes verify-static-binary.sh and an independent readelf/ldd check
• mise run test (full Rust + Python unit suite) passes with 0 failures
• Unit tests added/updated — n/a (build-tooling change; validated by the verifier battery, real builds, and the new CI validation workflow)
• E2E tests added/updated — not run (no runtime code changed; requires a running gateway)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (architecture/build.md)

---

Original commit authored by Mrunal Patel mrunalp@gmail.com; carried and submitted here.

Co-authored-by: Emilien Macchi emacchi@redhat.com

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

Thank you for your interest in contributing to OpenShell, @EmilienM.

This project uses a vouch system for first-time contributors. Before submitting a pull request, you need to be vouched by a maintainer.

To get vouched:

1. Open a Vouch Request discussion.
2. Describe what you want to change and why.
3. Write in your own words — do not have an AI generate the request.
4. A maintainer will comment /vouch if approved.
5. Once vouched, open a new PR (preferred) or reopen this one after a few minutes.

See CONTRIBUTING.md for details.

[mrunalp]

Review findings:

1. [P2] Static verifier accepts invalid files (tasks/scripts/verify-static-binary.sh:52-66)

   Both readelf failures are suppressed. I reproduced an ASCII file being reported as “statically linked” with exit status 0 because both counts become zero. Since this script is the local/CI safety gate, it should require a successful ELF-header inspection before checking PT_INTERP and DT_NEEDED (and ideally verify that the input is an executable ELF).

2. [P2] The glibc-static CI path is unreachable and untested (.github/workflows/rust-native-build.yml:28-32)

   This workflow is workflow_call-only, and its sole caller in .github/workflows/docker-build.yml does not expose or forward supervisor-libc. Every current CI invocation therefore uses the default musl; neither the new GNU build branch nor arm64 static-glibc support is exercised. Please thread the input through docker-build.yml or add a dedicated validation job for the optional variant.

Minor documentation note: architecture/build.md says changing libc “does not change runtime behavior,” although changing resolver behavior is the motivation for this work. “Does not change runtime layout or the supervisor image base” would be more accurate.

I confirmed that the verifier rejects a dynamic binary and accepts both an existing static-musl binary and a static-PIE glibc binary. The non-ELF false positive is reproducible.

[mrunalp]

Re-review result: the previous feedback is only partially addressed, so I would still request changes.

1. [P2] The verifier still accepts a truncated ELF (tasks/scripts/verify-static-binary.sh:52-68)

   The new guard correctly rejects plain text, but it does not establish that the complete ELF is readable. I tested the latest script with only the first 64 bytes of /bin/ls: the ELF header contains Type: DYN, the later readelf diagnostics are suppressed, both linkage counts become zero, and the script prints statically linked: no PT_INTERP, no DT_NEEDED with exit status 0. Please require readable program headers (for example, capture them and require at least one LOAD segment) and reject readelf diagnostics instead of treating inspection failures as zero matches.

2. [P2] The validation workflow has insufficient trigger coverage (.github/workflows/supervisor-static-validate.yml:14-26)

   The dedicated amd64/arm64 job addresses the original CI reachability concern, but its path filters omit Cargo.toml, Cargo.lock, and supervisor/dependency sources. Those changes can alter native dependencies or final linkage without running the glibc-static validation, allowing the optional variant to regress unnoticed. Please broaden the filters or use an unfiltered/periodic validation strategy.

The documentation correction is addressed. The dedicated matrix workflow is also structurally correct, but its current run is action_required with no jobs executed, so neither architecture has been proven in CI yet: https://github.com/NVIDIA/OpenShell/actions/runs/31412338352

[mrunalp]

Re-review update: the previously reported verifier-hardening, CI-trigger coverage, and documentation issues are now addressed.

One remaining issue:

[P2] Preserve the supported macOS staging path (tasks/scripts/verify-static-binary.sh:111)

The verifier now uses stat -c '%s' -- "$binary" to obtain the file size. -c is a GNU stat option and is unavailable with macOS/BSD stat. stage-prebuilt-binaries.sh explicitly supports building the default musl supervisor from macOS via cargo-zigbuild, and it invokes this verifier after the build, so a valid macOS staging run will abort at this line under set -e.

Please use a portable byte count such as:

file_size=$(wc -c <"$binary")

or an OS-aware GNU/BSD stat fallback.

I retested the latest verifier: non-ELF, truncated ELF, and dynamic inputs are rejected, while the existing classic-static musl and static-PIE glibc supervisors pass. Once the macOS portability issue is addressed, we can trigger the pending CI validation.

[mrunalp]

/ok to test 6368c37

[mrunalp]

/ok to test fc7b262

[github-actions]

Label test:e2e applied for fc7b262. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.