Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 51 additions & 6 deletions .github/workflows/rust-native-build.yml
Original file line numberDiff line numberDiff line change
Expand Up@@ -5,10 +5,14 @@ name: Rust Image Binary Build (openshell-gateway / openshell-sandbox / openshell

# Build Rust binaries per Linux architecture before the Docker image build
# consumes them as prebuilt artifacts. Gateway images use GNU-linked binaries
# for the NVIDIA distroless C/C++ runtime; supervisor and cli images use musl/static
# for the NVIDIA distroless C/C++ runtime; supervisor and cli images use static
# binaries so the final image can remain scratch. Gateway GNU binaries are
# built with an explicit glibc 2.28 floor so image, package, and tarball
# artifacts share the same host portability contract.
#
# The supervisor libc is selectable via the `supervisor-libc` input (musl or
# glibc-static). Both variants are fully static and are verified as such,
# because the supervisor is executed from inside arbitrary sandbox images.

on:
workflow_call:
Expand All@@ -21,6 +25,11 @@ on:
description: "Linux architecture to build (amd64 or arm64)"
required: true
type: string
supervisor-libc:
description: "libc variant for the sandbox component (musl or glibc-static)"
required: false
type: string
default: "musl"
cargo-version:
description: "Pre-computed cargo version (skips internal git-based computation)"
required: false
Expand DownExpand Up@@ -76,10 +85,12 @@ jobs:
COMPONENT: ${{ inputs.component }}
ARCH: ${{ inputs.arch }}
FEATURES: ${{ inputs.features }}
SUPERVISOR_LIBC: ${{ inputs['supervisor-libc'] }}
# Partition the GHA sccache cache per (component, arch). Without this,
# concurrent jobs collide on the same cache key and later-starting
# writers hit 409 Conflict.
SCCACHE_GHA_VERSION: ${{ inputs.component }}-${{ inputs.arch }}
# writers hit 409 Conflict. The sandbox component also partitions per
# libc variant so musl and glibc-static builds do not evict each other.
SCCACHE_GHA_VERSION: ${{ inputs.component }}-${{ inputs.arch }}${{ inputs.component == 'sandbox' && format('-{0}', inputs['supervisor-libc']) || '' }}
container:
image: ghcr.io/nvidia/openshell/ci:latest
credentials:
Expand DownExpand Up@@ -132,9 +143,28 @@ jobs:
;;
esac

# The sandbox binary must stay fully static. musl gets there via the
# musl target; glibc-static uses the GNU target with +crt-static and
# relies on this job running natively on the target architecture,
# because zig cannot statically link glibc.
static_libc=musl
if [[ "$COMPONENT" == "sandbox" ]]; then
case "$SUPERVISOR_LIBC" in
musl) static_libc=musl ;;
glibc-static) static_libc=gnu ;;
*)
echo "unsupported supervisor-libc: $SUPERVISOR_LIBC (expected musl or glibc-static)" >&2
exit 1
;;
esac
fi

case "$ARCH" in
amd64)
if [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
if [[ "$COMPONENT" == "sandbox" && "$static_libc" == "gnu" ]]; then
target=x86_64-unknown-linux-gnu
zig_target=
elif [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
target=x86_64-unknown-linux-musl
zig_target=x86_64-linux-musl
else
Expand All@@ -143,7 +173,10 @@ jobs:
fi
;;
arm64)
if [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
if [[ "$COMPONENT" == "sandbox" && "$static_libc" == "gnu" ]]; then
target=aarch64-unknown-linux-gnu
zig_target=
elif [[ "$COMPONENT" == "sandbox" || "$COMPONENT" == "cli" ]]; then
target=aarch64-unknown-linux-musl
zig_target=aarch64-linux-musl
else
Expand All@@ -167,7 +200,7 @@ jobs:
- name: Cache Rust target and registry
uses: Swatinem/rust-cache@c19371144df3bb44fab255c43d04cbc2ab54d1c4 # v2
with:
shared-key: rust-native-${{ inputs.component }}-${{ inputs.arch }}-zig-wrapper-${{ hashFiles('tasks/scripts/setup-zig-cc-wrapper.sh') }}
shared-key: rust-native-${{ inputs.component }}-${{ inputs.arch }}${{ inputs.component == 'sandbox' && format('-{0}', inputs['supervisor-libc']) || '' }}-zig-wrapper-${{ hashFiles('tasks/scripts/setup-zig-cc-wrapper.sh') }}
cache-directories: .cache/sccache
cache-targets: "true"

Expand DownExpand Up@@ -239,6 +272,11 @@ jobs:
cargo_cmd=(cargo zigbuild)
build_target="${{ steps.target.outputs.zig_target }}"
args+=(--features bundled-z3)
elif [[ "${{ inputs.component }}" == "sandbox" && "$SUPERVISOR_LIBC" == "glibc-static" ]]; then
# Static glibc requires the native toolchain's libc.a (build-essential
# in the CI image); cargo-zigbuild is not usable here because zig
# accepts -static for *-linux-gnu and links dynamically anyway.
export RUSTFLAGS="${RUSTFLAGS:-} -C target-feature=+crt-static"
fi
args+=(
--release
Expand DownExpand Up@@ -276,6 +314,13 @@ jobs:
BIN="target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}"
tasks/scripts/verify-glibc-symbols.sh 2.28 "$BIN"

- name: Verify static linkage
if: inputs.component == 'sandbox'
run: |
set -euo pipefail
BIN="target/${{ steps.target.outputs.target }}/release/${{ steps.target.outputs.binary }}"
tasks/scripts/verify-static-binary.sh "$BIN"

- name: Stage binary for prebuilt layout
run: |
set -euo pipefail
Expand Down
70 changes: 70 additions & 0 deletions .github/workflows/supervisor-static-validate.yml
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

name: Supervisor Static Linkage Validation

# The glibc-static supervisor variant (SUPERVISOR_LIBC=glibc-static) has no
# other CI caller: docker-build.yml builds the default musl variant, so the
# GNU + crt-static build branch and its native-only, per-arch requirements are
# never exercised by image or release CI. Build the variant here on both
# architectures so it cannot regress unnoticed. rust-native-build.yml runs
# verify-static-binary.sh for the sandbox component, which fails the job on any
# dynamic linkage.
#
# Linkage can change from a new/updated dependency or a source change, not just
# from the build scripts, so the push path filters cover the workspace manifests
# and crate sources in addition to the build tooling. A nightly schedule is the
# unfiltered backstop for anything the filters miss.
#
# rust-native-build.yml runs on NVIDIA self-hosted runners, which reject jobs
# triggered by `pull_request`. This workflow therefore follows the repo's
# self-hosted convention (see branch-checks.yml / branch-e2e.yml): validate in
# the merge queue (pre-merge), on push to main (post-merge), nightly, and on
# demand — never on `pull_request`.

on:
merge_group:
types: [checks_requested]
push:
branches: [main]
paths:
- "Cargo.toml"
- "Cargo.lock"
- "crates/**"
- "rust-toolchain.toml"
- "mise.toml"
- "mise.lock"
- ".cargo/config.toml"
- "tasks/scripts/stage-prebuilt-binaries.sh"
- "tasks/scripts/verify-static-binary.sh"
- ".github/workflows/rust-native-build.yml"
- ".github/workflows/supervisor-static-validate.yml"
schedule:
# Nightly (04:17 UTC) unfiltered run so a linkage regression cannot slip
# through the path filters unnoticed. Schedules run only on the default branch.
- cron: "17 4 * * *"
workflow_dispatch:

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false

permissions:
contents: read
packages: read

jobs:
glibc-static:
name: glibc-static supervisor (${{ matrix.arch }})
strategy:
fail-fast: false
matrix:
arch: [amd64, arm64]
uses: ./.github/workflows/rust-native-build.yml
with:
component: sandbox
arch: ${{ matrix.arch }}
supervisor-libc: glibc-static
artifact-name: supervisor-glibc-static-${{ matrix.arch }}
retention-days: 1
secrets: inherit
47 changes: 39 additions & 8 deletions architecture/build.md
Original file line numberDiff line numberDiff line change
Expand Up@@ -68,6 +68,31 @@ The gateway bundles z3 into the release binary so Linux packages, standalone
tarballs, and gateway images do not depend on distro-specific z3 shared-library
SONAMEs.

The supervisor is the one binary whose libc is selectable, because it is the one
binary executed inside a userland OpenShell does not control. `SUPERVISOR_LIBC`
chooses between `musl` (default) and `glibc-static`. Both produce a fully static
binary; the choice does not change the runtime layout or the supervisor image base.
Static linkage is a hard requirement rather than a preference, so both variants
are verified by `tasks/scripts/verify-static-binary.sh`, which fails the build on
any `PT_INTERP` or `DT_NEEDED` entry.

The two variants differ only in build-time constraints:

| | `musl` (default) | `glibc-static` |
|---|---|---|
| Cross-compiles | yes, via `cargo zigbuild` | no — must build natively per architecture |
| Host requirement | zig + cargo-zigbuild | glibc static libraries (`glibc-static` on Fedora/RHEL, `libc6-dev` on Debian/Ubuntu) |
| libc license | MIT | LGPL-2.1-or-later, statically linked |

`cargo zigbuild` cannot produce the `glibc-static` variant: `zig cc` accepts
`-static` for `*-linux-gnu` targets and emits a dynamically linked binary
anyway. The staging script therefore refuses to cross-compile that variant
instead of silently degrading linkage.

Selecting `glibc-static` statically links LGPL glibc into a redistributed
binary, which carries relinking obligations that musl (MIT) does not. Treat the
default as the shipping configuration unless that has been reviewed.

## Container Builds

The Docker image pipeline is a two-step flow: build the Rust binary natively
Expand All@@ -91,9 +116,11 @@ package-managed VM support does not raise the package runtime requirement.
Gateway staging and release workflows set up the Zig C/C++ wrapper before
bundled Z3 builds and verify the maximum referenced `GLIBC_*` symbol version
before publishing or copying artifacts.
Supervisor binaries remain static musl and use `cargo zigbuild` when available,
including native CPU architectures, so C dependencies are compiled for the musl
target instead of the host GNU libc target. Local Docker image tasks infer the
Supervisor binaries are static in every configuration. The default `musl`
variant uses `cargo zigbuild` when available, including native CPU
architectures, so C dependencies are compiled for the musl target instead of the
host GNU libc target. The `glibc-static` variant uses plain `cargo build` with
`+crt-static` and requires a native per-architecture build. Local Docker image tasks infer the
target architecture from `DOCKER_PLATFORM` when set. Otherwise, they require
valid container engine host metadata and fail when the engine query is
unavailable or reports an unsupported architecture, avoiding host-kernel
Expand All@@ -114,11 +141,15 @@ Runtime layout:
as a release artifact. Linux GNU VM driver binaries must not reference
`GLIBC_*` symbols newer than `GLIBC_2.28`; release workflows verify this
before publishing artifacts.
- **Supervisor**: Alpine base with `nftables`, static musl binary at
`/openshell-sandbox`. Static linkage keeps the binary usable when the image
is mounted/extracted into sandbox environments (Docker extraction, Podman
image volumes, Kubernetes init-container copy-self), while `nftables` supports
Kubernetes supervisor sidecar egress enforcement.
- **Supervisor**: Alpine base with `nftables`, static binary at
`/openshell-sandbox` (musl by default; see `SUPERVISOR_LIBC` above). Static
linkage keeps the binary usable when the image is mounted/extracted into
sandbox environments (Docker extraction, Podman image volumes, Kubernetes
init-container copy-self), whose libc and glibc version are not known at build
time, while `nftables` supports Kubernetes supervisor sidecar egress
enforcement. The VM driver bundles its own supervisor build
(`tasks/scripts/vm/build-supervisor-bundle.sh`) and does not read
`SUPERVISOR_LIBC`.

Gateway image builds bake the corresponding supervisor image tag into the
gateway binary so Docker sandboxes do not depend on `:latest` by default.
Expand Down
9 changes: 6 additions & 3 deletions deploy/docker/Dockerfile.supervisor
Original file line numberDiff line numberDiff line change
Expand Up@@ -15,9 +15,12 @@
#
# Use tasks/scripts/docker-build-image.sh supervisor (or `mise run build:docker:supervisor`)
# to stage the binary and build the image in one step. CI builds the binary
# per-architecture via the `rust-native-build.yml` workflow (with the musl
# target) and uploads it as an artifact, which is downloaded into the same
# staging directory before the image build job runs.
# per-architecture via the `rust-native-build.yml` workflow and uploads it as an
# artifact, which is downloaded into the same staging directory before the image
# build job runs.
#
# The binary is static under either supported libc variant (`SUPERVISOR_LIBC`:
# musl by default, or glibc-static), so this Alpine base runs it unchanged.

FROM alpine:3.22 AS supervisor

Expand Down
Loading
Loading