test: cross-repo fresh-shell last-mile E2E harness (install -> CLI -> cluster info -> push)

[LukasWodka]

Part of #737

Summary

Adds the two-leg install-journey harness that closes the gap which let a PATH-persistence regression ship green: no existing test opens a fresh shell after install and asserts the documented next command works. distro-prereqs and e2e-cluster both assert in the same shell that ran the installer, so a binary that lands somewhere a new terminal can't see is invisible to them. This harness reproduces "customer opens a new terminal and types the next command."

Related

• Part of tracebloc/backend#737 (keystone last-mile E2E)
• Guards the PATH-persistence class fixed on cli's fix/install-path-persist branch
• Leg 2 reproduces the namespace/context incident addressed by client#208 (installer sets the kube context — already merged)
• A thin cli-side caller (separate PR) points Leg 1 at a cli PR's install.sh for pre-merge cross-repo coverage

What's in here

Leg 1 — scripts/tests/path-persist.sh (cheap, wide, no cluster, no creds)
Runs in a plain distro container. Installs the tracebloc CLI via cli's install.sh (ref configurable via TRACEBLOC_CLI_REF — URL or local path), then for each shell present among bash / zsh / fish spawns a fresh login AND non-login shell and asserts command -v tracebloc resolves and tracebloc version runs. A fresh non-login bash reads ~/.bashrc, not~/.profile / ~/.bash_profile — so this is the cell that goes red on the pre-fix installer and green on the fixed one. Prints a PASS/FAIL line per shell x mode cell.

Leg 2 — scripts/tests/e2e-journey.sh (full journey on a real cluster, amd64, no secrets)
Extends the e2e-cluster.sh pattern (sources common.sh / setup-linux.sh / cluster.sh, isolated CLUSTER_NAME, TRACEBLOC_NO_AUTOSTART=1):

1. create_cluster() + wait nodes Ready.
2. Install the CLI via install.sh.
3. Apply a credential-free stub matching the CLI's real discovery contract (internal/cluster/discover.go): a *-jobs-manager Deployment carrying app.kubernetes.io/name=client + app.kubernetes.io/managed-by=Helm + instance/version/chart labels (image registry.k8s.io/pause:3.9), plus an ingestor ServiceAccount (so the cluster info TokenRequest path doesn't exit 5). Point the kubeconfig context's namespace at it and assert tracebloc cluster info(a) succeeds and (b) succeeds from a fresh login + non-login shell.
4. tracebloc dataset push --dry-run smoke on a tiny sample CSV (offline-validatable; no creds).
5. Teardown via EXIT trap.

Every long step runs under a watchdog timeout so a hang FAILS (exit 124 -> hard error) instead of spinning to the GitHub ceiling.

Note on the stub labels: the issue's shorthand says app: manager; the CLI actually discovers on app.kubernetes.io/name=client,app.kubernetes.io/managed-by=Helm + a *-jobs-manager Deployment name. The manifest uses the real selector so the assertion exercises the actual code path, and keeps app: manager as a cosmetic extra label. Documented inline.

Note on context-on-default: client#208 (installer sets the kube context) is merged, so the supported state is context-namespace == workspace namespace — that's the core assertion. The opposite case (context left on default, CLI auto-discovers across namespaces) needs a CLI change that is not merged yet, so it runs as a non-fatal pending probe, flippable to a hard assertion via TB_EXPECT_NS_AUTODISCOVER=1 once that lands.

CI (.github/workflows/installer-tests.yaml)

• path-persist — distro matrix (ubuntu:22.04 / 24.04, debian:12, fedora:latest, almalinux:9, opensuse/leap:15.6, alpine:3), fail-fast: false, one fresh container per distro; the script iterates shell x mode inside. Runs on the same scripts/** paths as the rest of the file.
• e2e-journey — amd64, gated to nightly schedule + workflow_dispatch + the e2e PR label (mirrors cli's e2e.yml label gating), to control cluster cost.
• Both new scripts added to the static job's shellcheck list (error + advisory passes).

Type of change

• Tech-debt / refactor (test infrastructure)
• Feature
• Bug fix
• Docs
• Security / hardening
• Breaking change

Test plan — Verified locally / Needs CI

Verified locally (green):

• bash -n on path-persist.sh and e2e-journey.sh — both parse.
• shellcheck --severity=error (the CI gate) on both new scripts — no findings; --severity=warning advisory pass — also clean.
• actionlint on installer-tests.yaml — clean (also runs shellcheck on the inline run blocks).
• yaml.safe_load on the workflow — parses.
• Isolated sanity-check of the fresh-shell assertion mechanism (capture command -v output, check non-empty + version exit code): positive case resolves a real on-PATH binary, negative control (PATH stripped) correctly yields empty -> the guard would fail. Confirms the assert logic.

Needs CI (requires GitHub runners / Docker — NOT run locally, not claimed to pass):

• The full path-persist distro x shell x mode matrix (needs per-distro containers + zsh/fish install).
• e2e-journey end to end (needs a real k3d cluster on a Linux runner with Docker, plus a reachable cli install.sh).
• The "red on the pre-fix installer / green on the fixed one" proof — needs the matrix to actually run against both installer refs.

The default TRACEBLOC_CLI_REF currently points at cli's fix/install-path-persist raw install.sh (with an inline TODO(cli#61) to switch to releases/latest/download/install.sh once the fix ships in a public release — otherwise the guard would test the old installer from the latest release and report a false red).

Checklist

• Tests added (this PR is the tests)
• No secrets / credentials in the diff (stub is credential-free)
• Customer identifiers scrubbed — generic phrasing only in added lines
• Docs updated if behavior or config changed (N/A — test-only)

[LukasWodka]

👋 Heads-up — Code review queue is at 17 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#102 — chore(kanban): declare staging as prod-truth branch via .kanban.yml · author: @LukasWodka · no reviewer assigned
• averaging-service#103 — fix(averaging): correctness foundations — DP, shape guard, sklearn weighting, bounded ensembles (chore: add default CODEOWNERS for auto-reviewer assignment #73) · author: @LukasWodka · reviewer: @shujaatTracebloc
• averaging-service#98 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• backend#732 — feat(experiments): add per-experiment aggregation_strategy field (fix(installer): the stdin write must not throw when the child closed the pipe #730) · author: @LukasWodka · reviewer: @saqlainsyed007, @shujaatTracebloc
• backend#735 — fix(flops): atomic flops_used + fire stop once on threshold crossing (feat(chart): stop injecting the fixed training-job envelope; let jobs-manager size it (backend#664) #733) · author: @saadqbal · no reviewer assigned
• cli#61 — fix(install.sh): persist PATH to the shell rc (parity with install.ps1) · author: @LukasWodka · reviewer: @saadqbal
• client#207 — test(charts): add helm-unittest suite for ingestion-authz-configmap · author: @saadqbal · no reviewer assigned
• client#211 — fix(installer): bound the apt dpkg-lock wait so a held lock can't hang the install (Installer hangs at 'Installing conntrack…' on Ubuntu when the apt/dpkg lock is held (no lock timeout) #210) · author: @saadqbal · no reviewer assigned
• data-ingestors#146 — test(schema): lock category enum to the engine's TaskCategory (anti-drift) · author: @LukasWodka · reviewer: @saadqbal
• data-ingestors#148 — Develop to prod · author: @divyasinghds · reviewer: @saadqbal

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[aptracebloc]

cc @LukasWodka@saadqbal — post-merge review of this PR. The design and documentation are genuinely excellent (cost-tiered legs, real discovery contract in the stub, honest test plan), but there are two significant issues that should get a follow-up PR, one of which defeats Leg 2 entirely.

🔴 Critical: guard() swallows every failure — e2e-journey.sh can never go red

if! timeout --kill-after=15s "$secs""$@";thenlocal rc=$?

In bash, $? inside the then branch of if ! cmd is the exit status of the negated condition — always 0 when the branch is entered (verified empirically). So rc captures 0, the rc -eq 124 hang check never fires, and guardreturns 0 on failure — which also means the script's set -e never triggers. Since every assertion runs under guard (including tracebloc cluster info and the --dry-run smoke), the job prints E2E JOURNEY PASS even if every step fails. The watchdog also can't do the one thing it promises: turn hangs into red failures.

Fix:

local rc=0
timeout --kill-after=15s "$secs""$@"|| rc=$?
[[ $rc-eq 124 ]] && error "step '${label}' exceeded ${secs}s — treating the hang as a failure."return$rc

Note bash -n and shellcheck both pass on the buggy version — worth adding a negative-control run (point the journey at a command that exits non-zero, confirm the job fails) as part of validating the fix.

🟠 High: the fresh-shell mechanism in path-persist.sh likely produces no signal (and false negatives)

Two compounding issues:

1. Non-interactive shells don't read rc files. "A fresh non-login bash reads ~/.bashrc" is only true for interactive non-login shells. bash -c '...' is non-interactive and reads no rc files; zsh -c reads only .zshenv; zsh -lc reads .zprofile/.zlogin but not.zshrc. So the cells don't reproduce "customer opens a new terminal" (an interactive shell), and a correctly fixed installer that persists PATH to ~/.bashrc/~/.zshrc would still fail the non-login cells whenever the binary lands outside the default PATH — a false red. The interactive cells should use bash -ic / zsh -ic (fish is fine as commented).
2. Everything runs as root, so install.sh installs to /usr/local/bin — already on every default PATH. The rc-file persistence path (the ~/.local/bin fallback where the cli#61 class actually manifests) is never exercised, so all cells pass trivially on both the pre-fix and fixed installer. To get the promised "red on pre-fix / green on fixed" signal, at least part of the matrix should install as a non-root user (or force the installer's prefix to a non-default-PATH directory).

The PR honestly flags that the red/green proof hasn't run — that proof should gate the follow-up, because as written the matrix probably can't distinguish the two installers.

🟡 Medium

• No timeout-minutes on either new job.static sets one; path-persist (7 containers doing package installs) and e2e-journey have nothing between them and the 6-hour ceiling — doubly so while guard() is broken. install_kubectl / install_k3d / install_helm in Leg 2 also run unguarded.
• Label gating won't trigger on labeling. The workflow's pull_request trigger uses default types (opened, synchronize, reopened), so adding the e2e label won't start a run until the next push. Add types: [opened, synchronize, reopened, labeled] if label-to-run is the intended flow.

⚪ Low / docs drift

• The PR body says the default TRACEBLOC_CLI_REF points at cli's fix/install-path-persist branch with a TODO(cli#61); the merged scripts default to releases/latest/download/install.sh (fix shipped in v0.3.1). Body is stale relative to the code.
• "Nightly schedule" is inaccurate twice over: the workflow's schedule is weekly (Mondays 03:00 UTC), and if: github.event_name != 'pull_request' also runs e2e-journey on every push to main/develop/openshift touching scripts/** — broader than the described nightly + dispatch + label gating.

What's good

• Clear cost-tiered design; fail-fast: false matrix; isolated CLUSTER_NAME + TRACEBLOC_NO_AUTOSTART + EXIT-trap teardown mirroring e2e-cluster.sh.
• The stub manifest targets the CLI's real discovery contract (labels + *-jobs-manager name + ingestor SA) rather than the issue's app: manager shorthand, with the discrepancy documented inline.
• The non-fatal pending probe for unmerged CLI auto-discovery (TB_EXPECT_NS_AUTODISCOVER=1) is a nice pattern — avoids both false reds and silent gaps.
• No secrets, read-only mounts, permissions: contents: read.

Since this is already on develop: the guard() fix is one line and I'd prioritize it — until it lands, e2e-journey green runs assert nothing.

🤖 Generated with Claude Code