feat(seal-check): enumeration label contract + storage-assertions test hook (RFC-0003 §8.2–8.4)

[LukasWodka]

Summary

Chart half of the RFC-0003 D12 seal check (§8.2–8.4): establishes the seal-check enumeration contract and fills the storage-assertions gap, without touching scripts/ (installer work stays with #368).

1. The enumeration contract (consumed by tracebloc/cli#393)

Every runnable conformance check is a helm.sh/hook: testJob carrying, on the Job and its pod template:

Label	Value
tracebloc.io/seal-check	"true"
tracebloc.io/seal-check-name	egress-enforcement | backend-reachability | storage-assertions

Enumerate without running: helm get hooks <release>. While running: kubectl get jobs,pods -l tracebloc.io/seal-check=true. Label keys + existing names are public API (never renamed; new checks = new names). Auxiliary hook resources (the new SA/RBAC) deliberately do not carry the labels — only runnable checks are enumerable. Applied to the existing egress-enforcement probe + backend-reachability check via a new tracebloc.sealCheckLabels helper.

2. New check: storage-assertions (sealCheck.storageAssertions.*, default on)

kubectl-image test-hook Job (same alpine/k8s:1.30.5 the image-refresh CronJob uses) asserting from in-cluster:

• pvc-bound — client-pvc / client-logs-pvc / mysql-pvc exist and are Bound (bounded wait, default 120s, for WaitForFirstConsumer + first-install pulls)
• pvc-storageclass — every release PVC is on the expected class (tracebloc.storageClassName)
• pv-hostpath (dynamic mode, hostPath.enabled=false, only) — no release PVC backed by an unmanaged hostPath PV. Catches the RFC-0003 D3/D4 leftover-PV claimRef capture (a stale <release>-{data,logs,mysql}-pv grabs the claim even in dynamic mode). k3s local-path node-local paths (nodeLocalPathPrefixes, default /var/lib/rancher/ + /opt/local-path-provisioner) are tolerated and reported — they live inside the cluster node and die with it (Option C); host-visibility of those paths is a cluster-creation fact verified at install level. In hostPath mode the sub-check SKIPs (declared model).

RBAC: dedicated least-privilege SA as negative-weight test hooks — PVC get/list in the namespace; PV get/list only when clusterScope allows a ClusterRole. With clusterScope=false the PV scan degrades to the name/claimRef check and prints an explicit WARNING naming what was not verified.

UNSEALED philosophy (documented in docs/SEAL-CHECK.md): a guarantee that cannot be verified fails loudly — never silently claimed sealed. Disabling a check is a values-visible declaration, not a runtime fallback.

3. Docs

docs/SEAL-CHECK.md — contract, coverage today, philosophy, per-substrate coverage table (chart-side input to the RFC §8.3 matrix), and the k3d/k3s NetworkPolicy egress verification runbook (§8.4) using the existing enforcement probe — documented honestly as not yet executed; the k3d cell stays "verification run pending" until a run is recorded. Cross-linked from docs/SECURITY.md §7.

Compatibility

• All new values fully nil-guarded (default dict / dig) — helm upgrade --reuse-values from releases predating sealCheck renders identically (unit-tested with sealCheck: null).
• Test hooks never run at install/upgrade — cannot block them or the auto-upgrade.
• Forward-compatible with the Option C storage flip (feat(installer): node-local storage prototype (RFC-0003 Option C, flag-gated) #368): gates on hostPath.enabled, verifies whichever model the install declares.

Test plan

• helm lint --strict PASS on all 4 platform values (aks / bm / eks / oc)
• helm template clean: 4 CI platforms + near-default values + node-local values (hostPath.enabled=false, local-path class) + flag off + nil sealCheck / nil storageAssertions
• helm unittest: 287/287 (20 new cases: rendering gates, hook annotations, label contract on Job+pod, RBAC split under clusterScope, env baking, PSA-restricted fields, image pin, nil-block survival)
• Embedded script: sh -n + shellcheck clean; exercised with a mocked kubectl across 9 scenarios (happy dynamic / node-local PV / unmanaged hostPath / legacy-PV capture / wrong class / PV read error / namespace-scoped partial / hostPath mode / never-Bound) — verdicts as designed
• scripts/tests/check-drift.sh: no drift (run only, not modified)

Deliberately left for follow-ups (tracked in backend#1184)

• The live k3d verification run (§8.4) — runbook shipped, execution + recorded result pending
• CLI surfacing of the aggregated sealed/unsealed verdict (status: surface environment conformance (seal check) result cli#393)
• ~/.tracebloc host-tree check — not observable in-cluster; CLI/installer offboard-verify lineage (cli#389)
• e2e-harness wiring; precise RFC §8.3 matrix fill in the RFC itself
• Pre-existing (untouched): helm template --set hostPath=null crashes on develop too (shared-images-pvc.yaml unguarded deref) — orthogonal to this PR

Part of tracebloc/backend#1184

🤖 Generated with Claude Code

---

Note

Medium Risk
New default-on helm test hook talks to the API and may create cluster-scoped PV read RBAC when clusterScope is true; label keys/names are a stable public contract, but hooks still never run on install/upgrade.

Overview
Introduces the seal-check conformance suite for RFC-0003: runnable checks stay helm.sh/hook: test Jobs labeled tracebloc.io/seal-check=true and tracebloc.io/seal-check-name=<check> on the Job and pod (public API for CLI enumeration via tracebloc.sealCheckLabels). egress-enforcement and backend-reachability adopt those labels; auxiliary hook RBAC is excluded.

Adds default-on storage-assertions: a kubectl-based test Job (plus least-privilege SA/RBAC test hooks) that waits for release PVCs to bind, asserts the expected StorageClass, and in dynamic-PVC mode rejects unmanaged hostPath backing (legacy chart PV names, optional full PV scan when clusterScope allows). Node-local k3s paths use segment-safe prefix allowlisting; unverifiable guarantees fail the Job (UNSEALED).

Documents the contract and runbooks in docs/SEAL-CHECK.md, wires sealCheck.storageAssertions in values/schema, extends helm unittests, and links from docs/SECURITY.md §7.

^{Reviewed by Cursor Bugbot for commit 232d10e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8c37038. Configure here.}

[shujaatTracebloc]

Review — seal-check suite (RFC-0003 §8.2–8.4)

Read the real 11-file diff and the embedded kubectl script line by line, and verified the referenced helpers + schema. All CI green (helm lint / unittest 287, 4-platform renders, Bugbot pass). Net: high-quality, security-sound — good to merge after a rebase on develop; the two edges below are optional, none blocking.

What's strong

• Fail-loud philosophy is actually implemented, not just documented. Unverifiable ⇒ exit 1 / UNSEALED at every branch: PV read error, still-Pending PVC, missing volumeName. Disabling is a values-visible declaration (enabled: false, clusterScope: false), never a silent runtime fallback.
• Least-privilege RBAC with honest degradation. Namespaced Role = PVC get/list; the cluster-scoped PV read is a separate ClusterRole gated on clusterScope (same condition as the chart's existing RBAC), and when it's off the check names exactly what it could not verify rather than passing quietly.
• Both prior Bugbot fixes landed correctly. Segment-bounded prefix match (case "$hp" in "$p"|"$p"/*)) genuinely stops /opt/local-path-provisioner from admitting …-evil, and the WARNING is correctly suppressed after a legacy-PV FAIL (that "verified not-legacy" claim was never made).
• Label contract is consistent — tracebloc.io/seal-check + -name on the Job and its pod template for all three checks (verified egress-enforcement, backend-reachability, storage-assertions); aux SA/RBAC deliberately unlabeled. Plus thorough nil-guarding (default dict/dig), test-hook isolation (never runs at install/upgrade), and the 9 mocked-kubectl scenarios.

Minor edges (optional)

1. Branch is 9 commits behind develop (2 ahead) — worth a merge/rebase before merging.
2. pvc-storageclass false-FAIL in one config. The check compares each PVC's .spec.storageClassName to the chart's expected class — exact-match-safe for all standard installs (the PVC sets that very value). The gap: storageClass.create=false + name="" (use the cluster default class) → the PVC renders a null class → default-class admission injects e.g. gp2 → the check compares against empty EXPECTED_STORAGE_CLASS and FAILs a healthy install as UNSEALED. Suggest treating an empty EXPECTED_STORAGE_CLASS as "skip the class assertion." It's the one path that reports unsealed when nothing is wrong.
3. Schema vs sealCheck: null. The template guards sealCheck: null, but values.schema.json types it as object, so an explicit helm install --set sealCheck=null would fail schema validation. --reuse-values from a pre-sealCheck release sends it absent (not null), which is fine — so real-world impact is ~nil; just noting the "unit-tested with sealCheck: null" claim would not survive real schema validation.

Cosmetic: when a legacy-PV FAIL fires and PV_SCAN=true, the same PV is still re-inspected below (harmless double output).

— @shujaatTracebloc (reviewed with Claude)

[LukasWodka]

Reviewed the seal-check suite end to end — the render contract, script logic, RBAC minimalism, and fail-loud semantics all hold up, and the two Bugbot findings are genuinely fixed with regression pins. LGTM with nits; one real (docs-only) defect:

1. [Docs, real footgun] The single-check filter formula silently no-ops for backend-reachability.SEAL-CHECK.md says to run one check with --filter name=<release>-<check>-check, but the backend-reachability Job is named <release>-egress-reachability-check (job name ≠ check name). helm test myrel --filter name=myrel-backend-reachability-check matches zero hooks and exits 0 — a silent pass, exactly what this suite exists to prevent. Fix: list the literal job name per check in the suite table (the runbook further down already uses the correct name).

2. [Process] Rebase before merge. The branch is 13 commits behind develop, and develop's #368 (node-local storage) changes the very storage model storage-assertions verifies. No textual conflict, but the unittest + node-local render should re-run on the merged tree. (Being done now.)

3. [Housekeeping] Cluster-scoped hook RBAC can orphan. After a failed helm test + helm uninstall, the ClusterRole/CRB survive release and namespace deletion indefinitely (uninstall never touches hook resources; the namespaced SA/Role/RB die with the namespace). Inert risk (read-only PV get/list bound to a deleted SA), but the offboard/uninstall path should learn to sweep clusterrole/clusterrolebinding <release>-storage-assertions-check.

4. [Nit] activeDeadlineSeconds includes image pull. Budget = timeout + 180s while worst-case script runtime is ~timeout + ~125s of kubectl request-timeouts — a cold alpine/k8s pull on a slow link can eat the rest and die as DeadlineExceeded instead of printing its verdict. Fail-loud direction, and cache-hit is likely (same image as the image-refresh CronJob), so diagnosability-only.

5. [Test honesty, acknowledged] The Bugbot behavior fixes are pinned via matchRegex on the embedded script text — fair given helm-unittest can't execute shell, but the 9-scenario mocked-kubectl harness from the PR body would make those pins real if committed.

Also verified against @shujaatTracebloc's notes: the storageClass.create=false + name="" false-FAIL concern is defused by the values schema (storageClass.name has minLength: 1 and is required, so an empty expected class can't reach the check); the sealCheck: null unit-test observation is accurate but ~nil real-world impact (--reuse-values sends the key absent, and helm-unittest bypasses schema validation).

—
Review assisted by Claude Code.

[LukasWodka]

Review follow-ups done on this branch:

• F2 — merged develop (13 commits, incl. feat(installer): node-local storage prototype (RFC-0003 Option C, flag-gated) #368 node-local storage) and re-ran the suite on the merged tree: helm lint ✔ (with creds set, the CI pattern), helm unittest297/297 ✔, and the node-local render (storageClass.create=false, name=local-path, hostPath.enabled=false + creds) renders the storage-assertions check coherently (EXPECTED_STORAGE_CLASS=local-path, PV scan gated) ✔.
• F1 fixed — SEAL-CHECK.md now lists the literal --filter Job name per check; the generic <release>-<check>-check formula silently no-ops for backend-reachability (Job: egress-reachability-check — zero matched hooks, exit 0, silent pass).
• F3 (cluster-scoped hook RBAC offboard sweep) and F4 (deadline vs cold image pull) stay as noted follow-ups.

—
Review assisted by Claude Code.

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 232d10e. Configure here.}

[LukasWodka]

Functional review \u2014 passed\n\nBasis: the behavioural suites that ran on this PR at merge against real environments, not mocks:\n\n- + + \u2014 a real chart install and auto-upgrade cycle on a live k3d cluster\n\nI could not reach the dev API (no dev credentials), so rather than rubber-stamp I used the strongest evidence available: these suites exercise the actual behaviour this change alters, on real infrastructure. Advancing \u2192 .\n\nIf the functional reviewer wants a manual pass on dev in addition, please move it back and say so.