From a285bb5da2951a0849c0f2f1f864964cc775957e Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Tue, 21 Jul 2026 10:22:13 +0200 Subject: [PATCH 1/7] =?UTF-8?q?docs(rfc):=20add=20RFC-0003=20=E2=80=94=20d?= =?UTF-8?q?ataset=20storage=20&=20offboard=20hygiene=20(draft)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commits the drafted RFC-0003 into cli/docs/rfcs/ alongside 0001/0002. Concept-only design doc covering two user-visible problems that share one root cause: offboard should leave a true clean slate, and ingested datasets are stored as world-readable host files bind-mounted into the cluster (engineered to survive cluster deletion), which sits awkwardly against "your data stays within the secure environment". Storage options + open decisions (section 7) pending Lukas + Asad. Companion to backend#1151. Co-Authored-By: Claude Opus 4.8 --- .../rfcs/0003-storage-and-offboard-hygiene.md | 250 ++++++++++++++++++ 1 file changed, 250 insertions(+) create mode 100644 docs/rfcs/0003-storage-and-offboard-hygiene.md diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md new file mode 100644 index 00000000..f8a4275c --- /dev/null +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -0,0 +1,250 @@ +# RFC 0003 — Dataset storage & offboard hygiene: where data lives, and what "delete" means + +> **Status: DRAFT — for discussion.** Owner: @LukasWodka. Last updated: 2026-07-21. +> +> This RFC examines two user-visible problems that turn out to share one +> root cause: +> +> 1. **`tracebloc delete` (offboard) does not reliably leave a clean slate.** +> Leftover data from earlier installs/versions can be silently re-adopted +> by the next install. +> 2. **Ingested datasets are stored as world-readable files on the *host* +> filesystem**, bind-mounted into the cluster — which sits awkwardly +> against the promise that a user's data stays "within the secure +> environment." +> +> Both stem from one design choice: **datasets live as `chmod 777` files +> under `~/.tracebloc`, bind-mounted into the local cluster, and engineered +> to survive cluster deletion.** +> +> Grounded in a code-level read of the shipped installer (`client` +> `scripts/lib/{common,cluster}.sh`), the offboard command (CLI +> `internal/cli/delete.go`), and a controlled reproduction on a local k3d +> install (2026-07-21). **Concept only — no code proposed to merge yet.** +> +> Related: RFC-0001 (auth & client provisioning), RFC-0002 (data ingest flow +> & terminology), and the terminology source-of-truth (v2 `TERMINOLOGY.md`, +> docs `main`). Spans two repos: `cli` (offboard) and `client` (installer + +> chart). + +## 1. Summary + +Two problems, one root. + +- **Problem A — "delete" doesn't reliably mean gone (§3).** For the default + data directory, offboard *does* wipe correctly. But data written by an + *earlier* install — a different on-host layout, an older version, or a + custom directory — can be silently re-adopted by the next install, so a + "fresh" reinstall isn't fresh. The failure isn't in the wipe; it's that + **the installer adopts whatever it finds with no guard.** + +- **Problem B — data lives "outside" the secure environment (§4).** + Datasets are plain files in `~/.tracebloc` on the host (mode `777`), + bind-mounted into the cluster. That is on the user's *machine* (so the + real privacy promise holds) but is **not** sealed inside the cluster's + isolation boundary, and it is readable by any local process. + +- **Root cause.** The two are the same choice seen from different angles. + Storing datasets as host files that *outlive the cluster* is what makes + offboard fragile (A) **and** what puts the data outside the environment + boundary (B). Fix the storage model and both improve together. + +The guiding principle this RFC lands on: **separate restart-persistence +from delete-persistence** (§4.4). Users reboot their laptops and must keep +their data; users who offboard expect their data destroyed. Today's design +conflates the two. + +## 2. How it works today (verified) + +### 2.1 Where datasets physically live +- MySQL runs **in-cluster**; ingested datasets are tables in the + `training_test_datasets` database. +- The MySQL PersistentVolume uses a **hostPath** under `/tracebloc//mysql`. +- The local k3d cluster **bind-mounts `~/.tracebloc → /tracebloc`** + (`HOST_DATA_DIR`, default `$HOME/.tracebloc` — `common.sh:329`; + mapping documented at `cluster.sh:40-41`). +- Net effect: dataset tables physically live at + `~/.tracebloc//mysql` on the host disk. *(Confirmed: `.ibd` + files present there before the repro.)* +- **Dataset *files*** (as opposed to the MySQL tables) can go to a separate + location via `HOST_DATASET_DIR` — a network mount by design (backend#743) + — bind-mounted to a **different** path, `/tracebloc-data` (`cluster.sh:312`). + +### 2.2 Permissions +- The installer runs `chmod -R 777` on the mysql + logs directories + (`cluster.sh:31` flat layout, `cluster.sh:50` per-release layout). The + data is **world-readable/writable** to any local user or process. + +### 2.3 Install-time behavior — the actual bug +- The installer does `mkdir -p "$HOST_DATA_DIR/…/{logs,mysql}"` + (`cluster.sh:30`, `:49`). There is **no wipe and no "existing data + detected" guard** — if data is already present, MySQL silently adopts it. +- Two layouts exist in the same script: a **flat** layout + (`$HOST_DATA_DIR/{logs,mysql}`, `:30-31`) and a **per-release** layout + (`$HOST_DATA_DIR//{logs,mysql}`, `:49-50`). A machine that has + seen both (e.g. across a version bump) accumulates both — this is exactly + the transition artifact behind Problem A. + +### 2.4 Offboard behavior (`tracebloc delete`) +- Teardown order (`internal/cli/delete.go`): revoke credential → clear the + active-client pointer → Helm uninstall → k3d cluster delete → prune images + → `removeHostDataDir()` → remove self. +- `removeHostDataDir()` = `os.RemoveAll(config.Dir())` = + `os.RemoveAll(~/.tracebloc)`. +- Flags: `--yes` (skip name confirmation), `--keep-data` (keep + `~/.tracebloc`), `--force` (skip the online guard). +- **`HOST_DATASET_DIR` is never touched** — by design it's a network mount, + but that means dataset *files* there survive offboard silently. +- Prints `✔ Removed local tracebloc data and config` — but the removal is + **not verified** before the message prints. + +### 2.5 What the 2026-07-21 reproduction actually showed +- An earlier hypothesis — "delete isn't deleting data" — was **wrong**. + `tracebloc delete --force --yes` fully wiped `~/.tracebloc`: data, install + marker, all 4 `.ibd` files, **0 survivors**. +- The earlier "data survived" observation was a **version/multi-install + transition artifact**: mixed flat + per-release MySQL layouts across a + 0.9.2 → 0.9.3 bump, plus two CLI binaries installed + (`~/.local/bin/tracebloc`, removed by delete, and `/usr/local/bin/tracebloc`, + which survived). +- Conclusion: for the *default* dir, delete already clean-slates. The real + gaps are (a) **alternate/legacy locations** delete doesn't know about + (custom `HOST_DATA_DIR` from a prior install; `HOST_DATASET_DIR`), and (b) + **the installer silently re-adopting** whatever it finds (§2.3). + +## 3. Problem A — offboard should leave a clean slate + +### 3.1 What we want +"Delete" should mean **the environment's data is gone** — current version +*and* any leftover from earlier versions/installs — so a reinstall starts +empty. This is both a UX expectation ("I deleted it") and a privacy +expectation (offboard = data destroyed). + +### 3.2 The one real objection: scope +"Wipe *all* tracebloc data on the machine" is dangerous, because: + +- A machine can host **more than one environment**. *(Confirmed: a second + k3d cluster, `tb-copyreview`, exists on this Mac right now.)* A blind + machine-wide wipe would destroy the other environment. +- `HOST_DATASET_DIR` may be a **shared network mount** used by other tools. + Recursively deleting it is not ours to do by default. + +So clean-slate must be **scoped to the environment being offboarded**, never +a machine-wide `rm -rf`. + +### 3.3 Proposal +1. **Verified, scoped wipe.** Delete removes everything belonging to the + offboarded environment — its `~/.tracebloc/` across *both* layouts + (flat + per-release), its `HOST_DATASET_DIR` data for that env, and any + legacy remnants for *that* env — and **verifies the paths are gone before + printing success** (don't claim `✔ Removed` on an unverified `RemoveAll`). +2. **Installer leftover-guard.** If `HOST_DATA_DIR/…/mysql` is non-empty at + install time, the installer **stops and makes the user choose** + (reuse / wipe / pick a new dir) instead of silently adopting it. *This is + the guard that actually prevents the bug that was hit.* +3. **Keep `--keep-data`** as the explicit opt-out of the wipe. +4. **Optional `--purge-all`**, clearly labeled, for the rare "remove every + tracebloc trace on this machine" case — **never** the default, and it must + enumerate what it will destroy (including other environments) before + proceeding. + +## 4. Problem B — data lives "outside" the secure environment + +### 4.1 The tension +We tell users their data stays "within the secure environment." Physically, +datasets are **plain files in `~/.tracebloc` on the host** (mode `777`), +bind-mounted into the cluster. That is not "inside the cluster's isolation +boundary." + +### 4.2 What's true vs. what's imprecise +- **True (and the promise that matters):** the data never leaves the user's + own machine / infrastructure. Nothing is uploaded anywhere. The + federated-learning trust model holds completely. +- **Imprecise:** "within the secure environment," read as "sealed inside the + cluster," is not accurate. The data is host files, readable by any local + process — not sandboxed, not encrypted at rest. + +### 4.3 Why it's on the host at all +Local Kubernetes (k3d) has no durable storage except host-backed volumes. +tracebloc bind-mounts to `~/.tracebloc` specifically so data survives +**cluster deletion + recreation**. Note the subtlety: a **restart** +(Docker off/on, `k3d cluster start`) already preserves the node container and +its storage — the bind-mount is only needed to survive a **delete + recreate**, +which is exactly the persistence questioned in §3. **So the storage choice +directly causes Problem A.** + +### 4.4 The key principle +**Separate restart-persistence from delete-persistence.** + +- Restart-persistence (must keep): laptop reboots, Docker restarts, cluster + stop/start. The k3d node container already survives these. +- Delete-persistence (must not keep): offboard, `cluster delete`. Data + surviving these is the bug, not the feature. + +Today's bind-mount conflates them by making data survive *everything*. + +### 4.5 Storage options (trade-offs) + +| Option | Restart-persists | Delete wipes it | Host exposure | Notes | +|---|---|---|---|---| +| **A. Status quo** — bind-mount `~/.tracebloc`, mode `777` | ✅ | ⚠️ only if delete chases every path | ❌ world-readable files in `$HOME` | Survives delete → causes Problem A | +| **B. Docker managed named volume** (env-scoped) | ✅ | ✅ if the volume is removed on offboard | ✅ under Docker's dir, not `$HOME`, not `777` | Less exposed; lifecycle can bind to the env | +| **C. Node-local storage** (local-path provisioner, inside the k3d node) | ✅ (node container persists across stop/start) | ✅ (dies with `cluster delete`) | ✅ not visible as host files | Clean-slate by construction; loses "survive delete + recreate" | +| **D. B or C + drop `777` + encrypt at rest** | ✅ | ✅ | ✅✅ | Makes "within your secure environment" literally truer | + +For **real clusters** (EKS/AKS/OpenShift/bare-metal), dataset storage already +lands on proper PVs on the customer's own infrastructure — that is fine and +stays as-is. This RFC's storage question is about the **local** install. + +### 4.6 Security posture +- **Drop the `chmod 777`** regardless of which storage option — customer data + should not be world-readable/writable. (Low-risk, high-value; can ship + independently of the bigger decision.) +- **Consider encryption at rest**, so "secure environment" means more than + "a directory on your box." + +## 5. Messaging reconciliation +Whatever we build, the copy must match reality. Two directions: + +- **Tighten the wording** to the always-defensible claim — "your data never + leaves your infrastructure" — rather than implying cluster-sealed + isolation; **or** +- **Change the storage** (Option C/D) so "within the secure environment" + becomes literally true, and keep the stronger wording. + +Best outcome: do the storage work *and* keep the strong claim honest. Align +final wording with the terminology source-of-truth ("secure environment" is +the decided term). + +## 6. Non-goals +- Not changing how *remote* (cloud) clusters store data — PVs there are fine. +- Not breaking restart-persistence. +- Not (yet) selecting a specific volume implementation — §4.5 is the menu + §7 decides from. + +## 7. Decisions to make (Lukas + Asad) +- **[decision 7.1]** Offboard clean-slate: adopt the scoped, verified wipe + (incl. `HOST_DATASET_DIR`) **plus** the installer leftover-guard? (§3.3) +- **[decision 7.2]** Local storage target: **A** (status quo), **B** (managed + volume), **C** (node-local), or **D** (B/C + perms + encryption)? (§4.5) +- **[decision 7.3]** Drop `chmod 777` now, independent of the larger storage + decision? (§4.6 — low-risk, high-value) +- **[decision 7.4]** Messaging: tighten copy, change storage, or both? (§5) +- **[decision 7.5]** Migration for existing installs if we move the store (§8). + +## 8. Migration / rollout +If we move the store (Option B/C), existing installs have data in +`~/.tracebloc`. Options: a one-time migration (copy into the new store on +upgrade) or an accepted "re-ingest after upgrade" for local dev. Real +clusters are unaffected. Sequence this behind the installer's version +detection, and guard it so we **never silently strand data**. + +## 9. Appendix — evidence (verified 2026-07-21) +- `client/scripts/lib/common.sh:329` — `HOST_DATA_DIR="${HOST_DATA_DIR:-$HOME/.tracebloc}"` +- `client/scripts/lib/common.sh:335` — `HOST_DATASET_DIR="${HOST_DATASET_DIR:-}"` +- `client/scripts/lib/cluster.sh:30`, `:49` — `mkdir -p …/{logs,mysql}` (no existing-data guard) +- `client/scripts/lib/cluster.sh:31`, `:50` — `chmod -R 777 …/logs …/mysql` +- `client/scripts/lib/cluster.sh:40-41` — PVs `/tracebloc//{data,logs,mysql}` ↔ `$HOST_DATA_DIR//…` via the k3d `-v` mount +- `client/scripts/lib/cluster.sh:312` — dataset bind mount `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` +- `cli/internal/cli/delete.go` — `removeHostDataDir()` = `os.RemoveAll(config.Dir())`; flags `--yes` / `--keep-data` / `--force`; `HOST_DATASET_DIR` untouched; success message unverified +- Repro: `tracebloc delete --force --yes` → `~/.tracebloc` gone, 0 `.ibd` survivors; earlier "survival" = version/multi-install transition artifact From afe9cd193991efe0a23556bca3f4fb01f3e1a610 Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Wed, 22 Jul 2026 17:23:52 +0200 Subject: [PATCH 2/7] =?UTF-8?q?docs(rfc):=20RFC-0003=20v2=20=E2=80=94=20re?= =?UTF-8?q?cord=20decisions,=20define=20the=20secure-environment=20boundar?= =?UTF-8?q?y?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §7 decision menu -> §10 decision log (D1-D14): Option C node-local, leftover-guard, no data-copier, 777 dies with C, encryption phase 2 - New §1-2: secure-environment definition (3-in/1-out channel list) + threat model with honest ceilings (owner-root, data-derived weights) - New §6: model IP protection — watermarking + audit now, envelope encryption + crypto-shredding for at-rest weights (keys in memory, not weight files), TEE as phase 2, HE/MPC rejected - New §7: in-cluster walls — dataset-scoped mounts, per-experiment DB grants, spawned-pod hardening completion - New §8: egress-lockdown flip dependency, seal check, per-substrate guarantee matrix, verify k3d enforcement - §3/§13 evidence corrected + refreshed (777 scoping, cluster reuse, stale line numbers) - Open items: O1 default-flip, O2 retention, O3 key custody, O4 watermark Co-Authored-By: Claude Fable 5 --- .../rfcs/0003-storage-and-offboard-hygiene.md | 687 +++++++++++------- 1 file changed, 443 insertions(+), 244 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index f8a4275c..bd80c293 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -1,250 +1,449 @@ -# RFC 0003 — Dataset storage & offboard hygiene: where data lives, and what "delete" means +# RFC 0003 — The secure environment: dataset storage, offboard hygiene & the boundary -> **Status: DRAFT — for discussion.** Owner: @LukasWodka. Last updated: 2026-07-21. +> **Status: DRAFT v2 — decisions recorded 2026-07-22.** Owner: @LukasWodka. +> Co-design & validation: @saadqbal. > -> This RFC examines two user-visible problems that turn out to share one -> root cause: +> **v1 → v2.** v1 (2026-07-21) framed two problems — offboard hygiene and +> dataset storage — and left §7 as a decision menu. v2 records the decisions +> taken (with prototype validation on a real install, client#368), folds in +> the review feedback, and widens the lens to what those decisions were +> always serving: a precise, honest definition of the **secure environment** +> — its boundary (§1), threat model (§2), model-IP stance (§6), in-cluster +> walls (§7), and per-substrate conformance (§8). > -> 1. **`tracebloc delete` (offboard) does not reliably leave a clean slate.** -> Leftover data from earlier installs/versions can be silently re-adopted -> by the next install. -> 2. **Ingested datasets are stored as world-readable files on the *host* -> filesystem**, bind-mounted into the cluster — which sits awkwardly -> against the promise that a user's data stays "within the secure -> environment." -> -> Both stem from one design choice: **datasets live as `chmod 777` files -> under `~/.tracebloc`, bind-mounted into the local cluster, and engineered -> to survive cluster deletion.** -> -> Grounded in a code-level read of the shipped installer (`client` -> `scripts/lib/{common,cluster}.sh`), the offboard command (CLI -> `internal/cli/delete.go`), and a controlled reproduction on a local k3d -> install (2026-07-21). **Concept only — no code proposed to merge yet.** +> Section map from v1: §2→§3, §3→§4, §4→§5, §5→§9, §6→§11, §7→§10 +> (now a decision log), §8→§12, §9→§13. > > Related: RFC-0001 (auth & client provisioning), RFC-0002 (data ingest flow -> & terminology), and the terminology source-of-truth (v2 `TERMINOLOGY.md`, -> docs `main`). Spans two repos: `cli` (offboard) and `client` (installer + -> chart). - -## 1. Summary - -Two problems, one root. - -- **Problem A — "delete" doesn't reliably mean gone (§3).** For the default - data directory, offboard *does* wipe correctly. But data written by an - *earlier* install — a different on-host layout, an older version, or a - custom directory — can be silently re-adopted by the next install, so a - "fresh" reinstall isn't fresh. The failure isn't in the wipe; it's that - **the installer adopts whatever it finds with no guard.** - -- **Problem B — data lives "outside" the secure environment (§4).** - Datasets are plain files in `~/.tracebloc` on the host (mode `777`), - bind-mounted into the cluster. That is on the user's *machine* (so the - real privacy promise holds) but is **not** sealed inside the cluster's - isolation boundary, and it is readable by any local process. - -- **Root cause.** The two are the same choice seen from different angles. - Storing datasets as host files that *outlive the cluster* is what makes - offboard fragile (A) **and** what puts the data outside the environment - boundary (B). Fix the storage model and both improve together. - -The guiding principle this RFC lands on: **separate restart-persistence -from delete-persistence** (§4.4). Users reboot their laptops and must keep -their data; users who offboard expect their data destroyed. Today's design -conflates the two. - -## 2. How it works today (verified) - -### 2.1 Where datasets physically live +> & terminology), terminology source-of-truth (v2 `TERMINOLOGY.md`, docs +> `main`). Design ticket: backend#1151. Storage prototype: client#368 +> (flag-gated, draft). Offboard honesty fix: cli#389. Spans `cli`, `client`, +> `client-runtime`, `backend`. + +## 1. The secure environment — definition + +A **secure environment** is a sealed enclave on infrastructure the customer +controls — a laptop (k3d), a cloud cluster (EKS/AKS/OpenShift), or bare +metal; the promise is the same in every form. It has **exactly three +ingress channels, one egress channel, and nothing sideways**: + +| Direction | Channel | Carries | Transport | +|---|---|---|---| +| In | **Data ingester** | raw datasets, from sources the customer configures | ingest jobs (local files / network share) | +| In | **tracebloc control plane** | models, weights, experiment instructions | backend API + Service Bus (TLS) | +| In | **tracebloc software** | container images, chart, CLI | registries — all images digest-pinned | +| Out | **tracebloc control plane** | trained weights, metrics, status/logs | Service Bus + allowlisted FQDNs (TLS) | + +Everything else is sealed: the environment does not reach into host +directories (§5), nothing on the host casually reads the environment's data +(§5), and workloads inside cannot reach arbitrary networks (§8). The +environment only dials out — no inbound ports are required. + +**Wording discipline.** Externally we say: *defined, auditable ingress and +egress — and raw data is never an egress channel.* We do **not** say +"air-gapped": the environment *requires* outbound connectivity (it +authenticates to the backend to obtain its Service Bus credentials; no +backend egress ⇒ experiments sit Pending), so a literal air-gap claim fails +the first serious security review. "Almost air-gapped" is internal +shorthand only. + +## 2. Threat model — protect what, from whom + +The environment protects **two assets for two parties**: the data owner's +datasets, and the model provider's model IP. Adversary by adversary: + +| Adversary | Datasets protected by | Model IP protected by | Status | +|---|---|---|---| +| External network attacker | TLS everywhere; outbound-only; NetworkPolicy | same | enforced | +| Other services/users on the host | node-local storage — no host files, no 777, no bind-mount (§5) | same | decided (D1/D2) | +| Vendor training code in-cluster | egress lockdown (§8.1) + dataset-scoped mounts + scoped DB grants (§7) | n/a — it *is* the model | built-inert / decided | +| tracebloc itself | raw data never leaves; only weights + defined metrics egress | n/a | by design | +| Cloud provider (cloud form) | customer-run infra; encrypted PVs; TEE later | TEE later (§6.5) | partial | +| **Environment owner (root)** | out of scope — the data is theirs | hygiene + watermark/audit now (§6.3–6.4); TEE phase 2 (§6.5) | decided (D7/D8) | + +**Honest ceilings — never claim past these:** + +1. **The machine owner has root, Docker, and the disk.** On classical + hardware, nothing that executes there can be made absolutely + inaccessible to them. For datasets that is fine (the data is theirs); + for model IP, see §6. +2. **The sanctioned exit carries data-derived information.** Weights and + metrics are functions of the training data — that is the product. The + defensible claim is "**raw data never leaves**", not "nothing derived + from your data leaves". Residual leakage (memorization, membership + inference) is a known property of federated learning; differential + privacy / secure aggregation are future options, out of scope here. +3. **Tampering can be made detectable, not impossible** (owner-root + again). Dataset integrity & score reproducibility is deliberately a + separate RFC (§11). + +## 3. How it works today (verified 2026-07-21; evidence refreshed 2026-07-22) + +### 3.1 Where data and weights physically live - MySQL runs **in-cluster**; ingested datasets are tables in the - `training_test_datasets` database. -- The MySQL PersistentVolume uses a **hostPath** under `/tracebloc//mysql`. -- The local k3d cluster **bind-mounts `~/.tracebloc → /tracebloc`** - (`HOST_DATA_DIR`, default `$HOME/.tracebloc` — `common.sh:329`; - mapping documented at `cluster.sh:40-41`). -- Net effect: dataset tables physically live at - `~/.tracebloc//mysql` on the host disk. *(Confirmed: `.ibd` - files present there before the repro.)* -- **Dataset *files*** (as opposed to the MySQL tables) can go to a separate - location via `HOST_DATASET_DIR` — a network mount by design (backend#743) - — bind-mounted to a **different** path, `/tracebloc-data` (`cluster.sh:312`). - -### 2.2 Permissions -- The installer runs `chmod -R 777` on the mysql + logs directories - (`cluster.sh:31` flat layout, `cluster.sh:50` per-release layout). The - data is **world-readable/writable** to any local user or process. - -### 2.3 Install-time behavior — the actual bug -- The installer does `mkdir -p "$HOST_DATA_DIR/…/{logs,mysql}"` - (`cluster.sh:30`, `:49`). There is **no wipe and no "existing data - detected" guard** — if data is already present, MySQL silently adopts it. -- Two layouts exist in the same script: a **flat** layout - (`$HOST_DATA_DIR/{logs,mysql}`, `:30-31`) and a **per-release** layout - (`$HOST_DATA_DIR//{logs,mysql}`, `:49-50`). A machine that has - seen both (e.g. across a version bump) accumulates both — this is exactly - the transition artifact behind Problem A. - -### 2.4 Offboard behavior (`tracebloc delete`) -- Teardown order (`internal/cli/delete.go`): revoke credential → clear the - active-client pointer → Helm uninstall → k3d cluster delete → prune images - → `removeHostDataDir()` → remove self. -- `removeHostDataDir()` = `os.RemoveAll(config.Dir())` = - `os.RemoveAll(~/.tracebloc)`. -- Flags: `--yes` (skip name confirmation), `--keep-data` (keep - `~/.tracebloc`), `--force` (skip the online guard). -- **`HOST_DATASET_DIR` is never touched** — by design it's a network mount, - but that means dataset *files* there survive offboard silently. -- Prints `✔ Removed local tracebloc data and config` — but the removal is - **not verified** before the message prints. - -### 2.5 What the 2026-07-21 reproduction actually showed -- An earlier hypothesis — "delete isn't deleting data" — was **wrong**. - `tracebloc delete --force --yes` fully wiped `~/.tracebloc`: data, install - marker, all 4 `.ibd` files, **0 survivors**. -- The earlier "data survived" observation was a **version/multi-install - transition artifact**: mixed flat + per-release MySQL layouts across a - 0.9.2 → 0.9.3 bump, plus two CLI binaries installed - (`~/.local/bin/tracebloc`, removed by delete, and `/usr/local/bin/tracebloc`, - which survived). -- Conclusion: for the *default* dir, delete already clean-slates. The real - gaps are (a) **alternate/legacy locations** delete doesn't know about - (custom `HOST_DATA_DIR` from a prior install; `HOST_DATASET_DIR`), and (b) - **the installer silently re-adopting** whatever it finds (§2.3). - -## 3. Problem A — offboard should leave a clean slate - -### 3.1 What we want -"Delete" should mean **the environment's data is gone** — current version -*and* any leftover from earlier versions/installs — so a reinstall starts -empty. This is both a UX expectation ("I deleted it") and a privacy -expectation (offboard = data destroyed). - -### 3.2 The one real objection: scope -"Wipe *all* tracebloc data on the machine" is dangerous, because: - -- A machine can host **more than one environment**. *(Confirmed: a second - k3d cluster, `tb-copyreview`, exists on this Mac right now.)* A blind - machine-wide wipe would destroy the other environment. -- `HOST_DATASET_DIR` may be a **shared network mount** used by other tools. - Recursively deleting it is not ours to do by default. - -So clean-slate must be **scoped to the environment being offboarded**, never -a machine-wide `rm -rf`. - -### 3.3 Proposal -1. **Verified, scoped wipe.** Delete removes everything belonging to the - offboarded environment — its `~/.tracebloc/` across *both* layouts - (flat + per-release), its `HOST_DATASET_DIR` data for that env, and any - legacy remnants for *that* env — and **verifies the paths are gone before - printing success** (don't claim `✔ Removed` on an unverified `RemoveAll`). -2. **Installer leftover-guard.** If `HOST_DATA_DIR/…/mysql` is non-empty at - install time, the installer **stops and makes the user choose** - (reuse / wipe / pick a new dir) instead of silently adopting it. *This is - the guard that actually prevents the bug that was hit.* -3. **Keep `--keep-data`** as the explicit opt-out of the wipe. -4. **Optional `--purge-all`**, clearly labeled, for the rare "remove every - tracebloc trace on this machine" case — **never** the default, and it must - enumerate what it will destroy (including other environments) before - proceeding. - -## 4. Problem B — data lives "outside" the secure environment - -### 4.1 The tension -We tell users their data stays "within the secure environment." Physically, -datasets are **plain files in `~/.tracebloc` on the host** (mode `777`), -bind-mounted into the cluster. That is not "inside the cluster's isolation -boundary." - -### 4.2 What's true vs. what's imprecise -- **True (and the promise that matters):** the data never leaves the user's - own machine / infrastructure. Nothing is uploaded anywhere. The - federated-learning trust model holds completely. -- **Imprecise:** "within the secure environment," read as "sealed inside the - cluster," is not accurate. The data is host files, readable by any local - process — not sandboxed, not encrypted at rest. - -### 4.3 Why it's on the host at all -Local Kubernetes (k3d) has no durable storage except host-backed volumes. -tracebloc bind-mounts to `~/.tracebloc` specifically so data survives -**cluster deletion + recreation**. Note the subtlety: a **restart** -(Docker off/on, `k3d cluster start`) already preserves the node container and -its storage — the bind-mount is only needed to survive a **delete + recreate**, -which is exactly the persistence questioned in §3. **So the storage choice -directly causes Problem A.** - -### 4.4 The key principle -**Separate restart-persistence from delete-persistence.** - -- Restart-persistence (must keep): laptop reboots, Docker restarts, cluster - stop/start. The k3d node container already survives these. -- Delete-persistence (must not keep): offboard, `cluster delete`. Data - surviving these is the bug, not the feature. - -Today's bind-mount conflates them by making data survive *everything*. - -### 4.5 Storage options (trade-offs) - -| Option | Restart-persists | Delete wipes it | Host exposure | Notes | -|---|---|---|---|---| -| **A. Status quo** — bind-mount `~/.tracebloc`, mode `777` | ✅ | ⚠️ only if delete chases every path | ❌ world-readable files in `$HOME` | Survives delete → causes Problem A | -| **B. Docker managed named volume** (env-scoped) | ✅ | ✅ if the volume is removed on offboard | ✅ under Docker's dir, not `$HOME`, not `777` | Less exposed; lifecycle can bind to the env | -| **C. Node-local storage** (local-path provisioner, inside the k3d node) | ✅ (node container persists across stop/start) | ✅ (dies with `cluster delete`) | ✅ not visible as host files | Clean-slate by construction; loses "survive delete + recreate" | -| **D. B or C + drop `777` + encrypt at rest** | ✅ | ✅ | ✅✅ | Makes "within your secure environment" literally truer | - -For **real clusters** (EKS/AKS/OpenShift/bare-metal), dataset storage already -lands on proper PVs on the customer's own infrastructure — that is fine and -stays as-is. This RFC's storage question is about the **local** install. - -### 4.6 Security posture -- **Drop the `chmod 777`** regardless of which storage option — customer data - should not be world-readable/writable. (Low-risk, high-value; can ship - independently of the bigger decision.) -- **Consider encryption at rest**, so "secure environment" means more than - "a directory on your box." - -## 5. Messaging reconciliation -Whatever we build, the copy must match reality. Two directions: - -- **Tighten the wording** to the always-defensible claim — "your data never - leaves your infrastructure" — rather than implying cluster-sealed - isolation; **or** -- **Change the storage** (Option C/D) so "within the secure environment" - becomes literally true, and keep the stronger wording. - -Best outcome: do the storage work *and* keep the strong claim honest. Align -final wording with the terminology source-of-truth ("secure environment" is -the decided term). - -## 6. Non-goals -- Not changing how *remote* (cloud) clusters store data — PVs there are fine. -- Not breaking restart-persistence. -- Not (yet) selecting a specific volume implementation — §4.5 is the menu - §7 decides from. - -## 7. Decisions to make (Lukas + Asad) -- **[decision 7.1]** Offboard clean-slate: adopt the scoped, verified wipe - (incl. `HOST_DATASET_DIR`) **plus** the installer leftover-guard? (§3.3) -- **[decision 7.2]** Local storage target: **A** (status quo), **B** (managed - volume), **C** (node-local), or **D** (B/C + perms + encryption)? (§4.5) -- **[decision 7.3]** Drop `chmod 777` now, independent of the larger storage - decision? (§4.6 — low-risk, high-value) -- **[decision 7.4]** Messaging: tighten copy, change storage, or both? (§5) -- **[decision 7.5]** Migration for existing installs if we move the store (§8). - -## 8. Migration / rollout -If we move the store (Option B/C), existing installs have data in -`~/.tracebloc`. Options: a one-time migration (copy into the new store on -upgrade) or an accepted "re-ingest after upgrade" for local dev. Real -clusters are unaffected. Sequence this behind the installer's version -detection, and guard it so we **never silently strand data**. - -## 9. Appendix — evidence (verified 2026-07-21) -- `client/scripts/lib/common.sh:329` — `HOST_DATA_DIR="${HOST_DATA_DIR:-$HOME/.tracebloc}"` -- `client/scripts/lib/common.sh:335` — `HOST_DATASET_DIR="${HOST_DATASET_DIR:-}"` -- `client/scripts/lib/cluster.sh:30`, `:49` — `mkdir -p …/{logs,mysql}` (no existing-data guard) -- `client/scripts/lib/cluster.sh:31`, `:50` — `chmod -R 777 …/logs …/mysql` -- `client/scripts/lib/cluster.sh:40-41` — PVs `/tracebloc//{data,logs,mysql}` ↔ `$HOST_DATA_DIR//…` via the k3d `-v` mount -- `client/scripts/lib/cluster.sh:312` — dataset bind mount `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` -- `cli/internal/cli/delete.go` — `removeHostDataDir()` = `os.RemoveAll(config.Dir())`; flags `--yes` / `--keep-data` / `--force`; `HOST_DATASET_DIR` untouched; success message unverified -- Repro: `tracebloc delete --force --yes` → `~/.tracebloc` gone, 0 `.ibd` survivors; earlier "survival" = version/multi-install transition artifact + `training_test_datasets` database. Weight files at rest also live inside + the environment (DB/data volume): an environment runs **tens of + concurrent trainings**, so weights cannot simply be held in memory. +- The MySQL PersistentVolume uses a **hostPath** under + `/tracebloc//mysql`; the local k3d cluster **bind-mounts + `~/.tracebloc → /tracebloc`** (`HOST_DATA_DIR`, default + `$HOME/.tracebloc`). Net effect: dataset tables are host files. +- Dataset *files* can additionally come from `HOST_DATASET_DIR` (a network + mount by design, backend#743), bind-mounted **cluster-wide** (`@all`) at + `/tracebloc-data` — a wider window than a single ingest needs (§7). + +### 3.2 Permissions (corrected from v1) +- `_ensure_tracebloc_dirs` chmods **the `logs`/`mysql`/`data` subdirs** to + `777` — not all of `~/.tracebloc`, and `values.yaml` is spared. v1 + overstated the blast radius. +- On the hostPath model that 777 is **load-bearing** (the host user writes + into the shared dirs, and kubelet does not apply fsGroup to hostPath + volumes) — so v1 §4.6's "can ship independently" was wrong. The 777 + disappears **with Option C**, not before it (D2). + +### 3.3 Install & upgrade behavior +- Install does `mkdir -p` with **no existing-data guard** — leftover data + is silently adopted. Flat and per-release layouts coexist across + versions; a machine that has seen both accumulates both (Problem A). +- On re-run, the installer **reuses** an existing cluster ("already exists" + → use it; `cluster start`) — it does not recreate. In-place upgrades + therefore keep data; only an explicit `cluster delete` destroys it. This + fact is load-bearing for D1 and D4. + +### 3.4 Offboard (`tracebloc delete`) +- Teardown order: revoke credential → clear active-client pointer → Helm + uninstall → k3d cluster delete → prune images → remove host data dir → + remove self. Flags: `--yes`, `--keep-data`, `--force`. +- The host-data wipe is now **verified before printing ✔** (cli#389) — + a nil `RemoveAll` is no longer treated as proof. +- `HOST_DATASET_DIR` is never touched (by design — shared network mount). + +### 3.5 Egress today (added in v2) +- The training NetworkPolicy ships **enabled**, allowing DNS, in-cluster + MySQL, the requests-proxy, and the egress gateway — **plus a + `0.0.0.0/0:443` rule** (`networkPolicy.training.allowExternalHttps` + defaults to `true`). The squid egress gateway (FQDN allowlist: backend + + App Insights) ships **inert** (`egressProxy.routeWorkloads: false`). +- Translation: the lockdown is **built but not flipped** — today a training + pod can still reach any external host on :443 (§8.1). +- Enforcement tooling exists: a `helm test` probe verifies the CNI actually + blocks egress, and a reachability check verifies required backend egress + works. + +### 3.6 Spawned-job hardening today +- New-architecture images run with `readOnlyRootFilesystem`, write weights + and scratch to a pod-scoped `emptyDir` (`EXPERIMENT_SCRATCH_PATH` — dies + with the pod), and get read-only shared mounts. **Legacy images are + carved out** of parts of this (they write inside the image filesystem). + +### 3.7 The 2026-07-21 reproduction (unchanged from v1) +- "Delete isn't deleting" was **wrong**: `tracebloc delete --force --yes` + fully wiped `~/.tracebloc` (0 survivors). The earlier "survival" was a + flat/per-release **transition artifact** across a version bump, plus a + second CLI binary in `/usr/local/bin`. The real gaps: alternate/legacy + locations, and the installer silently re-adopting whatever it finds. + +## 4. Problem A — offboard leaves a clean slate → DECIDED + +**What we want:** "delete" means the environment's data is gone — current +version and leftovers — so a reinstall starts empty. UX expectation and +privacy expectation at once. + +**Scope constraint (unchanged):** never a machine-wide wipe. A machine can +host multiple environments, and `HOST_DATASET_DIR` may be a shared network +mount other tools use. Clean-slate is scoped to the environment being +offboarded. + +**Decision (D3):** +1. **Installer leftover-guard** — if data is present at install time, stop + and make the user choose (reuse / wipe / new dir) instead of silently + adopting. This is the guard that prevents the bug that was actually hit, + and it doubles as the migration prompt (D4). +2. **Drop the heavy multi-path wipe** proposed in v1 — under node-local + storage (§5) there are no host data paths left to chase. `delete` keeps + config/token cleanup and **verifies the wipe before printing ✔** + (shipped as cli#389). +3. `--keep-data` stays as the explicit opt-out. A `--purge-all` (enumerate, + confirm, never default) is deferred until someone actually needs it. + +## 5. Problem B — where datasets live → DECIDED: Option C (node-local) + +**The principle (unchanged from v1): separate restart-persistence from +delete-persistence.** Laptops reboot and data must survive; offboard/delete +means destroyed. The hostPath bind-mount conflated the two by making data +survive everything — which is exactly what made offboard fragile (Problem +A) *and* put data outside the environment boundary (Problem B). + +The v1 menu, with the decision marked: + +| Option | Restart-persists | Delete wipes it | Host exposure | +|---|---|---|---| +| A. Status quo — bind-mount `~/.tracebloc`, 777 subdirs | ✅ | ⚠️ only if delete chases every path | ❌ world-readable files in `$HOME` | +| B. Docker managed named volume (env-scoped) | ✅ | ✅ if removed on offboard | ✅ | +| **C. Node-local storage (k3s `local-path`, inside the k3d node) — CHOSEN (D1)** | ✅ (node container persists across stop/start) | ✅ (dies with `cluster delete`) | ✅ not visible as host files | +| D. C + encryption at rest | ✅ | ✅ | ✅✅ — **phase 2** (D5) | + +**Why C:** datasets and MySQL move onto k3s's built-in `local-path` +provisioner *inside the k3d node*. Delete destroys the data by +construction; there is no browsable host folder; the 777 goes away by +construction (D2); restarts keep working (the node container and its +Docker volume survive stop/start); and upgrades keep data because the +installer **reuses** the cluster (§3.3). The v1 objection to C — +"loses survive-delete+recreate" — dissolves: delete+recreate is precisely +the case where data *should* die. + +**Validated on a real credentialed dev install (client#368, 2026-07-22):** +single-node cluster, all PVCs Bound on `local-path`, zero hostPath PVs; +real jobs-manager-spawned ingest jobs mounted the shared PVC and registered +datasets against the backend; stop/start preserved data; `cluster delete` +destroyed the node, its Docker volume, and all data; nothing under +`~/.tracebloc`. The prototype run also caught and fixed a real leak (a +second `_ensure_tracebloc_dirs` call site still creating empty 777 dirs). + +**Constraint C1 — single-node.** `local-path` is RWO/WaitForFirstConsumer, +and spawned Jobs must land on the node that holds the volume, so node-local +forces `AGENTS=0`. Since `AGENTS` defaults to `1` today, **flipping the +default also changes the default local topology from two nodes to one** — +call this out in release notes. (k3s agents on one Docker host provide no +real isolation, so nothing of value is lost.) + +**Out of scope for now:** `HOST_DATASET_DIR` (network-mount ingest sources) +stays on the hostpath path; combining it with node-local is a follow-up. + +**Migration (D4): no data-copier.** `--reuse-values` upgrades keep existing +installs on their current `~/.tracebloc`; they move to C on a clean +delete + reinstall — consistent with "delete means gone". The leftover-guard +(D3) ensures data is never silently stranded or adopted. + +**Encryption at rest (D5): phase 2.** For local installs, recommend host +full-disk encryption (FileVault/LUKS); in cloud, encrypted PVs. App-level +crypto for the volume is not warranted for a local dev tool unless the +threat model changes — and the real answer to "protected from a local +admin" is §6.5, not filesystem crypto. + +**Open (O1):** flip node-local from flag-gated (`TB_STORAGE_MODE`) to +**default** for local installs — the step that actually delivers this RFC +to users. Recommendation: flip after one green end-to-end **training run** +on node-local (the only client#368-checklist item not yet exercised; it was +blocked by an orthogonal dev auth issue, not by storage). + +## 6. Model IP protection — two-sided trust (new in v2) + +### 6.1 Goal +The environment owner must not be able to access the vendor models that +run inside their environment. This completes the marketplace trust story in +both directions — the data owner's data is protected from the vendor, +**and** the vendor's model is protected from the data owner. Almost nobody +in the FL space offers the second half; it is worth building toward +deliberately. + +### 6.2 The ceiling — state it before designing around it +To execute, weights must exist in plaintext in RAM/VRAM, and any decryption +key must be present in the environment. Root can dump process memory, GPU +memory, or the disk at any moment *during* execution — so at-rest +encryption alone, obfuscation, and delete-after-run all raise the bar +without changing the outcome. **On classical hardware, prevention against +owner-root is impossible; only trusted execution environments (§6.5) +change that.** Everything in 6.3–6.4 is deterrence, detection, and +minimization — valuable, and never to be sold as prevention. + +### 6.3 Now — watermarking + audit (D7) +Fingerprint delivered weights **per environment** (traitor tracing): a +leaked model is attributable to the environment it leaked from, which makes +the contractual protection enforceable. Log model-delivery events. This is +detection and legal recourse, not prevention — and it is what makes 6.4 +credible commercially. + +### 6.4 Now — runtime hygiene, maximum practical (D8) +1. **Weights enter at runtime over TLS, never baked into images** (already + true — images stay generic per task type; `docker save` yields no IP). +2. **The active working copy** lives in pod-scoped scratch + (`EXPERIMENT_SCRATCH_PATH`, an `emptyDir`) and dies with the pod — + already true on new-architecture images; close the legacy-image + carve-out (D11). +3. **At rest: envelope encryption + crypto-shredding.** An environment runs + tens of concurrent trainings, so weight files cannot live in RAM — they + rest in the environment's DB / data volume. Therefore: store weight + blobs as **ciphertext**; the per-experiment data-encryption key + (~32 bytes) is issued by the backend at cycle start and held **only in + memory**, never persisted inside the environment. N concurrent + experiments cost N small keys in RAM — not N weight files. On experiment + completion (or offboard, or revocation) the key is discarded and the + backend refuses re-issue: every at-rest copy — including disk snapshots + and backups — becomes unrecoverable. **Crypto-shredding is deletion that + is instant, size-independent, and verifiable**, and it upgrades what + "delete" means for weights on top of §4/§5. +4. **Lifecycle:** TTL finished Jobs (`ttlSecondsAfterFinished`); offboard + already prunes images (§3.4). + +Stated ceiling: root can still capture plaintext *during* execution — 6.4 +stops casual and after-the-fact access, not a determined owner. That gap +belongs to 6.5. + +### 6.5 Phase 2 — confidential computing (TEE) (D14) +AMD SEV-SNP / Intel TDX confidential VMs plus NVIDIA H100/H200 +confidential-computing mode encrypt memory with keys held in silicon; the +host, hypervisor, and root **cannot** read enclave memory. Remote +attestation lets tracebloc act as key broker: weights decrypt **only** +inside a measured, attested enclave. This is the one mechanism that +actually delivers "the owner cannot access the model" — and it also +strengthens the *data* story in cloud deployments (data protected from the +cloud provider). Azure sells confidential GPU VMs today, so a pilot on +cloud-form environments is the natural first step **when build capacity +allows — explicitly deferred for now** (decided 2026-07-22). On-prem +follows where customer hardware supports it; local k3d never gets it. +Bridge until then: **sensitivity tiering** — vendors can flag crown-jewel +models to run only on TEE-attested environments; commodity/open-backbone +models run anywhere. + +### 6.6 Rejected +Homomorphic encryption / MPC for training: orders of magnitude too slow for +deep learning at this scale. Named here so it doesn't resurface. + +## 7. In-cluster walls — dataset & experiment scoping (new in v2, DECIDED) + +Being inside the box must not mean seeing everything in the box. The +precise promise is "no service **outside the sanctioned training flow** +touches the data" — and the sanctioned flow is *per experiment*. A training +job must reach exactly its own dataset and its own artifacts, nothing else. + +1. **Dataset-scoped mounts (D9).** Jobs-manager currently mounts the whole + shared data PVC into spawned jobs. Near term: mount only the job's + dataset directory (`subPath`), read-only for training. End state under + Option C: **one PVC per dataset** — `local-path` dynamic provisioning + makes per-dataset PVCs cheap — so a job physically cannot see other + datasets. This also narrows the cluster-wide `HOST_DATASET_DIR` window + where that path is used. +2. **The database layer (D10).** Tabular and time-series datasets are + MySQL tables, and the training NetworkPolicy allows training→MySQL — + PVC scoping alone does not cover them. Adopt **per-experiment DB + credentials whose grants cover only that experiment's dataset tables** + (and its own weight rows, which composes with §6.4): short-lived, + injected at job start, revoked after. +3. **Finish spawned-pod hardening (D11).** Close the legacy-image + read-only carve-out; set `automountServiceAccountToken: false` on + spawned jobs; keep `readOnlyRootFilesystem` + read-only shared mounts as + the floor. + +## 8. Boundary enforcement & conformance (new in v2) + +### 8.1 Egress lockdown — flip it (D6, separate ticket) +The outbound wall exists (training NetworkPolicy, squid FQDN-allowlist +gateway, enforcement probe, backend-reachability check) but **ships +permissive**: `allowExternalHttps: true` keeps a `0.0.0.0/0:443` rule and +`routeWorkloads: false` leaves the gateway inert. Until the per-fleet flip +(verify gateway → route workloads → drop the 443 rule), "nothing gets out" +is not an enforced property for training pods on :443. Tracked as its own +ticket; this RFC records the dependency: **the golden-box claim is gated on +that flip** more than on anything else in this document. + +### 8.2 The seal check (D12) +Productize the existing probes into **one conformance suite** — the +egress-enforcement probe, the required-backend-reachability check, and +storage checks (post-C: no hostPath PVs, PVCs on the expected class, +nothing under `~/.tracebloc`) — runnable at install, at upgrade, and on +demand, surfaced pass/fail in the CLI. Design stance the chart already +takes: **silent non-protection is worse than explicit disabling.** An +environment that cannot enforce a guarantee is explicitly marked *unsealed* +— never silently claimed sealed. + +### 8.3 The guarantee matrix (D12) +Enforcement differs per substrate; the matrix is the honest artifact a +customer security review can quote. To be filled precisely as part of D12 +(cells: enforced / conditional / recommended / not available): + +| Guarantee | k3d local | EKS | AKS | OpenShift | bare metal | +|---|---|---|---|---|---| +| Storage inside cluster boundary (§5) | decided (C) | native PV | native PV | native PV | native PV | +| NetworkPolicy egress enforcement | verify (k3s embedded) | conditional (CNI mode) | conditional (CNI) | native (OVN) | conditional (CNI) | +| Encryption at rest | host FDE (recommended) | encrypted EBS | encrypted disks | platform | site policy | +| Confidential compute (§6.5) | not available | phase 2 | phase 2 (pilot) | phase 2 | hardware-dependent | + +### 8.4 Verify local enforcement (D12) +Do not assume k3d enforces NetworkPolicy — verify the k3s-embedded +controller blocks egress on a local install and fold that probe into the +seal check. + +## 9. Messaging reconciliation + +- The **channel list in §1 is the quotable definition** of the secure + environment — align docs, website, and sales material with it and with + the terminology source-of-truth. +- Keep the strong "within the secure environment" wording — Option C makes + it literally true for local storage. Internal precision: node-local means + *not-host-visible + dies-with-cluster*, not *cryptographically secure*. +- Never "air-gapped" in external copy (§1). +- Model-IP claims: "technically protected" only where TEE-attested (§6.5, + phase 2). Until then the accurate sentence is: *runtime-hygienic, + watermarked, and contractually protected.* + +## 10. Decision log (2026-07-22, Lukas + Asad) + +| # | Decision (v1 ref) | Outcome | +|---|---|---| +| D1 | Local storage target (§7.2) | **Option C — node-local**, validated in client#368 | +| D2 | Drop `chmod 777` (§7.3) | **Yes — by construction with C** (load-bearing on hostpath, cannot ship separately) | +| D3 | Offboard clean-slate (§7.1) | **Leftover-guard yes; heavy multi-path wipe no; verify-before-✔** (cli#389) | +| D4 | Migration (§7.5) | **No data-copier**; guard prevents stranding; installs move to C on delete+reinstall | +| D5 | At-rest encryption (§7.4) | **Phase 2**; recommend host FDE; keep wording honest | +| D6 | Egress lockdown flip | **Separate ticket**; golden-box claim gated on it (§8.1) | +| D7 | Model IP — now | **Watermarking + audit** (§6.3) | +| D8 | Weights at rest | **Envelope encryption + crypto-shredding — keys in memory, not weight files** (§6.4) | +| D9 | Dataset scoping | **Scoped mounts → per-dataset PVCs under C** (§7.1) | +| D10 | DB scoping | **Per-experiment DB credentials, table-scoped grants** (§7.2) | +| D11 | Pod hardening | **Close legacy carve-out; no SA token; read-only floor** (§7.3) | +| D12 | Conformance | **Guarantee matrix + seal check + verify k3d enforcement** (§8.2–8.4) | +| D13 | Score/tamper integrity | **Separate RFC** — versioning+integrity root cause, not storage (§11) | +| D14 | Confidential compute | **Phase 2 — deferred for capacity**; Azure confidential-GPU pilot first; tiering as bridge (§6.5) | + +Open items: + +| # | Question | Recommendation | +|---|---|---| +| O1 | Flip node-local to **default** for local installs (also flips default local topology to single-node) | Yes — after one green training run on node-local | +| O2 | Weight retention: exactly when does crypto-shred fire (experiment completion? grace window? resume/audit needs?) | Shred at completion + configurable grace window | +| O3 | DEK custody: backend-issued per cycle vs in-environment key service | Backend-issued — revocation/offboard instant; the environment already requires backend egress | +| O4 | Watermarking mechanics & owner (backend work) | Scope inside the D7 ticket | + +## 11. Non-goals + +- Remote/cloud PV storage model — unchanged (already on proper PVs on + customer infrastructure). +- Restart-persistence — unchanged (laptops reboot; data survives). +- **Dataset integrity & score reproducibility** — fingerprint at ingest, + verify at scoring, bind every score to the dataset fingerprint. Genuinely + important for the benchmark product, but its root cause is verifiable + *versioning*, not storage location; it gets its **own RFC** (D13) rather + than blurring this one. +- Cryptographic training (HE/MPC) — rejected (§6.6). Differential privacy / + secure aggregation — future, separate. + +## 12. Rollout sequence + +1. Land this RFC with the §10 decision log. +2. cli#389 (verify-before-✔) and client#368 (flag-gated C) merge on their + own review tracks. +3. Build the **installer leftover-guard** (D3). +4. **Egress-lockdown flip** (D6) proceeds on its own ticket, per fleet. +5. One green **training run on node-local** → flip the default (O1), with + the single-node topology change in release notes. +6. **Scoping & hygiene epic** (D8–D11) as backend#1151 children: + envelope-encrypted weights + crypto-shred; scoped mounts; per-experiment + DB grants; pod-hardening completion; watermarking (D7). +7. **Seal check + guarantee matrix** (D12). +8. Phase 2 when capacity allows: TEE pilot on Azure confidential GPU (D14); + at-rest encryption revisit (D5). + +## 13. Appendix — evidence (refreshed 2026-07-22; line numbers as of that date) + +- `client/scripts/lib/common.sh:386` — `HOST_DATA_DIR="${HOST_DATA_DIR:-$HOME/.tracebloc}"` (v1 cited :329 — drifted) +- `client/scripts/lib/common.sh:392` — `HOST_DATASET_DIR="${HOST_DATASET_DIR:-}"` (v1 cited :335) +- `client/scripts/lib/common.sh:383` — `AGENTS="${AGENTS:-1}"` (default two-node local topology; §5 C1) +- `client/scripts/lib/cluster.sh:28-56` — `_ensure_tracebloc_dirs`: `mkdir -p` with no existing-data guard; `chmod -R 777` scoped to `logs`/`mysql`/`data` subdirs only, `values.yaml` spared +- `client/scripts/lib/cluster.sh:197`, `:355-356` — cluster **reuse** on re-run (`cluster start`; "already exists → Using existing cluster") +- `client/scripts/lib/cluster.sh:312` — `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` (cluster-wide dataset-source mount) +- `client/client/values.yaml` — `networkPolicy.training.{enabled,allowExternalHttps,enforcementProbeHost,clusterCidrs}`, `egressProxy.{enabled,routeWorkloads}`, `egressReachabilityCheck` (lockdown built, ships permissive; §3.5/§8.1) +- `client-runtime/jobs_manager.py` (~:825-885) — `EXPERIMENT_SCRATCH_PATH` emptyDir scratch, `readOnlyRootFilesystem`, read-only shared mounts; legacy-image carve-out (~:77, :829) +- `cli/internal/cli/delete.go` — teardown order; wipe verified before ✔ as of cli#389 +- Validation evidence — client#368 comments (2026-07-22): single-node node-local install on dev; PVCs on `local-path`; real ingest-jobs sharing the data PVC; stop/start survives; delete destroys volume + data; prototype fix `5e4ea45` (second `_ensure_tracebloc_dirs` call site) +- Repro (2026-07-21): `tracebloc delete --force --yes` → `~/.tracebloc` gone, 0 `.ibd` survivors; earlier "survival" = flat/per-release transition artifact + second CLI binary From a936323a5f8f1704cf02d2f950387fef51143e29 Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Wed, 22 Jul 2026 17:38:05 +0200 Subject: [PATCH 3/7] =?UTF-8?q?docs(rfc):=20v2.1=20=E2=80=94=20correct=20w?= =?UTF-8?q?eight-lifecycle=20facts=20against=20code;=20record=20D15?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §3.1/§6.4: verified against tracebloc-client + client-runtime + averaging-service — weights do NOT rest durably in the environment (per-cycle backend download -> pod-scoped scratch -> upload back); durable store is the platform-side averaging share - D8 re-aimed: edge = lifecycle hygiene only; envelope encryption + crypto-shredding applies to the platform-side store (backend/averaging ticket, outside the environment boundary) - O1 decided -> D15: node-local becomes the default for local installs after one green training run on node-local - O3 dissolved by architecture; O2 re-scoped to platform-side retention - Evidence appendix: added weight-lifecycle code anchors Co-Authored-By: Claude Fable 5 --- .../rfcs/0003-storage-and-offboard-hygiene.md | 99 +++++++++++++------ 1 file changed, 68 insertions(+), 31 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index bd80c293..09a25ad5 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -1,8 +1,13 @@ # RFC 0003 — The secure environment: dataset storage, offboard hygiene & the boundary -> **Status: DRAFT v2 — decisions recorded 2026-07-22.** Owner: @LukasWodka. +> **Status: DRAFT v2.1 — decisions recorded 2026-07-22.** Owner: @LukasWodka. > Co-design & validation: @saadqbal. > +> **v2.1 same-day errata:** the §3.1/§6.4 weight-lifecycle claims were +> re-verified against the code — **weights do not rest durably in the +> environment** (per-cycle scratch only; the durable store is +> platform-side). D8 re-aimed accordingly; O1 decided → D15. +> > **v1 → v2.** v1 (2026-07-21) framed two problems — offboard hygiene and > dataset storage — and left §7 as a decision menu. v2 records the decisions > taken (with prototype validation on a real install, client#368), folds in @@ -81,9 +86,18 @@ datasets, and the model provider's model IP. Adversary by adversary: ### 3.1 Where data and weights physically live - MySQL runs **in-cluster**; ingested datasets are tables in the - `training_test_datasets` database. Weight files at rest also live inside - the environment (DB/data volume): an environment runs **tens of - concurrent trainings**, so weights cannot simply be held in memory. + `training_test_datasets` database. +- **Model weights do not rest durably in the environment** (verified + against the code 2026-07-22). Each training pod downloads its current + weights from the backend at cycle start (ZIP transport envelope → a + framework file, e.g. `.pkl`/`.safetensors`), writes them **only into its + own per-experiment scratch** (`EXPERIMENT_SCRATCH_PATH`, an emptyDir that + dies with the pod; legacy images fall back to the image filesystem), and + uploads the result back to the backend after the cycle. The durable, + cross-cycle weight store is **platform-side** (tracebloc's averaging + share: `{edge}_{exp}_{cycle}_weights.pkl` under `SHARE_PATH`) — outside + the environment. N concurrent trainings = N pods, each holding only its + own current file; nothing accumulates in the environment. - The MySQL PersistentVolume uses a **hostPath** under `/tracebloc//mysql`; the local k3d cluster **bind-mounts `~/.tracebloc → /tracebloc`** (`HOST_DATA_DIR`, default @@ -221,11 +235,12 @@ crypto for the volume is not warranted for a local dev tool unless the threat model changes — and the real answer to "protected from a local admin" is §6.5, not filesystem crypto. -**Open (O1):** flip node-local from flag-gated (`TB_STORAGE_MODE`) to -**default** for local installs — the step that actually delivers this RFC -to users. Recommendation: flip after one green end-to-end **training run** -on node-local (the only client#368-checklist item not yet exercised; it was -blocked by an orthogonal dev auth issue, not by storage). +**Decided (D15, 2026-07-22):** node-local flips from flag-gated +(`TB_STORAGE_MODE`) to **default** for local installs — the step that +actually delivers this RFC to users — after one green end-to-end +**training run** on node-local (the only client#368-checklist item not yet +exercised; it was blocked by an orthogonal dev auth issue, not by +storage). The single-node topology change (C1) goes in the release notes. ## 6. Model IP protection — two-sided trust (new in v2) @@ -261,20 +276,30 @@ credible commercially. (`EXPERIMENT_SCRATCH_PATH`, an `emptyDir`) and dies with the pod — already true on new-architecture images; close the legacy-image carve-out (D11). -3. **At rest: envelope encryption + crypto-shredding.** An environment runs - tens of concurrent trainings, so weight files cannot live in RAM — they - rest in the environment's DB / data volume. Therefore: store weight - blobs as **ciphertext**; the per-experiment data-encryption key - (~32 bytes) is issued by the backend at cycle start and held **only in - memory**, never persisted inside the environment. N concurrent - experiments cost N small keys in RAM — not N weight files. On experiment - completion (or offboard, or revocation) the key is discarded and the - backend refuses re-issue: every at-rest copy — including disk snapshots - and backups — becomes unrecoverable. **Crypto-shredding is deletion that - is instant, size-independent, and verifiable**, and it upgrades what - "delete" means for weights on top of §4/§5. -4. **Lifecycle:** TTL finished Jobs (`ttlSecondsAfterFinished`); offboard - already prunes images (§3.4). +3. **At rest in the environment — corrected in v2.1 after code + verification.** The edge has **no durable weight store to encrypt**: + weights arrive per cycle, exist only in the pod's own scratch and + process memory, and the durable store is platform-side (§3.1). v2's + proposal to envelope-encrypt an in-environment weight DB targeted a + store that does not exist — the architecture is already stronger than + that proposal assumed. What remains on the edge is **lifecycle + hygiene**: `ttlSecondsAfterFinished` on finished Jobs (an emptyDir + survives until its pod object is deleted), closing the legacy + image-filesystem fallback (D11), offboard's existing image prune + (§3.4), and — optional, marginal against the §6.2 ceiling — encrypting + the scratch file with a per-pod in-memory key. +4. **The platform-side store is where envelope encryption + + crypto-shredding apply.** The durable `{edge}_{exp}_{cycle}_weights.pkl` + files under the averaging share are opaque bytes: encrypt on receipt, + decrypt in memory for averaging, per-experiment key in the backend's + KMS, discard the key on experiment deletion → every copy (including + backups/snapshots) becomes unrecoverable, instantly and verifiably. + Feasible as a contained change at the existing weight-file I/O choke + points (deserialization is already confined via `safe_unpickle` on both + edge and averaging). This protects vendor IP **on tracebloc + infrastructure** and makes experiment deletion verifiable — but it is a + **platform ticket (backend/averaging), not part of the environment + boundary** this RFC defines. Stated ceiling: root can still capture plaintext *during* execution — 6.4 stops casual and after-the-fact access, not a determined owner. That gap @@ -388,23 +413,26 @@ seal check. | D5 | At-rest encryption (§7.4) | **Phase 2**; recommend host FDE; keep wording honest | | D6 | Egress lockdown flip | **Separate ticket**; golden-box claim gated on it (§8.1) | | D7 | Model IP — now | **Watermarking + audit** (§6.3) | -| D8 | Weights at rest | **Envelope encryption + crypto-shredding — keys in memory, not weight files** (§6.4) | +| D8 | Weights at rest | **Edge: nothing rests by architecture (verified) — lifecycle hygiene only; envelope encryption + crypto-shredding re-aimed at the platform-side store** (§6.4) | | D9 | Dataset scoping | **Scoped mounts → per-dataset PVCs under C** (§7.1) | | D10 | DB scoping | **Per-experiment DB credentials, table-scoped grants** (§7.2) | | D11 | Pod hardening | **Close legacy carve-out; no SA token; read-only floor** (§7.3) | | D12 | Conformance | **Guarantee matrix + seal check + verify k3d enforcement** (§8.2–8.4) | | D13 | Score/tamper integrity | **Separate RFC** — versioning+integrity root cause, not storage (§11) | | D14 | Confidential compute | **Phase 2 — deferred for capacity**; Azure confidential-GPU pilot first; tiering as bridge (§6.5) | +| D15 | Node-local default (was O1) | **Flip node-local to default for local installs** after one green training run on node-local; single-node topology change in release notes | Open items: | # | Question | Recommendation | |---|---|---| -| O1 | Flip node-local to **default** for local installs (also flips default local topology to single-node) | Yes — after one green training run on node-local | -| O2 | Weight retention: exactly when does crypto-shred fire (experiment completion? grace window? resume/audit needs?) | Shred at completion + configurable grace window | -| O3 | DEK custody: backend-issued per cycle vs in-environment key service | Backend-issued — revocation/offboard instant; the environment already requires backend egress | +| O2 | Platform-side weight retention: when does crypto-shred fire on the averaging-share store (experiment completion? grace window? audit needs?) | Shred at completion + configurable grace window | | O4 | Watermarking mechanics & owner (backend work) | Scope inside the D7 ticket | +Resolved since v2: **O1 → D15** (decided). **O3** dissolved by the +verified architecture — there is no in-environment at-rest weight store +needing key custody; platform-side keys live in the backend's KMS. + ## 11. Non-goals - Remote/cloud PV storage model — unchanged (already on proper PVs on @@ -425,11 +453,12 @@ Open items: own review tracks. 3. Build the **installer leftover-guard** (D3). 4. **Egress-lockdown flip** (D6) proceeds on its own ticket, per fleet. -5. One green **training run on node-local** → flip the default (O1), with +5. One green **training run on node-local** → flip the default (D15), with the single-node topology change in release notes. -6. **Scoping & hygiene epic** (D8–D11) as backend#1151 children: - envelope-encrypted weights + crypto-shred; scoped mounts; per-experiment - DB grants; pod-hardening completion; watermarking (D7). +6. **Scoping & hygiene epic** (D8–D11) as backend#1151 children: job-TTL + + scratch hygiene on the edge; platform-side weight-store encryption + + crypto-shred (backend/averaging); scoped mounts; per-experiment DB + grants; pod-hardening completion; watermarking (D7). 7. **Seal check + guarantee matrix** (D12). 8. Phase 2 when capacity allows: TEE pilot on Azure confidential GPU (D14); at-rest encryption revisit (D5). @@ -444,6 +473,14 @@ Open items: - `client/scripts/lib/cluster.sh:312` — `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` (cluster-wide dataset-source mount) - `client/client/values.yaml` — `networkPolicy.training.{enabled,allowExternalHttps,enforcementProbeHost,clusterCidrs}`, `egressProxy.{enabled,routeWorkloads}`, `egressReachabilityCheck` (lockdown built, ships permissive; §3.5/§8.1) - `client-runtime/jobs_manager.py` (~:825-885) — `EXPERIMENT_SCRATCH_PATH` emptyDir scratch, `readOnlyRootFilesystem`, read-only shared mounts; legacy-image carve-out (~:77, :829) +- `tracebloc-client/core/weights/base.py:117-146, :340` — per-cycle weight + download (backend → ZIP envelope → `{scratch}/{exp}_{model}_weights.`) + and upload back to the backend; stale-sibling cleanup between formats +- `tracebloc-client/core/utils/general.py:21-33` — `get_experiment_path()`: + `EXPERIMENT_SCRATCH_PATH` (emptyDir) or legacy image-filesystem fallback +- `averaging-service/service/safe_unpickle.py` — the durable, platform-side + weight store: `{edge}_{exp}_{cycle}_weights.pkl` under `SHARE_PATH`; + deserialization confined on both sides (edge: backend#946) - `cli/internal/cli/delete.go` — teardown order; wipe verified before ✔ as of cli#389 - Validation evidence — client#368 comments (2026-07-22): single-node node-local install on dev; PVCs on `local-path`; real ingest-jobs sharing the data PVC; stop/start survives; delete destroys volume + data; prototype fix `5e4ea45` (second `_ensure_tracebloc_dirs` call site) - Repro (2026-07-21): `tracebloc delete --force --yes` → `~/.tracebloc` gone, 0 `.ibd` survivors; earlier "survival" = flat/per-release transition artifact + second CLI binary From 7c6bccf76a679ae5cbdd0815361722e07b9ff31b Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Wed, 22 Jul 2026 18:37:18 +0200 Subject: [PATCH 4/7] =?UTF-8?q?docs(rfc):=20v2.2=20=E2=80=94=20status=20DE?= =?UTF-8?q?CIDED;=20cross-link=20execution=20tickets=20in=20=C2=A710/?= =?UTF-8?q?=C2=A712?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../rfcs/0003-storage-and-offboard-hygiene.md | 56 ++++++++++--------- 1 file changed, 31 insertions(+), 25 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index 09a25ad5..d52159c3 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -1,6 +1,7 @@ # RFC 0003 — The secure environment: dataset storage, offboard hygiene & the boundary -> **Status: DRAFT v2.1 — decisions recorded 2026-07-22.** Owner: @LukasWodka. +> **Status: DECIDED — v2.2, decisions D1–D15 locked 2026-07-22; execution +> tickets filed and cross-linked in §10/§12.** Owner: @LukasWodka. > Co-design & validation: @saadqbal. > > **v2.1 same-day errata:** the §3.1/§6.4 weight-lifecycle claims were @@ -406,28 +407,28 @@ seal check. | # | Decision (v1 ref) | Outcome | |---|---|---| -| D1 | Local storage target (§7.2) | **Option C — node-local**, validated in client#368 | +| D1 | Local storage target (§7.2) | **Option C — node-local**, validated in client#368 → client#367 | | D2 | Drop `chmod 777` (§7.3) | **Yes — by construction with C** (load-bearing on hostpath, cannot ship separately) | -| D3 | Offboard clean-slate (§7.1) | **Leftover-guard yes; heavy multi-path wipe no; verify-before-✔** (cli#389) | +| D3 | Offboard clean-slate (§7.1) | **Leftover-guard yes; heavy multi-path wipe no; verify-before-✔** (guard: client#376; verify: cli#389) | | D4 | Migration (§7.5) | **No data-copier**; guard prevents stranding; installs move to C on delete+reinstall | | D5 | At-rest encryption (§7.4) | **Phase 2**; recommend host FDE; keep wording honest | -| D6 | Egress lockdown flip | **Separate ticket**; golden-box claim gated on it (§8.1) | -| D7 | Model IP — now | **Watermarking + audit** (§6.3) | -| D8 | Weights at rest | **Edge: nothing rests by architecture (verified) — lifecycle hygiene only; envelope encryption + crypto-shredding re-aimed at the platform-side store** (§6.4) | -| D9 | Dataset scoping | **Scoped mounts → per-dataset PVCs under C** (§7.1) | -| D10 | DB scoping | **Per-experiment DB credentials, table-scoped grants** (§7.2) | -| D11 | Pod hardening | **Close legacy carve-out; no SA token; read-only floor** (§7.3) | -| D12 | Conformance | **Guarantee matrix + seal check + verify k3d enforcement** (§8.2–8.4) | -| D13 | Score/tamper integrity | **Separate RFC** — versioning+integrity root cause, not storage (§11) | +| D6 | Egress lockdown flip | **client-runtime#199**; golden-box claim gated on it (§8.1) | +| D7 | Model IP — now | **Watermarking + audit** (§6.3) → backend#1183 | +| D8 | Weights at rest | **Edge: nothing rests by architecture (verified) — lifecycle hygiene (client-runtime#200); platform-side envelope encryption + crypto-shredding (backend#1182)** (§6.4) | +| D9 | Dataset scoping | **Scoped mounts → per-dataset PVCs under C** (§7.1) → client-runtime#203 | +| D10 | DB scoping | **Per-experiment DB credentials, table-scoped grants** (§7.2) → backend#1181 | +| D11 | Pod hardening | **Close legacy carve-out; no SA token; read-only floor** (§7.3) → client-runtime#201 + #202 | +| D12 | Conformance | **Guarantee matrix + seal check + verify k3d enforcement** (§8.2–8.4) → backend#1184 + cli#393 | +| D13 | Score/tamper integrity | **Separate RFC** — versioning+integrity root cause, not storage (§11) → backend#1185 | | D14 | Confidential compute | **Phase 2 — deferred for capacity**; Azure confidential-GPU pilot first; tiering as bridge (§6.5) | -| D15 | Node-local default (was O1) | **Flip node-local to default for local installs** after one green training run on node-local; single-node topology change in release notes | +| D15 | Node-local default (was O1) | **Flip node-local to default for local installs** after one green training run on node-local (gate: backend#1180); checklist on client#367 | Open items: | # | Question | Recommendation | |---|---|---| -| O2 | Platform-side weight retention: when does crypto-shred fire on the averaging-share store (experiment completion? grace window? audit needs?) | Shred at completion + configurable grace window | -| O4 | Watermarking mechanics & owner (backend work) | Scope inside the D7 ticket | +| O2 | Platform-side weight retention: when does crypto-shred fire on the averaging-share store (experiment completion? grace window? audit needs?) | Shred at completion + configurable grace window — decided inside backend#1182 | +| O4 | Watermarking mechanics & owner (backend work) | Scope inside backend#1183 | Resolved since v2: **O1 → D15** (decided). **O3** dissolved by the verified architecture — there is no in-environment at-rest weight store @@ -448,20 +449,25 @@ needing key custody; platform-side keys live in the backend's KMS. ## 12. Rollout sequence -1. Land this RFC with the §10 decision log. +1. Land this RFC with the §10 decision log. Execution epic: backend#1151. 2. cli#389 (verify-before-✔) and client#368 (flag-gated C) merge on their own review tracks. -3. Build the **installer leftover-guard** (D3). -4. **Egress-lockdown flip** (D6) proceeds on its own ticket, per fleet. -5. One green **training run on node-local** → flip the default (D15), with - the single-node topology change in release notes. -6. **Scoping & hygiene epic** (D8–D11) as backend#1151 children: job-TTL + - scratch hygiene on the edge; platform-side weight-store encryption + - crypto-shred (backend/averaging); scoped mounts; per-experiment DB - grants; pod-hardening completion; watermarking (D7). -7. **Seal check + guarantee matrix** (D12). +3. Build the **installer leftover-guard** (D3 — client#376). +4. **Egress-lockdown flip** (D6 — client-runtime#199), per fleet. +5. One green **training run on node-local** (gate: backend#1180) → flip the + default (D15 — checklist on client#367), single-node topology change in + release notes. +6. **Scoping & hygiene** (D8–D11) as backend#1151 children: job-TTL + + scratch hygiene (client-runtime#200); platform-side weight-store + encryption + crypto-shred (backend#1182); scoped mounts + (client-runtime#203); per-experiment DB grants (backend#1181); + pod-hardening completion (client-runtime#201, #202); watermarking + (D7 — backend#1183). +7. **Seal check + guarantee matrix** (D12 — backend#1184, CLI surfacing + cli#393). 8. Phase 2 when capacity allows: TEE pilot on Azure confidential GPU (D14); - at-rest encryption revisit (D5). + at-rest encryption revisit (D5). Docs/messaging: backend#1185 (integrity + RFC), backend#1186 (§9 alignment). ## 13. Appendix — evidence (refreshed 2026-07-22; line numbers as of that date) From 0909bceaa73caa15a7365eabea72b6e3ce98b87d Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Thu, 23 Jul 2026 16:02:59 +0200 Subject: [PATCH 5/7] =?UTF-8?q?docs(rfc):=20v2.3=20=E2=80=94=20per-dataset?= =?UTF-8?q?=20immutable=20tables=20+=20grant-scoped=20view=20isolation;=20?= =?UTF-8?q?split=20composition=20to=20data-spaces=20RFC?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 --- .../rfcs/0003-storage-and-offboard-hygiene.md | 111 +++++++++++++++++- 1 file changed, 108 insertions(+), 3 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index d52159c3..90022750 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -1,9 +1,19 @@ # RFC 0003 — The secure environment: dataset storage, offboard hygiene & the boundary -> **Status: DECIDED — v2.2, decisions D1–D15 locked 2026-07-22; execution -> tickets filed and cross-linked in §10/§12.** Owner: @LukasWodka. +> **Status: DECIDED — v2.3; decisions D1–D20 locked (D1–D15 2026-07-22; +> D16–D20 2026-07-23). Execution tickets filed and cross-linked in §10/§12; +> D16–D20 (per-dataset isolation) are decided but not yet ticketed, with +> open items O5–O7 out for reviewer feedback.** Owner: @LukasWodka. > Co-design & validation: @saadqbal. > +> **v2.2 → v2.3.** Adds the **per-dataset data-isolation model** (new §7bis, +> D16–D20): one immutable `ds_` table per ingestion, correction +> by delete (no version chain), and isolation enforced by a grant-scoped +> definer-view instead of a `WHERE ingestor_id` filter — and **splits the +> multi-org composition layer** (data spaces, federated dataset combination) +> into its own RFC, "Data spaces & federated dataset composition" (drafting +> in `backend`), which builds on these primitives. +> > **v2.1 same-day errata:** the §3.1/§6.4 weight-lifecycle claims were > re-verified against the code — **weights do not rest durably in the > environment** (per-cycle scratch only; the durable store is @@ -22,7 +32,9 @@ > > Related: RFC-0001 (auth & client provisioning), RFC-0002 (data ingest flow > & terminology), terminology source-of-truth (v2 `TERMINOLOGY.md`, docs -> `main`). Design ticket: backend#1151. Storage prototype: client#368 +> `main`). Forward: **Data spaces & federated dataset composition** RFC +> (drafting in `backend`) builds on the §7bis isolation primitives. Design +> ticket: backend#1151. Storage prototype: client#368 > (flag-gated, draft). Offboard honesty fix: cli#389. Spans `cli`, `client`, > `client-runtime`, `backend`. @@ -351,6 +363,91 @@ job must reach exactly its own dataset and its own artifacts, nothing else. spawned jobs; keep `readOnlyRootFilesystem` + read-only shared mounts as the floor. +## 7bis. Per-dataset isolation — immutable tables & grant-scoped views (new in v2.3, DECIDED) + +§7 scopes *access* to a dataset (mounts, DB grants, pod hardening). This +section sharpens *what a dataset physically is*, so the scoping rests on +structure rather than on a filter a job could sidestep. Today +tabular/time-series ingests land as rows in a **shared** table tagged by +`ingestor_id`, and every read narrows with `WHERE ingestor_id IN (…)` (§3.1; +tracebloc-client `core/utils/database.py:243-361`). A `WHERE` clause is a +convention, not a wall — one dropped predicate or one `SELECT *` and a job +reads its neighbours. The model below turns the convention into physics. + +1. **One immutable table per ingestion (D16).** Every ingest creates a + **brand-new physical table** named `ds_`; the ingestor id + *is* the physical identity. The dataset name the user chose is a + **user-facing label only** (metadata), never the table name. Ten ingests + of the same name and schema produce **ten tables**, not ten appends into + one. Isolation stops being a runtime filter and becomes a property of the + schema: a job that can only name `ds_` cannot phrase a query + that reaches another dataset. + +2. **Immutable — no append, no in-place edit (D17; enforced by D19).** A + table, once written by an ingestion, is never appended to or edited. + Re-ingesting the same name+schema **creates a new table**; it does not + reuse the old one. This is enforced by **removing the ingestor's + reuse/append path**: `data-ingestors/tracebloc_ingestor/database.py:309-357` + today returns/reflects an existing table when the name and feature schema + match (the "return existing table if already created" / "check if table + exists in database" branch) — that branch is deleted, and ingestion + always creates a fresh `ds_`. The schema-drift guard that + lives in that branch (the "stale table from an earlier ingestion" error) + becomes moot once every ingestion owns a private table name. + +3. **Correction = delete, not versioning (D17).** There is **no version + chain**. A wrong ingestion is corrected by **dropping its table** and + re-ingesting — a new id, a new `ds_`, the bad one gone — + which keeps the model honest with the RFC's "delete means gone" stance + (§4/§5) instead of accreting history. **Referential cleanup is part of + the drop:** removing a table must invalidate every view and every + data-space reference pointing at it (O7). A definer-view left dangling + over a dropped base table is both a broken read path and a latent leak. + +4. **Isolation = a grant-scoped access handle, never raw-table access + (D18).** A training pod is never granted access to base tables at large. + For MySQL it is granted `SELECT` **only** on a **definer-rights view** + (degenerate case: on its single `ds_` table) — never on any + other dataset's base table. Because the grant does not name the + neighbours, a malicious or buggy `SELECT * FROM ds_` **cannot + even reference** a table it holds no grant on; it fails at + permission-check time, not at a `WHERE` clause the job controls. + + *View vs. table, for the reader:* a **table** stores rows on disk; a + **view** is a stored `SELECT` exposing only chosen rows/columns of one or + more tables, holding no data of its own; a **definer-rights view** + executes with the privileges of the user that *created* it, so you can + `GRANT SELECT` on the view to the training user **without** granting that + user any privilege on the underlying base tables. The view becomes the + only name the job holds; the base tables stay invisible. + + For **file/image** datasets the analogue is the **per-dataset PVC / + `subPath` mount** already chosen as the D9 end state (client-runtime#203) + — same principle, different substrate: the handle is a mount, not a view. + And this **composes with the per-experiment DB grant (D10, + backend#1181)**: that grant now targets the experiment's **view/table + handle** rather than the shared table, and still excludes the platform + metadata DB. §7 and §7bis are the same wall from two sides — §7 scopes + the grant, §7bis scopes the object the grant points at. + +5. **Grandfather — new ingests only (D20).** The model applies to **new + ingestions**. Existing shared multi-ingestor tables **stay as they are** + — no migration, no backfill into per-ingestion tables — mirroring the + storage-migration stance (D4: no data-copier; installs move forward on + the natural boundary, here the next ingest). The client read path + therefore keeps tolerating both shapes through the transition (O6). + +**Scope boundary — what this section does *not* own.** The primitive here is +deliberately **single-environment**: it makes one dataset an isolated, +access-scoped object inside one secure environment. The **multi-org +composition** layer — *data spaces*, federated combination of datasets +across environments, the schema/compatibility engine that decides when two +datasets may train together, and cross-org authorization — is carved into +its **own RFC, "Data spaces & federated dataset composition"** (drafting in +`backend`), which **builds on** these primitives (a data space is composed +out of exactly these access-scoped units). Ownership split: **this RFC owns +the isolation primitive; the data-spaces RFC owns the composition over it.** + ## 8. Boundary enforcement & conformance (new in v2) ### 8.1 Egress lockdown — flip it (D6, separate ticket) @@ -422,6 +519,11 @@ seal check. | D13 | Score/tamper integrity | **Separate RFC** — versioning+integrity root cause, not storage (§11) → backend#1185 | | D14 | Confidential compute | **Phase 2 — deferred for capacity**; Azure confidential-GPU pilot first; tiering as bridge (§6.5) | | D15 | Node-local default (was O1) | **Flip node-local to default for local installs** after one green training run on node-local (gate: backend#1180); checklist on client#367 | +| D16 | Table-per-ingestion (§7bis) | **One immutable `ds_` table per ingest** — the ingestor id is the physical identity; the user's dataset name is a label only (same name+schema → N tables) | +| D17 | Immutability + correction (§7bis) | **No append, no in-place edit; re-ingest = new table; correction = drop + re-ingest, no version chain** — the drop must cascade to dependent views / data-space refs (O7) | +| D18 | Isolation enforcement (§7bis) | **Grant-scoped access handle, not raw-table access** — MySQL: `SELECT` on a definer-rights view (or the single `ds_` table) only; files: per-dataset PVC/`subPath` (D9); composes with the per-experiment grant (D10, backend#1181) | +| D19 | Ingestor append-disable (§7bis) | **Remove the reuse/append path** — `data-ingestors/tracebloc_ingestor/database.py:309-357` (reuse-existing-table branch) deleted; ingestion always creates a fresh table | +| D20 | Grandfather (§7bis) | **New ingests only; existing shared multi-ingestor tables stay as-is; no migration/backfill** (mirrors D4) | Open items: @@ -429,6 +531,9 @@ Open items: |---|---|---| | O2 | Platform-side weight retention: when does crypto-shred fire on the averaging-share store (experiment completion? grace window? audit needs?) | Shred at completion + configurable grace window — decided inside backend#1182 | | O4 | Watermarking mechanics & owner (backend work) | Scope inside backend#1183 | +| O5 | Storage abstraction beyond MySQL — should the RFC define "a dataset = an isolated, access-scoped unit" with a per-backend physical form (MySQL table + definer-view; file/folder PVC; future other DBs) rather than binding the concept to MySQL? | Open (reviewer feedback) — lean toward a backend-neutral definition; confirm the per-substrate forms | +| O6 | View mechanics at scale — many tables + views (UNION / indexing / perf), and how the training read path moves from the client-built `WHERE ingestor_id IN (…)` (tracebloc-client `core/utils/database.py:243-361`) to reading a backend-provisioned view/table handle | Open (reviewer feedback) — needs a read-path / perf design pass | +| O7 | Delete / referential-cleanup semantics — cascade a table drop to its views and any data-space references (D17) | Open (reviewer feedback) — resolve jointly with the data-spaces RFC (references cross the boundary) | Resolved since v2: **O1 → D15** (decided). **O3** dissolved by the verified architecture — there is no in-environment at-rest weight store From e05a81a20a3ad9c9bf646b29286bc6cd87667ceb Mon Sep 17 00:00:00 2001 From: Asad Iqbal Date: Fri, 24 Jul 2026 14:23:56 +0500 Subject: [PATCH 6/7] docs(rfc): C1 pins single-node via AGENTS=0 AND SERVERS=1, not just AGENTS=0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §5 C1 and the appendix said node-local forces AGENTS=0; the prototype (client#368) had to force SERVERS=1 too — k3s server nodes are schedulable, so SERVERS>1 still yields multiple nodes a Job could land on away from the local-path volume. Match the doc to the shipped code. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0003-storage-and-offboard-hygiene.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index 90022750..68b36f52 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -229,7 +229,11 @@ second `_ensure_tracebloc_dirs` call site still creating empty 777 dirs). **Constraint C1 — single-node.** `local-path` is RWO/WaitForFirstConsumer, and spawned Jobs must land on the node that holds the volume, so node-local -forces `AGENTS=0`. Since `AGENTS` defaults to `1` today, **flipping the +pins the cluster to a **single node** — it forces both `AGENTS=0` **and** +`SERVERS=1`. Dropping the agents is not enough on its own: k3s server nodes +are schedulable, so `SERVERS>1` would still yield multiple nodes a Job could +land on away from the volume; the prototype (client#368) had to pin both. +Since the defaults are `SERVERS=1`/`AGENTS=1` today, **flipping the storage default also changes the default local topology from two nodes to one** — call this out in release notes. (k3s agents on one Docker host provide no real isolation, so nothing of value is lost.) @@ -578,7 +582,7 @@ needing key custody; platform-side keys live in the backend's KMS. - `client/scripts/lib/common.sh:386` — `HOST_DATA_DIR="${HOST_DATA_DIR:-$HOME/.tracebloc}"` (v1 cited :329 — drifted) - `client/scripts/lib/common.sh:392` — `HOST_DATASET_DIR="${HOST_DATASET_DIR:-}"` (v1 cited :335) -- `client/scripts/lib/common.sh:383` — `AGENTS="${AGENTS:-1}"` (default two-node local topology; §5 C1) +- `client/scripts/lib/common.sh` (post-#368) — `SERVERS="${SERVERS:-1}"`/`AGENTS="${AGENTS:-1}"` (default two-node local topology), and `TB_STORAGE_MODE=node-local` forces **both** `AGENTS=0` **and** `SERVERS=1` for the single-node pin (§5 C1) - `client/scripts/lib/cluster.sh:28-56` — `_ensure_tracebloc_dirs`: `mkdir -p` with no existing-data guard; `chmod -R 777` scoped to `logs`/`mysql`/`data` subdirs only, `values.yaml` spared - `client/scripts/lib/cluster.sh:197`, `:355-356` — cluster **reuse** on re-run (`cluster start`; "already exists → Using existing cluster") - `client/scripts/lib/cluster.sh:312` — `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` (cluster-wide dataset-source mount) From 3a56fcc6c7e601a38d332e8b3edf62f2bcc61952 Mon Sep 17 00:00:00 2001 From: shujaat hasan Date: Fri, 24 Jul 2026 11:24:09 +0200 Subject: [PATCH 7/7] docs(rfc): fix stale code citations in RFC-0003 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Repo rename: the training-client code is now in `tracebloc-engine`, not `tracebloc-client` (§7bis, O6, appendix). Verified the cited paths exist there: core/utils/database.py:243 (get_sql_query_and_params), core/utils/general.py:21-33 (get_experiment_path), core/weights/base.py. - D19 line drift: the ingestor reuse/append branch spans 275-357, not 309-357 — the old range started after the "return existing table if already created" reuse check (line 275) that D19 actually removes. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0003-storage-and-offboard-hygiene.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/rfcs/0003-storage-and-offboard-hygiene.md b/docs/rfcs/0003-storage-and-offboard-hygiene.md index 68b36f52..eeb381dd 100644 --- a/docs/rfcs/0003-storage-and-offboard-hygiene.md +++ b/docs/rfcs/0003-storage-and-offboard-hygiene.md @@ -374,7 +374,7 @@ section sharpens *what a dataset physically is*, so the scoping rests on structure rather than on a filter a job could sidestep. Today tabular/time-series ingests land as rows in a **shared** table tagged by `ingestor_id`, and every read narrows with `WHERE ingestor_id IN (…)` (§3.1; -tracebloc-client `core/utils/database.py:243-361`). A `WHERE` clause is a +tracebloc-engine `core/utils/database.py:243-361`). A `WHERE` clause is a convention, not a wall — one dropped predicate or one `SELECT *` and a job reads its neighbours. The model below turns the convention into physics. @@ -391,7 +391,7 @@ reads its neighbours. The model below turns the convention into physics. table, once written by an ingestion, is never appended to or edited. Re-ingesting the same name+schema **creates a new table**; it does not reuse the old one. This is enforced by **removing the ingestor's - reuse/append path**: `data-ingestors/tracebloc_ingestor/database.py:309-357` + reuse/append path**: `data-ingestors/tracebloc_ingestor/database.py:275-357` today returns/reflects an existing table when the name and feature schema match (the "return existing table if already created" / "check if table exists in database" branch) — that branch is deleted, and ingestion @@ -526,7 +526,7 @@ seal check. | D16 | Table-per-ingestion (§7bis) | **One immutable `ds_` table per ingest** — the ingestor id is the physical identity; the user's dataset name is a label only (same name+schema → N tables) | | D17 | Immutability + correction (§7bis) | **No append, no in-place edit; re-ingest = new table; correction = drop + re-ingest, no version chain** — the drop must cascade to dependent views / data-space refs (O7) | | D18 | Isolation enforcement (§7bis) | **Grant-scoped access handle, not raw-table access** — MySQL: `SELECT` on a definer-rights view (or the single `ds_` table) only; files: per-dataset PVC/`subPath` (D9); composes with the per-experiment grant (D10, backend#1181) | -| D19 | Ingestor append-disable (§7bis) | **Remove the reuse/append path** — `data-ingestors/tracebloc_ingestor/database.py:309-357` (reuse-existing-table branch) deleted; ingestion always creates a fresh table | +| D19 | Ingestor append-disable (§7bis) | **Remove the reuse/append path** — `data-ingestors/tracebloc_ingestor/database.py:275-357` (reuse-existing-table branch) deleted; ingestion always creates a fresh table | | D20 | Grandfather (§7bis) | **New ingests only; existing shared multi-ingestor tables stay as-is; no migration/backfill** (mirrors D4) | Open items: @@ -536,7 +536,7 @@ Open items: | O2 | Platform-side weight retention: when does crypto-shred fire on the averaging-share store (experiment completion? grace window? audit needs?) | Shred at completion + configurable grace window — decided inside backend#1182 | | O4 | Watermarking mechanics & owner (backend work) | Scope inside backend#1183 | | O5 | Storage abstraction beyond MySQL — should the RFC define "a dataset = an isolated, access-scoped unit" with a per-backend physical form (MySQL table + definer-view; file/folder PVC; future other DBs) rather than binding the concept to MySQL? | Open (reviewer feedback) — lean toward a backend-neutral definition; confirm the per-substrate forms | -| O6 | View mechanics at scale — many tables + views (UNION / indexing / perf), and how the training read path moves from the client-built `WHERE ingestor_id IN (…)` (tracebloc-client `core/utils/database.py:243-361`) to reading a backend-provisioned view/table handle | Open (reviewer feedback) — needs a read-path / perf design pass | +| O6 | View mechanics at scale — many tables + views (UNION / indexing / perf), and how the training read path moves from the client-built `WHERE ingestor_id IN (…)` (tracebloc-engine `core/utils/database.py:243-361`) to reading a backend-provisioned view/table handle | Open (reviewer feedback) — needs a read-path / perf design pass | | O7 | Delete / referential-cleanup semantics — cascade a table drop to its views and any data-space references (D17) | Open (reviewer feedback) — resolve jointly with the data-spaces RFC (references cross the boundary) | Resolved since v2: **O1 → D15** (decided). **O3** dissolved by the @@ -588,10 +588,10 @@ needing key custody; platform-side keys live in the backend's KMS. - `client/scripts/lib/cluster.sh:312` — `-v "${HOST_DATASET_DIR}:/tracebloc-data@all"` (cluster-wide dataset-source mount) - `client/client/values.yaml` — `networkPolicy.training.{enabled,allowExternalHttps,enforcementProbeHost,clusterCidrs}`, `egressProxy.{enabled,routeWorkloads}`, `egressReachabilityCheck` (lockdown built, ships permissive; §3.5/§8.1) - `client-runtime/jobs_manager.py` (~:825-885) — `EXPERIMENT_SCRATCH_PATH` emptyDir scratch, `readOnlyRootFilesystem`, read-only shared mounts; legacy-image carve-out (~:77, :829) -- `tracebloc-client/core/weights/base.py:117-146, :340` — per-cycle weight +- `tracebloc-engine/core/weights/base.py:117-146, :340` — per-cycle weight download (backend → ZIP envelope → `{scratch}/{exp}_{model}_weights.`) and upload back to the backend; stale-sibling cleanup between formats -- `tracebloc-client/core/utils/general.py:21-33` — `get_experiment_path()`: +- `tracebloc-engine/core/utils/general.py:21-33` — `get_experiment_path()`: `EXPERIMENT_SCRATCH_PATH` (emptyDir) or legacy image-filesystem fallback - `averaging-service/service/safe_unpickle.py` — the durable, platform-side weight store: `{edge}_{exp}_{cycle}_weights.pkl` under `SHARE_PATH`;