docs(rfc): RFC-0003 — dataset storage & offboard hygiene (draft)

[LukasWodka]

Lands the drafted RFC-0003 — Dataset storage & offboard hygiene in cli/docs/rfcs/ (alongside 0001 and 0002), so it's version-controlled and reviewable.

• Status: DRAFT for discussion — concept only, no code proposed to merge.
• Companion to the design ticket backend#1151.
• Two problems, one root cause: tb delete should leave a true clean slate (Problem A), and "your data stays within the secure environment" should be literally true — today datasets are world-readable host files bind-mounted into k3d, engineered to survive cluster deletion (Problem B).
• Storage options (status quo / managed volume / node-local / +encryption) and the open decisions are in §7 — pending Lukas + Asad.

This PR just commits the doc; the decisions it raises are tracked in backend#1151.

---

Note

Low Risk
Documentation-only RFC; no runtime or deployment behavior changes in this PR.

Overview
Addsdocs/rfcs/0003-storage-and-offboard-hygiene.md as the decided RFC-0003 (v2.3), versioned next to RFC-0001/0002. The PR is documentation only; execution is tracked on backend#1151 and linked tickets in §10/§12.

The doc locks decisions D1–D20 (status DECIDED, not an open §7 menu): node-local storage (Option C) for local installs, offboard leftover-guard and verify-before-✔ (no heavy multi-path wipe), no data-copier migration, egress lockdown flip as a gate for the “golden box” claim, in-cluster scoping (mounts, per-experiment DB grants, pod hardening), and model-IP stance (watermark/audit, edge lifecycle hygiene, platform-side envelope encryption; TEE phase 2).

v2.3 introduces §7bis and D16–D20: per-ingestion immutable ds_<ingestor_id> tables, no append/reuse in the ingestor, correction by drop + re-ingest, isolation via grant-scoped definer views (not WHERE ingestor_id), grandfathering existing shared tables; multi-org data spaces are explicitly split to a separate RFC in backend.

v2.1 errata in the same file corrects weight lifecycle: no durable in-environment weight store (D8/O1→D15 node-local default after one green training run).

^{Reviewed by Cursor Bugbot for commit 3a56fcc. Bugbot is set up for automated code reviews on this repo. Configure here.}

---

Update 2026-07-22 — v2 pushed (afe9cd1). See §10 decision log.

[saadqbal]

§7 decisions — Asad's position

Talked this through and confirmed a couple of things against the current client/cli code. The requirement that drives everything: deleting the cluster must destroy the data, and the data must not sit as a browsable folder in ~/.tracebloc. That reframes 7.2 and simplifies the rest.

7.2 — Local storage: Option C (node-local). This is the one that actually matches the goal. Data lives in the k3d node's own storage (k3s local-path), so k3d cluster delete destroys it by construction, and it's no longer a host bind-mount anyone can browse. It also auto-solves Problem A — once nothing outlives the cluster on the host, offboard has no legacy paths to chase.

The objection to C was "loses survive-delete+recreate." I checked: the installer reuses an existing cluster on re-run (cluster.sh "already exists" → cluster start), it doesn't recreate on upgrade. So in-place upgrades keep the data; only an explicit cluster delete wipes it — exactly what we want. Net: C costs us nothing on the normal upgrade path.

7.3 — Drop chmod 777: yes, now. Independent of C. Note the code already scopes 777 to the data/logs/mysql subdirs (not all of ~/.tracebloc, and not values.yaml) — the RFC's evidence section is stale on this, worth refreshing.

7.1 — Scoped verified wipe + installer leftover-guard: keep the guard, the wipe mostly falls out of C. The leftover-guard (detect non-empty data → stop and ask reuse/wipe/new-dir) is still worth it as a safety net and doubles as the migration prompt (see 7.5). The elaborate scoped-path wipe becomes largely redundant under C — keep delete.go doing config/token cleanup and verifying before it prints ✔, but we no longer need it to chase flat/per-release/custom HOST_DATA_DIR.

7.5 — Migration: none needed as a data-copy tool. With reuse-on-upgrade, existing installs keep running on their current ~/.tracebloc until someone does a clean cluster delete + reinstall, at which point they start fresh on C. That's consistent with "delete means gone." Keep the leftover-guard so we never silently strand data; skip building a MySQL-datadir copier.

7.4 — Messaging. C lets us keep the stronger "within the secure environment" wording more honestly, but let's be precise internally: node-local gets us not-host-visible + dies-with-cluster, not cryptographically secure. Anyone with Docker access can still docker exec in. If "secure" needs to mean protected from a local admin, that's encryption at rest (Option D) — I'd treat that as phase 2, and lean on host full-disk encryption (FileVault/LUKS) rather than app-level crypto for a local dev tool unless the threat model demands otherwise.

Summary: 7.2 = C, 7.3 = yes, 7.1 = guard yes, heavy wipe no, 7.5 = guard only, no copier, 7.4 = keep wording, encryption is phase 2. Implementation is mostly in client (MySQL PV hostPath → local-path PVC, drop the ~/.tracebloc → /tracebloc mount + the host-dir/chmod steps); delete.go simplifies.

[saadqbal]

@LukasWodka — I dug into RFC-0003 and think we're ready to lock the §7 calls. Prototyped and validated Option C end-to-end on a real dev install; posted the full evidence in the §7 comment above and on client#368. Short version of where I land:

• 7.2 Storage → C (node-local, single-node). Data lives on k3s local-path inside the k3d node: dies with cluster delete, not a browsable ~/.tracebloc folder, no chmod 777. Verified on a real install — single-node, all PVCs on local-path, a real spawned ingest-job shared the data PVC on the one node, and delete destroyed the volume + data.
• 7.3 Drop 777 → yes. Falls out of C for free (no host data dirs).
• 7.1 → keep the installer leftover-guard, drop the heavy multi-path wipe. Under C there are no host paths for offboard to chase. The cli-side "verify before ✔" honesty fix is already up (cli#389).
• 7.5 Migration → no data-copier.--reuse-values means existing installs don't silently migrate; they move to C on a clean delete + reinstall. Guard prevents stranding data.
• 7.4 Encryption → phase 2. C gets us not-host-visible + dies-with-cluster; cryptographic-at-rest is a separate, bigger piece if we want it.

Two things I need from you:

1. Sign off (or push back on) the above so I can build the rest — the installer leftover-guard and the default-flip.
2. One call to make: node-local is opt-in behind a flag today. Making it the default for local installs is the step that actually delivers the RFC goal to users — happy to do it, want your agreement first.

Minor: the RFC §9 evidence line numbers are stale (the installer's been refactored since) — worth a refresh before we call it decided.

[saadqbal]

Two pieces of feedback came in on this RFC. Both are good, but they're different in kind — one sharpens what's already here, the other is a new concern the RFC doesn't cover. First, the framing that shapes both:

On a machine the customer owns, we can't make data inaccessible or un-tamperable absolutely — the owner has root, Docker, and the disk. The honest goals are "no other service casually touches it" and "tampering is detectable", not "impossible". A design promising absolute prevention on the customer's own box would be the same overclaim §4.2 was written to avoid.

1. "Data only accessible by the tracebloc cluster, not other services"

Three layers, different answers:

• Host filesystem (Spotlight, backups, other apps/users): this is the actual current hole — 777 world-readable files in ~/.tracebloc. Option C closes it (data moves into the node, off the host FS). Verified on a real install.
• Docker/root (docker exec/cp into the node): C does not close this. Encryption-at-rest (D) helps, but keys live on the same box, so it's obfuscation, not a guarantee — better delivered by host full-disk encryption (FileVault/LUKS).
• In-cluster (other pods, incl. untrusted vendor training pods): needs PVC mount-scoping / RBAC so a training job can't read other datasets' raw data, on top of the training NetworkPolicy we already ship.

Verdict: good, and in-scope — it's a sharpening of Problem B. It moves §7.2 from "C" toward "C now + D/host-FDE at rest + a follow-up for in-cluster PVC scoping". Note the phrasing is slippery: the point of tracebloc is that vendor models do process the data in-cluster, so it's "no service outside the sanctioned training flow", not "no service". Worth pinning the exact layer with whoever raised it.

2. "Data should be tamper-free so scores stay valid"

The real ask: the test/scoring dataset must be fixed and verifiable, so leaderboard scores are reproducible and comparable across competitors — a tampered re-run mustn't silently produce a new score.

Achievable version is tamper-evidence, not tamper-proofing:

• ingest → compute a dataset fingerprint (content hash / Merkle root), store it;
• scoring → mount test data read-only, verify the hash, record it with the score;
• backend → bind each score to the fingerprint; flag/segregate scores from a different hash.

Verdict: good and genuinely important for a benchmark product — but orthogonal to this RFC. Its root cause (verifiable, versioned scoring data) is unrelated to the storage-location root cause here; Option C barely touches it. Caveats: we can't stop the owner tampering, only detect it; the threat model is really "fairness for competing vendors + reproducibility"; and legitimate updates mean this is versioning + integrity, not blanket immutability. It spans ingestor + jobs-manager scoring + backend leaderboard — bigger than this doc.

Recommendation

• (1) fold into RFC-0003 — update §4.6 to "C + D/host-FDE + in-cluster PVC scoping", with the honest Docker/root ceiling stated.
• (2) spin a separate RFC — "dataset integrity & score reproducibility". Bolting it on would blur a doc that's tight around one idea.
• For both, the highest-value next step is pinning the threat model ("protect from whom") — that's what separates the cheap wins (C, hash-at-ingest) from the expensive low-value ones (true crypto isolation on an owned box).

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit afe9cd1. Configure here.}

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a936323. Configure here.}

[LukasWodka]

Signed off — and thanks for the prototype + validation, that's what settled it. Everything is recorded in the doc, now at v2.1:

• Your §7 package: agreed as posted. 7.2 = C (D1), 7.3 = yes via C (D2), 7.1 = guard yes / heavy wipe no (D3), 7.5 = no copier (D4), 7.4 = keep wording, encryption phase 2 (D5/D14).
• Default-flip: agreed (D15). Node-local becomes the default for local installs, gated on one green end-to-end training run on node-local — the last unchecked client#368 item. The dev 401 blocking it gets its own P1 ticket. Single-node topology change goes in the release notes.
• v2 also pins what these decisions serve: the secure-environment definition + threat model (§1–2), model-IP stance (§6 — watermarking + audit now, TEE phase 2), in-cluster scoping incl. your PVC point + per-experiment DB grants (§7), seal check + guarantee matrix (§8). Your stale-evidence note is fixed (§13).
• One v2.1 correction worth your eyes: weights never rest durably in the environment (per-cycle scratch only — verified in code), so envelope-encryption/crypto-shredding moved to the platform-side averaging store as a backend/averaging ticket.

Go ahead on the leftover-guard and the flip. Execution tickets for the rest are being filed under backend#1151 — links will follow here. When you get a chance, review v2.1 so we can merge this.

[LukasWodka]

bugbot run

[LukasWodka]

As promised — execution tickets are filed; the doc is now v2.2 (DECIDED) with everything cross-linked in §10/§12. Epic: backend#1151.

P1: client-runtime#199 (egress-lockdown flip, D6) · client#376 (leftover-guard, D3 — yours) · backend#1180 (dev 401 — gates the D15 training run) · client#367 (now carries the D15 default-flip checklist — yours)

P2: client-runtime#200 (job TTL, D8-edge) · client-runtime#201 (legacy carve-out, D11) · client-runtime#202 (SA-token + securityContext floor, D11) · client-runtime#203 (dataset-scoped mounts, D9) · backend#1181 (per-experiment DB grants, D10) · backend#1182 (platform weight-store encryption + crypto-shred, D8/O2) · backend#1183 (watermarking + delivery audit, D7/O4) · backend#1184 (seal check + guarantee matrix, D12) · cli#393 (CLI seal-status surfacing) · backend#1185 (integrity RFC draft, D13) · backend#1186 (messaging alignment, §9)

Work starts in parallel on the non-yours lanes now; PRs will reference the tickets. Review of v2.2 whenever you're ready → then Lukas merges this.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7c6bccf. Configure here.}

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 0909bce. Configure here.}

[shujaatTracebloc]

Review — RFC-0003

Doc-only; all builds + fixtures-drift + Bugbot green, and it's already had heavy co-review (Lukas + Asad, v1→v2.3). Reviewed it as a design doc + verified the load-bearing §7bis code citations against the actual repos. Net: strong, unusually honest RFC — good to merge once the status line is reconciled.

What stands out

• Honesty discipline is exemplary for a security doc: refuses "air-gapped" (§1), commits to "raw data never leaves" rather than "nothing derived leaves" (§2.2), states the owner-root ceiling before designing around it (§6.2), and adopts "silent non-protection is worse than explicit disabling" (§8.2). This survives a customer security review.
• The v2.1 self-correction (re-verifying against code that weights don't rest durably in-environment, and retracting the proposal to encrypt a store that doesn't exist) is the doc catching its own error — confidence-building.
• Option C reasoning is clean — separating restart-persistence from delete-persistence is the right frame, validated on a real install (client#368), with the single-node AGENTS=0/SERVERS=1 tradeoff (C1) called out honestly.

Fix items

1. Status contradiction (PR body vs doc) — the PR description still opens "Status: DRAFT for discussion — concept only, no code proposed to merge," but the doc header says "DECIDED — v2.3; D1–D20 locked." Please update the PR body so reviewers aren't misled about whether the decisions are live. (left for you — it's the PR description)
2. Stale repo name — the RFC cited tracebloc-client (§7bis, O6, appendix), but that repo was renamed to tracebloc-engine. Verified the paths exist there (core/utils/database.py:243get_sql_query_and_params, core/utils/general.py:21-33get_experiment_path, core/weights/base.py). ✅ fixed & pushed (3a56fcc).
3. D19 line drift — the ingestor reuse/append branch spans 275-357, not 309-357; the old range started after the "return existing table if already created" reuse check (line 275) that D19 actually removes. ✅ fixed & pushed (3a56fcc).

Design points (not blockers)

• O6 is the crux of §7bis, not a footnote. Today the read path takes a list of ingestor_ids (database.py:188 ingestor_ids: List[str]) and reads them in one indexed WHERE ingestor_id IN (…). Under D16 (one immutable ds_<ingestor_id> per ingestion), a dataset spanning N ingestions becomes a UNION over N tables behind a definer-view — a real read-path/perf redesign that determines whether §7bis is buildable as written. Correctly filed as O6; just flagging it gates feasibility rather than being a loose end.
• Definer-view isolation holds — GRANT SELECT on a definer-rights view needs no grant on base tables, and MySQL won't let the training user reference or enumerate (information_schema) tables it has no privilege on. The hidden assumption is that per-experiment creds carry no broader grant (db.*/*.*); worth stating that as the invariant D10/D18 must preserve.
• Grandfathering (D20) makes isolation forward-only — pre-existing shared tables keep the WHERE ingestor_id convention indefinitely (no backfill). Consistent with D4 and honest, but the strong guarantee applies only to post-flip ingests for a long time; make sure §9 messaging doesn't overclaim during that window.
• Scope vs filename — the doc grew to "the secure environment" while the file is still 0003-storage-and-offboard-hygiene.md. RFC numbers are stable so I wouldn't rename; minor discoverability wrinkle.

— @shujaatTracebloc (reviewed with Claude)