From a2f6fa8416ef1b709878085cdb94fa58c48cac42 Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Sat, 11 Jul 2026 14:23:16 +0200 Subject: [PATCH] docs(rfc-0002): dataset immutability principle + reject --append (cli#156) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds Principle 6 (datasets are immutable snapshots) and rewrites the §13 --append non-goal into an explicit rejection now that cli#156 is closed obsolete. In-place append breaks dataset integrity/reproducibility; growth = dataset versioning, not mutation. Rationale of record: backend#1073. Co-Authored-By: Claude Opus 4.8 --- docs/rfcs/0002-data-ingest-flow.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/docs/rfcs/0002-data-ingest-flow.md b/docs/rfcs/0002-data-ingest-flow.md index acae1913..1a3f6814 100644 --- a/docs/rfcs/0002-data-ingest-flow.md +++ b/docs/rfcs/0002-data-ingest-flow.md @@ -50,6 +50,12 @@ tasks currently lives, hand-maintained, in five places across three repos. dataset facts we cannot see. 5. **One-liners.** Every task, flag, and prompt carries a plain one-line gloss of what it means / predicts. +6. **Datasets are immutable snapshots.** A dataset is the fixed basis for its + use cases and for training — reproducibility depends on it not changing + under already-trained models. Ingest **creates** a dataset, or replaces one + wholesale (`--overwrite`); it never **appends** in place. Growth means a new + (versioned) dataset, never mutation. (This is why `--append` is rejected — + §13; rationale of record: backend#1073.) ## 3. Current state (as shipped on `develop`) @@ -459,7 +465,13 @@ type per dataset. v0.1 caps: 1 GiB total, 500 MiB per file. - Cloud-source datasets (S3/GCS/HTTPS) beyond the 1 GiB cap — v0.2. - Changing the wire schema field name (`category` stays on the wire; only the CLI surface becomes `--task`). -- Server-side `--append` (tracked separately, cli#156). +- **Server-side `--append` — rejected** (cli#156 closed obsolete, 2026-07-11). + In-place append breaks dataset integrity & reproducibility: a dataset is the + fixed basis for its use cases and training, so appending after models have + trained makes their results non-comparable and non-reproducible. Datasets are + **immutable snapshots** (Principle 6). If incremental growth is ever genuinely + needed the correct shape is **dataset versioning** (a new immutable version), + not append. Rationale of record: backend#1073. ## Revision history @@ -486,3 +498,7 @@ type per dataset. v0.1 caps: 1 GiB total, 500 MiB per file. questions (§12.5–8) from reconciling the CLI vs. ingestor: drop vestigial `--label-column` for sidecar-labeled vision, the forecasting-timestamp gap, `--target-size` placement, and schema-required-for-regression. +- **Rev 5 (2026-07-11)** — record **dataset immutability** as Principle 6 and + turn the §13 `--append` non-goal into an explicit **rejection** (cli#156 + closed obsolete; in-place append breaks integrity/reproducibility; growth = + dataset versioning, not mutation; rationale of record backend#1073).