From c7d111847d6cec984a5f948fa5025b41407c7052 Mon Sep 17 00:00:00 2001 From: Divya Date: Mon, 1 Jun 2026 15:10:50 +0530 Subject: [PATCH 1/5] docs: make declarative-ingest staging self-contained (issue #131 B/C) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes the docs side of data-ingestors#131: - B2 / B1: section 2 of the declarative path linked out for the staging recipe and described `kubectl cp` while the Detailed Setup section (further down on the same page) prescribed a host-path `cp -R`. Replaced section 2 with an inline host-path recipe that matches the Detailed Setup section, and demoted `kubectl cp` to a Note for multi-node / EKS deployments. The recipe now uses a `` subdirectory so the path lines up with the `/data/shared//...` style used in ingest.yaml examples. - C2: section 4 was silent on where CLIENT_ID / CLIENT_PASSWORD come from in the declarative path. Added a sentence noting the ingestor Pod inherits them from the Kubernetes Secret the parent tracebloc/client chart creates in at install time — no creds are passed on the `helm install` line. - C5: section 4 mentioned the run-twice rule only as a trailing parenthetical. Promoted it to bolded prose and added a worked train + test pair (two `helm install` invocations, distinct release names + `table:` + `intent:`) so the rule is concrete. Co-Authored-By: Claude Opus 4.7 (1M context) --- create-use-case/prepare-dataset.mdx | 32 +++++++++++++++++++++++++---- 1 file changed, 28 insertions(+), 4 deletions(-) diff --git a/create-use-case/prepare-dataset.mdx b/create-use-case/prepare-dataset.mdx index 05d77a3..3133093 100644 --- a/create-use-case/prepare-dataset.mdx +++ b/create-use-case/prepare-dataset.mdx @@ -50,7 +50,23 @@ Append `--version ` to pin a specific chart version. ### 2. Stage your data on the cluster's shared PVC -The chart **doesn't transport data into the cluster** — it points at data already accessible to the cluster's shared PVC (`client-pvc` by default, mounted at `/data/shared/` inside the ingestor Pod). Before installing, get your raw files there. The simplest pattern for a small dataset is a throwaway `kubectl cp` Pod that mounts the PVC; for production you'd typically use an init container with cloud-storage sync. Full staging recipe and manifests live in the [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md#stage-your-data-on-the-shared-pvc). +The chart **doesn't transport data into the cluster** — it points at data already accessible to the cluster's shared PVC (`client-pvc` by default, mounted at `/data/shared/` inside the ingestor Pod). Before installing, get your raw files there. + +For a single-node workspace (the default install), the PVC is backed by a host directory the installer created at `~/.tracebloc//data/`. Drop your files into a per-dataset subdirectory: + +```bash +# Host path on the machine where the tracebloc client is installed. +# Pick a per dataset — it becomes the path you reference in ingest.yaml. +mkdir -p ~/.tracebloc//data/ +cp -R LOCAL_PATH/images ~/.tracebloc//data// +cp LOCAL_PATH/labels.csv ~/.tracebloc//data// +``` + +Inside the ingestor Pod those files appear at `/data/shared//...` — that's what you'll put in `ingest.yaml` below. + + +For multi-node or EKS deployments where the PVC isn't backed by a local host path, use a throwaway `kubectl cp` Pod or a cloud-storage init container instead. See the [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md#stage-your-data-on-the-shared-pvc) for those recipes. + ### 3. Write your `ingest.yaml` @@ -71,13 +87,21 @@ The top-level shape (`apiVersion`, `kind`, `category`, `table`, `intent`, `label ### 4. Install once per dataset +The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. **Run it twice per dataset** — once with `intent: train`, once with `intent: test` — using distinct `table:` names. The example below shows both releases: + ```bash -helm install my-cats-dogs tracebloc/ingestor \ +# Train release — points at the ingest.yaml from step 3 (table: cats_dogs_train, intent: train) +helm install cats-dogs-train tracebloc/ingestor \ + --namespace \ + --set-file ingestConfig=./ingest-train.yaml + +# Test release — same shape, with table: cats_dogs_test and intent: test +helm install cats-dogs-test tracebloc/ingestor \ --namespace \ - --set-file ingestConfig=./ingest.yaml + --set-file ingestConfig=./ingest-test.yaml ``` -The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. Repeat per dataset (one helm release per dataset, with different `table:` and `intent:` for train and test). +Each `helm install` is a separate release (the first argument is the release name), so the two runs don't collide. The ingestor Pod picks up `CLIENT_ID` / `CLIENT_PASSWORD` automatically from the Kubernetes Secret the parent `tracebloc/client` chart created in `` at install time — you don't pass credentials on the `helm install` command. Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md). From dc1e09a8381241e19bb62f882a8fda986a94a217 Mon Sep 17 00:00:00 2001 From: Divya Date: Tue, 2 Jun 2026 14:07:39 +0530 Subject: [PATCH 2/5] docs: address PR #46 review feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Reword overview sentence: "Regardless of where" → "Whether" (cleaner). - Move the `--reset-then-reuse-values` caveat above section 1 so the warning appears before any commands the user could run, and clarify it only applies to upgrades of the parent `tracebloc/client` chart (not the `helm install tracebloc/ingestor` runs in step 4). Co-Authored-By: Claude Opus 4.7 (1M context) --- create-use-case/prepare-dataset.mdx | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/create-use-case/prepare-dataset.mdx b/create-use-case/prepare-dataset.mdx index 3133093..46727b7 100644 --- a/create-use-case/prepare-dataset.mdx +++ b/create-use-case/prepare-dataset.mdx @@ -5,7 +5,7 @@ description: "Learn how to prepare and ingest your datasets into tracebloc using ## Overview -Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Regardless of where your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way. +Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Whether your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way. The data ingestor is a lightweight service that bridges your raw data and the cluster's persistent storage. It comes with ready-made templates (CSV, images, text) that you can use as starting points and customize for your own dataset. By containerizing the ingestion step, the ingestor validates data format and schema, enforces consistency, and transfers the dataset securely into cluster's SQL storage where it becomes accessible to all training and evaluation jobs. @@ -29,24 +29,24 @@ Start with the declarative method below. Drop down to the custom-template flow o Describe your dataset in ~8 lines of YAML, then `helm install`. The official ingestor image (published as `ghcr.io/tracebloc/ingestor`) runs it. No Dockerfile, no Python script. -### 1. Add the chart repo (one-time) + +**Before you run any commands in this section:** if you installed the client via the one-liner (`bash <(curl -fsSL https://tracebloc.io/i.sh)`), every later `helm upgrade tracebloc/client …` **must** include `--reset-then-reuse-values`, otherwise the upgrade drops the values the installer applied and breaks the workspace: ```bash -helm repo add tracebloc https://tracebloc.github.io/client -helm repo update +helm upgrade tracebloc/client -n --reset-then-reuse-values ``` -The `tracebloc/client` parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The `tracebloc/ingestor` subchart submits per-dataset ingestion runs against it. +Append `--version ` to pin a specific chart version. This caveat only affects upgrades of the parent `tracebloc/client` chart, not the `helm install tracebloc/ingestor` runs below. + - -If you installed the client via the one-liner (`bash <(curl -fsSL https://tracebloc.io/i.sh)`), use `--reset-then-reuse-values` so the helm upgrade doesn't drop the values the installer applied: +### 1. Add the chart repo (one-time) ```bash -helm upgrade tracebloc/client -n --reset-then-reuse-values +helm repo add tracebloc https://tracebloc.github.io/client +helm repo update ``` -Append `--version ` to pin a specific chart version. - +The `tracebloc/client` parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The `tracebloc/ingestor` subchart submits per-dataset ingestion runs against it. ### 2. Stage your data on the cluster's shared PVC From 340cc90fa13188088dca918abb22644134ba0149 Mon Sep 17 00:00:00 2001 From: Divya Date: Tue, 2 Jun 2026 14:21:40 +0530 Subject: [PATCH 3/5] docs: add masked_language_modeling to templates table The templates table in create-use-case/templates.mdx listed 9 supported tasks but was missing masked_language_modeling, even though the template exists in data-ingestors. Added the row alongside the others. Deep MLM-specific guidance (tokenizer.json requirements, validation, troubleshooting) lives in the data-ingestors template README, where the TokenizerValidator does. Co-Authored-By: Claude Opus 4.7 (1M context) --- create-use-case/templates.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/create-use-case/templates.mdx b/create-use-case/templates.mdx index 5b95571..72ac550 100644 --- a/create-use-case/templates.mdx +++ b/create-use-case/templates.mdx @@ -14,6 +14,7 @@ Each task tracebloc supports comes with a runnable data-ingestion template — a | Keypoint detection | [`templates/keypoint_detection`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/keypoint_detection) | | Semantic segmentation | [`templates/semantic_segmentation`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/semantic_segmentation) | | Text classification | [`templates/text_classification`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/text_classification) | +| Masked language modeling | [`templates/masked_language_modeling`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/masked_language_modeling) | | Tabular classification | [`templates/tabular_classification`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/tabular_classification) | | Tabular regression | [`templates/tabular_regression`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/tabular_regression) | | Time series forecasting | [`templates/time_series_forecasting`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/time_series_forecasting) | From c31c38b665588001db7b37aad3fb7507098984c6 Mon Sep 17 00:00:00 2001 From: Divya Date: Tue, 2 Jun 2026 14:40:22 +0530 Subject: [PATCH 4/5] docs: add helm repo update note for stale-chart validation errors A reviewer hit this on a fresh MLM install: ingest_config validation failed: : Additional properties are not allowed ('sequences' was unexpected) category: 'masked_language_modeling' is not one of [...] Both symptoms point at a stale local Helm chart cache that predates the newer category or schema field. `helm repo update` refreshes the cache and the next `helm install` picks up the current schema. Added a Warning callout in step 4 of the Declarative YAML section, scoped generically (any category / schema field, not just MLM). Co-Authored-By: Claude Opus 4.7 (1M context) --- create-use-case/prepare-dataset.mdx | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/create-use-case/prepare-dataset.mdx b/create-use-case/prepare-dataset.mdx index 46727b7..8743761 100644 --- a/create-use-case/prepare-dataset.mdx +++ b/create-use-case/prepare-dataset.mdx @@ -103,6 +103,16 @@ helm install cats-dogs-test tracebloc/ingestor \ Each `helm install` is a separate release (the first argument is the release name), so the two runs don't collide. The ingestor Pod picks up `CLIENT_ID` / `CLIENT_PASSWORD` automatically from the Kubernetes Secret the parent `tracebloc/client` chart created in `` at install time — you don't pass credentials on the `helm install` command. + +**Validation error like `'' is not one of [...]` or `Additional properties are not allowed`?** Your local Helm chart cache is stale and doesn't know about the newer category or schema field. Refresh it before retrying: + +```bash +helm repo update +``` + +Then re-run the `helm install` command above. + + Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md). ## Custom Python template (advanced) From 31da8733847cbd3efcc5332c0066ad5027ffb007 Mon Sep 17 00:00:00 2001 From: Divya Date: Wed, 3 Jun 2026 12:18:07 +0530 Subject: [PATCH 5/5] =?UTF-8?q?docs:=20correct=20troubleshooting=20note=20?= =?UTF-8?q?=E2=80=94=20server-side=20schema,=20not=20local=20chart=20cache?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Companion to data-ingestors PR #133 commit 5550d1a. Reviewer (@LukasWodka) pointed out the previous Warning was diagnostically wrong: the schema-validation error comes from jobs-manager's submit-time check against its own bundled schema, not from the local Helm chart index, so `helm repo update` is a no-op. The fix is to upgrade the parent `tracebloc/client` chart so jobs-manager redeploys with the current schema. Co-Authored-By: Claude Opus 4.7 (1M context) --- create-use-case/prepare-dataset.mdx | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/create-use-case/prepare-dataset.mdx b/create-use-case/prepare-dataset.mdx index 8743761..aee27d7 100644 --- a/create-use-case/prepare-dataset.mdx +++ b/create-use-case/prepare-dataset.mdx @@ -104,10 +104,11 @@ helm install cats-dogs-test tracebloc/ingestor \ Each `helm install` is a separate release (the first argument is the release name), so the two runs don't collide. The ingestor Pod picks up `CLIENT_ID` / `CLIENT_PASSWORD` automatically from the Kubernetes Secret the parent `tracebloc/client` chart created in `` at install time — you don't pass credentials on the `helm install` command. -**Validation error like `'' is not one of [...]` or `Additional properties are not allowed`?** Your local Helm chart cache is stale and doesn't know about the newer category or schema field. Refresh it before retrying: +**Validation error like `'' is not one of [...]` or `Additional properties are not allowed ( was unexpected)`?** This comes from the cluster's `jobs-manager` validating against its own bundled schema at submit time — the deployed schema is older than the ingestor image you're installing. `helm repo update` won't fix it (that only refreshes the local chart index, not the running server). The fix is on the cluster side: upgrade the parent chart so jobs-manager redeploys with the current schema. ```bash -helm repo update +helm upgrade tracebloc/client \ + -n --reset-then-reuse-values ``` Then re-run the `helm install` command above.