diff --git a/create-use-case/prepare-dataset.mdx b/create-use-case/prepare-dataset.mdx index 05d77a3..aee27d7 100644 --- a/create-use-case/prepare-dataset.mdx +++ b/create-use-case/prepare-dataset.mdx @@ -5,7 +5,7 @@ description: "Learn how to prepare and ingest your datasets into tracebloc using ## Overview -Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Regardless of where your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way. +Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Whether your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way. The data ingestor is a lightweight service that bridges your raw data and the cluster's persistent storage. It comes with ready-made templates (CSV, images, text) that you can use as starting points and customize for your own dataset. By containerizing the ingestion step, the ingestor validates data format and schema, enforces consistency, and transfers the dataset securely into cluster's SQL storage where it becomes accessible to all training and evaluation jobs. @@ -29,6 +29,16 @@ Start with the declarative method below. Drop down to the custom-template flow o Describe your dataset in ~8 lines of YAML, then `helm install`. The official ingestor image (published as `ghcr.io/tracebloc/ingestor`) runs it. No Dockerfile, no Python script. + +**Before you run any commands in this section:** if you installed the client via the one-liner (`bash <(curl -fsSL https://tracebloc.io/i.sh)`), every later `helm upgrade tracebloc/client …` **must** include `--reset-then-reuse-values`, otherwise the upgrade drops the values the installer applied and breaks the workspace: + +```bash +helm upgrade tracebloc/client -n --reset-then-reuse-values +``` + +Append `--version ` to pin a specific chart version. This caveat only affects upgrades of the parent `tracebloc/client` chart, not the `helm install tracebloc/ingestor` runs below. + + ### 1. Add the chart repo (one-time) ```bash @@ -38,19 +48,25 @@ helm repo update The `tracebloc/client` parent chart bootstraps the cluster (jobs-manager, MySQL, RBAC). The `tracebloc/ingestor` subchart submits per-dataset ingestion runs against it. - -If you installed the client via the one-liner (`bash <(curl -fsSL https://tracebloc.io/i.sh)`), use `--reset-then-reuse-values` so the helm upgrade doesn't drop the values the installer applied: +### 2. Stage your data on the cluster's shared PVC + +The chart **doesn't transport data into the cluster** — it points at data already accessible to the cluster's shared PVC (`client-pvc` by default, mounted at `/data/shared/` inside the ingestor Pod). Before installing, get your raw files there. + +For a single-node workspace (the default install), the PVC is backed by a host directory the installer created at `~/.tracebloc//data/`. Drop your files into a per-dataset subdirectory: ```bash -helm upgrade tracebloc/client -n --reset-then-reuse-values +# Host path on the machine where the tracebloc client is installed. +# Pick a per dataset — it becomes the path you reference in ingest.yaml. +mkdir -p ~/.tracebloc//data/ +cp -R LOCAL_PATH/images ~/.tracebloc//data// +cp LOCAL_PATH/labels.csv ~/.tracebloc//data// ``` -Append `--version ` to pin a specific chart version. - - -### 2. Stage your data on the cluster's shared PVC +Inside the ingestor Pod those files appear at `/data/shared//...` — that's what you'll put in `ingest.yaml` below. -The chart **doesn't transport data into the cluster** — it points at data already accessible to the cluster's shared PVC (`client-pvc` by default, mounted at `/data/shared/` inside the ingestor Pod). Before installing, get your raw files there. The simplest pattern for a small dataset is a throwaway `kubectl cp` Pod that mounts the PVC; for production you'd typically use an init container with cloud-storage sync. Full staging recipe and manifests live in the [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md#stage-your-data-on-the-shared-pvc). + +For multi-node or EKS deployments where the PVC isn't backed by a local host path, use a throwaway `kubectl cp` Pod or a cloud-storage init container instead. See the [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md#stage-your-data-on-the-shared-pvc) for those recipes. + ### 3. Write your `ingest.yaml` @@ -71,13 +87,32 @@ The top-level shape (`apiVersion`, `kind`, `category`, `table`, `intent`, `label ### 4. Install once per dataset +The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. **Run it twice per dataset** — once with `intent: train`, once with `intent: test` — using distinct `table:` names. The example below shows both releases: + ```bash -helm install my-cats-dogs tracebloc/ingestor \ +# Train release — points at the ingest.yaml from step 3 (table: cats_dogs_train, intent: train) +helm install cats-dogs-train tracebloc/ingestor \ --namespace \ - --set-file ingestConfig=./ingest.yaml + --set-file ingestConfig=./ingest-train.yaml + +# Test release — same shape, with table: cats_dogs_test and intent: test +helm install cats-dogs-test tracebloc/ingestor \ + --namespace \ + --set-file ingestConfig=./ingest-test.yaml +``` + +Each `helm install` is a separate release (the first argument is the release name), so the two runs don't collide. The ingestor Pod picks up `CLIENT_ID` / `CLIENT_PASSWORD` automatically from the Kubernetes Secret the parent `tracebloc/client` chart created in `` at install time — you don't pass credentials on the `helm install` command. + + +**Validation error like `'' is not one of [...]` or `Additional properties are not allowed ( was unexpected)`?** This comes from the cluster's `jobs-manager` validating against its own bundled schema at submit time — the deployed schema is older than the ingestor image you're installing. `helm repo update` won't fix it (that only refreshes the local chart index, not the running server). The fix is on the cluster side: upgrade the parent chart so jobs-manager redeploys with the current schema. + +```bash +helm upgrade tracebloc/client \ + -n --reset-then-reuse-values ``` -The ingestor runs once: validates your data, copies files into the destination directory on the PVC, inserts rows into MySQL, sends metadata to the tracebloc backend, then exits. Repeat per dataset (one helm release per dataset, with different `table:` and `intent:` for train and test). +Then re-run the `helm install` command above. + Full chart docs (data-staging recipe, schema, every category, update model, verification, override knobs) → [client ingestor README](https://github.com/tracebloc/client/blob/develop/ingestor/README.md). diff --git a/create-use-case/templates.mdx b/create-use-case/templates.mdx index 5b95571..72ac550 100644 --- a/create-use-case/templates.mdx +++ b/create-use-case/templates.mdx @@ -14,6 +14,7 @@ Each task tracebloc supports comes with a runnable data-ingestion template — a | Keypoint detection | [`templates/keypoint_detection`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/keypoint_detection) | | Semantic segmentation | [`templates/semantic_segmentation`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/semantic_segmentation) | | Text classification | [`templates/text_classification`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/text_classification) | +| Masked language modeling | [`templates/masked_language_modeling`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/masked_language_modeling) | | Tabular classification | [`templates/tabular_classification`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/tabular_classification) | | Tabular regression | [`templates/tabular_regression`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/tabular_regression) | | Time series forecasting | [`templates/time_series_forecasting`](https://github.com/tracebloc/data-ingestors/tree/develop/templates/time_series_forecasting) |