Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 1 addition & 14 deletions create-use-case/prepare-dataset.mdx
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,31 @@
---
title: "Prepare Data"
description: "Learn how to prepare and ingest your datasets into tracebloc using containerized data ingestors. Complete guide for CSV, image, and text data with Kubernetes deployment steps."

Check warning on line 3 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L3

Did you really mean 'tracebloc'?

Check warning on line 3 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L3

Did you really mean 'ingestors'?
---

## Overview

Make your data available to the Kubernetes cluster so it can be used for training and evaluation. Regardless of where your client runs on Azure, AWS, Google Cloud, or a local Minikube setup, the process of ingesting datasets works the same way.

Check warning on line 8 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L8

Did you really mean 'Minikube'?

The data ingestor is a lightweight service that bridges your raw data and the cluster's persistent storage. It comes with ready-made templates (CSV, images, text) that you can use as starting points and customize for your own dataset. By containerizing the ingestion step, the ingestor validates data format and schema, enforces consistency, and transfers the dataset securely into cluster's SQL storage where it becomes accessible to all training and evaluation jobs.

Check warning on line 10 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L10

Did you really mean 'ingestor'?

Check warning on line 10 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L10

Did you really mean 'ingestor'?

This guide covers:
- Customizing ingestor templates for different data types (CSV, images, text)

Check warning on line 13 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L13

Did you really mean 'ingestor'?
- Deploying the data ingestor for training and test data using Kubernetes

Check warning on line 14 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L14

Did you really mean 'ingestor'?
- Managing datasets through the tracebloc interface

Check warning on line 15 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L15

Did you really mean 'tracebloc'?

**IMPORTANT** Make sure that the data format and ML task is supported and that data standards are met by reviewing the [docs](/create-use-case/prerequisites). You must run the process twice, once to ingest training and once to ingest testing data.

## Quick Setup

Use this quick setup if you already have an ingestor configured and just want to switch datasets or toggle between training and testing. If you are setting up for the first time, go to the next section for the detailed walkthrough.

Check warning on line 21 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L21

Did you really mean 'ingestor'?

Check warning on line 21 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L21

Did you really mean 'walkthrough'?

### Steps

1. Pick a template script and edit it. E.g. `/templates/tabular_classification/tabular_classification.py`
- Update csv options and data_path

Check warning on line 26 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L26

Did you really mean 'csv'?

Check warning on line 26 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L26

Did you really mean 'data_path'?
- Only for tabular data: Update schema
- Set `schema` and `CSVIngestor()`parameters like category, intent, label_column, etc. to match data type, task and train/test purpose

Check warning on line 28 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L28

Did you really mean 'label_column'?

```python
ingestor = CSVIngestor(
Expand DownExpand Up@@ -59,9 +59,9 @@

### 1. Configure a Template

This section walks you through the step-by-step setup of a data ingestor. You will clone the repository, select the right template for your data type, and customize it to match your task. Follow this guide if you are setting up an ingestor for the first time or need full control beyond the quick setup.

Check warning on line 62 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L62

Did you really mean 'ingestor'?

Check warning on line 62 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L62

Did you really mean 'ingestor'?

### Clone the Data Ingestor Repository

Check warning on line 64 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L64

Did you really mean 'Ingestor'?

Clone the public [Data Ingestor GitHub repository](https://github.com/tracebloc/data-ingestors):

Expand DownExpand Up@@ -126,14 +126,14 @@
...
```

Both Database, APIClient and other values are configured automatically from the environment variables defined in `ingestor_job.yaml`.

Check warning on line 129 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L129

Did you really mean 'APIClient'?

- `config.LABEL_FILE`: Path to local csv label file

Check warning on line 131 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L131

Did you really mean 'csv'?
- `config.BATCH_SIZE`: Batch size used during ingestion

### Customize a Template

Templates provide a starting point, but every dataset has its own format and labels. In this step you adapt the template to your data by tuning CSV ingestion options and setting the ingestor parameters (category, label column, intent, data path and schema). The following example in `templates/tabular_classification/tabular_classification.py` shows how to ingest a tabular dataset, but the setup works the same way for image or text data.

Check warning on line 136 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L136

Did you really mean 'ingestor'?

#### Needed for Tabular Data: Define Schema

Expand DownExpand Up@@ -186,7 +186,7 @@
```

#### Set CSV ingestion options
Customize parsing, memory handling, and data cleaning with the csv_options dictionary:

Check warning on line 189 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L189

Did you really mean 'csv_options'?

```python
csv_options = {
Expand All@@ -201,9 +201,9 @@
}
```

#### Set Up the Ingestor

Check warning on line 204 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L204

Did you really mean 'Ingestor'?

Define the Ingestor instance with the required configuration. See the tabular data example below:

Check warning on line 206 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L206

Did you really mean 'Ingestor'?

```python
ingestor = CSVIngestor(
Expand DownExpand Up@@ -235,7 +235,7 @@

### Docker Hub Setup (first-time users)

The cluster pulls your ingestor image from a public Docker registry, so you need an account before you can push. If you already have one, skip to [Edit Dockerfile](#edit-dockerfile).

Check warning on line 238 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L238

Did you really mean 'ingestor'?

1. **Create a Docker Hub account** at [hub.docker.com/signup](https://hub.docker.com/signup) and verify your email.
2. **Log in from your terminal** so the `docker push` command can authenticate:
Expand All@@ -244,18 +244,18 @@
docker login
```

3. **Push the data ingestor image** to your account using the build/push commands in the next section. The image name takes the form `<your-docker-username>/<image-name>:<tag>` — the username segment must match the account you just created.

Check warning on line 247 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L247

Did you really mean 'ingestor'?
4. **Make the image public** so the cluster can pull it without credentials:
- Go to [hub.docker.com/repositories](https://hub.docker.com/repositories), open the repository you just pushed.
- Click **Settings → Visibility settings → Make public**.

Keeping the image private is also fine, but then you must create a Kubernetes `imagePullSecret` named `regcred` in the client namespace (the `ingestor-job.yaml` already references it).

Check warning on line 252 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L252

Did you really mean 'namespace'?

### Place data files on the client host

Datasets are **not** baked into the Docker image. They live on the client host in the per-workspace data directory and are mounted into the ingestor pod through the shared PVC (`client-pvc` → `/data/shared`).

Check warning on line 256 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L256

Did you really mean 'ingestor'?

Copy your dataset into the client's data directory, where `<workspace>` is the workspace name you chose during client install (which is also the Helm release name and the Kubernetes namespace — the chart uses the same value for all three). The directory `~/.tracebloc/<workspace>/data/` is created automatically by the installer; just drop your files into it:

Check warning on line 258 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L258

Did you really mean 'namespace'?

```bash
# Host path on the machine where the tracebloc client is installed.
Expand All@@ -264,20 +264,20 @@
cp LOCAL_PATH/labels.csv ~/.tracebloc/<workspace>/data/
```

Inside the ingestor pod this directory is mounted at `/data/shared`, so the same files appear as `/data/shared/images/...` and `/data/shared/labels.csv`. Set `SRC_PATH` and `LABEL_FILE` in `ingestor-job.yaml` to point at those in-pod paths (see [Configure Kubernetes](#3-configure-kubernetes) below).

Check warning on line 267 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L267

Did you really mean 'ingestor'?

For tabular data the same rule applies — drop the single `labels.csv` (with features and labels) into `~/.tracebloc/<workspace>/data/`.

### Edit Dockerfile

Check warning on line 271 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L271

Did you really mean 'Dockerfile'?

The Dockerfile only needs to package the ingestion script — the dataset is mounted at runtime, so do **not** `COPY` data into the image:

Check warning on line 273 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L273

Did you really mean 'Dockerfile'?

```dockerfile
# Copy the ingestion script into /app
COPY templates/tabular_classification/tabular_classification.py /app/ingestor.py
```

If the cluster enforces the `restricted` Pod Security Standard (see [Run as non-root](#run-as-non-root) below), also add a non-root user to the Dockerfile, **before** the `# Set the entrypoint` line:

Check warning on line 280 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L280

Did you really mean 'Dockerfile'?

```dockerfile
RUN groupadd -g 1000 app && \
Expand All@@ -292,22 +292,9 @@

### Build Docker Image

You need a docker user and password to proceed with the next step. Cloud platforms run a mix of x86 and ARM nodes (e.g. AWS Graviton, Azure Ampere, GCP Tau T2A). Building a multi-arch image with `--platform linux/amd64,linux/arm64` guarantees the image runs on either, particularly if you build on Apple Silicon (M1/M2) or other ARM-based systems. Pick a setup, build and deploy the image:

#### For Local Development/Testing

```bash
# Build for your local platform
docker build -t <your-username>/<image-name>:<tag> .

# Optional: Push to registry for sharing
docker push <your-username>/<image-name>:<tag>
```

#### For Cloud Deployment (AWS, Azure, GCP)
You need a docker user and password to proceed with the next step. Cloud platforms run a mix of x86 and ARM nodes (e.g. AWS Graviton, Azure Ampere, GCP Tau T2A). Building a multi-arch image with `--platform linux/amd64,linux/arm64` guarantees the image runs on either, particularly if you build on Apple Silicon (M1/M2) or other ARM-based systems. Build and push the image with a single command:

```bash
# Build a multi-arch image (works on x86 and ARM cloud nodes) and push directly to the registry
docker buildx build --platform linux/amd64,linux/arm64 -t <your-username>/<image-name>:<tag> --push .
```

Expand DownExpand Up@@ -389,14 +376,14 @@
- `image`, your Docker image (imagePullPolicy: Always for DockerHub, IfNotPresent for local)
- `CLIENT_ID`, `CLIENT_PASSWORD` from the [tracebloc client view](https://ai.tracebloc.io/clients)
- `TABLE_NAME`, unique per dataset, train and test use different names, no spaces. Different names for train and test data is mandatory
- `LABEL_FILE`, path inside the ingestor pod (under `/data/shared`) to the CSV with file paths and labels — must match the location of the file you placed in `~/.tracebloc/<workspace>/data/`

Check warning on line 379 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L379

Did you really mean 'ingestor'?
- `SRC_PATH`, root inside the pod where the dataset directory is mounted (`/data/shared`)
- `BATCH_SIZE` is the number of entries sent to the server per request. Optional — defaults to 4000. Keep it consistent across data types. It depends on available CPU memory, not for example image size. Too large can exhaust memory. It was tested up to 10,000, but 5,000 is a safe default for most systems.
- `LOG_LEVEL`, "WARNING" for all warnings and errors, "INFO" for all logs, "ERROR" for errors only

### 4. Deploy

Run the ingestor as a Kubernetes Job:

Check warning on line 386 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L386

Did you really mean 'ingestor'?

```bash
kubectl apply -f ingestor-job.yaml -n <workspace>
Expand All@@ -412,7 +399,7 @@

### Run as non-root

If the namespace enforces the `restricted` [Pod Security Standard](https://kubernetes.io/docs/concepts/security/pod-security-standards/), `kubectl apply` will be admitted but the pod will be rejected with a warning like:

Check warning on line 402 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L402

Did you really mean 'namespace'?

```text
Warning: would violate PodSecurity "restricted:latest":
Expand All@@ -438,7 +425,7 @@
type: RuntimeDefault
```

**2. Run the container as a non-root user.** Add the following to the Dockerfile **before** the `# Set the entrypoint` line so the image ships with a UID that satisfies `runAsNonRoot: true`:

Check warning on line 428 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L428

Did you really mean 'Dockerfile'?

```dockerfile
RUN groupadd -g 1000 app && \
Expand All@@ -450,7 +437,7 @@

Rebuild and push the image, then re-apply the job.

The data ingestor always runs a validation step before ingestion and moving files.

Check warning on line 440 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L440

Did you really mean 'ingestor'?


#### Verify Deployment
Expand All@@ -472,7 +459,7 @@
**Interface displays:**
- Dataset name, ID, and record count
- Data type (Tabular, Image, Text) and purpose (Training/Testing)
- Namespace and GPU requirements

Check warning on line 462 in create-use-case/prepare-dataset.mdx

View check run for this annotation

Mintlify/ Mintlify Validation (tracebloc) - vale-spellcheck

create-use-case/prepare-dataset.mdx#L462

Did you really mean 'Namespace'?

## Best Practices
- Deploy jobs for training and testing simultaneously using different job names
Expand Down
Loading