tracebloc Client 🔒

The runtime that keeps your data where it belongs — on your infrastructure.

The tracebloc client deploys inside your Kubernetes cluster and executes all model training, fine-tuning, and inference locally. It connects to the tracebloc backend for orchestration only. No data, no model weights, no artifacts ever leave your environment.

Architecture

Your infrastructure
┌─────────────────────────────────────────────────────────┐
│ │
│ ┌──────────────────┐ ┌───────────────────────┐ │
│ │ tracebloc │ │ Kubernetes cluster │ │
│ │ client │◄────►│ │ │
│ │ │ │ ● Training jobs │ │
│ │ Orchestrates │ │ ● Inference jobs │ │
│ │ training, │ │ ● Your datasets │ │
│ │ enforces budgets │ │ ● Fine-tuned weights │ │
│ └────────┬──────────┘ │ │ │
│ │ │ Everything stays here │ │
│ │ └───────────────────────┘ │
└────────────┼────────────────────────────────────────────┘
│
│ Encrypted (orchestration only — no data)
▼
┌─────────────────┐
│ tracebloc │
│ backend │
│ │
│ Coordinates │
│ experiments, │
│ serves web UI │
└─────────────────┘

What the client manages

• Training execution — runs vendor models in isolated, containerized sandboxes
• Compute budgets — enforces per-vendor FLOPs or runtime quotas
• Security boundaries — namespace isolation, encrypted communication, audit logging
• Multi-framework support — PyTorch, TensorFlow, custom containers
• Hardware scheduling — CPUs, GPUs, TPUs via Kubernetes-native orchestration

Security

For the threat model, defense layers, per-platform caveats, operator responsibilities, and verification steps, see docs/SECURITY.md. The chart ships hardened defaults against untrusted user-submitted ML code; deployment still requires a CNI that enforces NetworkPolicy — that file explains exactly what to check.

Deploy

This repo ships the tracebloc unified Helm chart — one chart for AKS, EKS, bare-metal, and OpenShift. See client/Chart.yaml or the releases page for the current chart version.

Quick install

A single command provisions a Kubernetes cluster, auto-detects and installs GPU drivers (NVIDIA or AMD), deploys the tracebloc client, and installs the tracebloc CLI (tracebloc dataset push). Use this when you don't already have a cluster — the result is a full client install, not a demo.

macOS / Linux

bash <(curl -fsSL https://tracebloc.io/i.sh)

Windows(PowerShell as Administrator)

irm https://tracebloc.io/i.ps1 | iex

The installer pulls helper scripts from this repo at runtime — see scripts/install-k8s.sh and scripts/install-k8s.ps1. Those scripts are pinned to an immutable release tag and each is verified against a cosign-signed manifest before it runs; the install fails closed if verification can't complete (it never silently runs unverified code). See docs/SUPPLY_CHAIN.md for the integrity model and how to verify a release by hand.

Kubernetes version, and why an existing cluster keeps its old one. New installs create the cluster on the pinned, validated k3s version (currently v1.36.3-k3s1 — the single source of truth is scripts/spec/facts.env). k3s's version is fixed when the cluster is created and cannot be changed on a running one, so an existing cluster stays on whatever it was born with, even across correctly-pinned re-runs. The installer detects that and prints the version it found alongside the pin, with the recreate command — it warns, it does not refuse, so a re-run on an older cluster keeps working. To actually move onto the pinned version you have to recreate the cluster — and release the secure environment first, because it is anchored to the cluster's identity and deleting the cluster first strands it on your dashboard for good:

tracebloc delete --keep-data # releases this secure environment; keeps your local data
k3d cluster delete tracebloc # then re-run the installer

(Nothing installed on this machine yet? Then just the k3d line.) This is the same two-step the installer itself prints when it detects a drifted cluster. Which data survives depends on the storage mode, selected by TB_STORAGE_MODE: in hostpath mode your data stays on the host under HOST_DATA_DIR and is rebound to the new cluster; in node-local mode (RFC-0003 Option C) the data lives inside the node and is destroyed with it. The installer refuses to continue if it cannot see your data directory from inside the new nodes, so a recreate cannot silently start writing into the node instead of onto your disk (#817).

Reinstalling on a machine that still holds data. A new install (one that creates a fresh cluster) will not silently adopt data left behind by an earlier install. If it finds existing data under HOST_DATA_DIR (default ~/.tracebloc, both the flat and per-release layouts), it stops and asks you to choose reuse / wipe / a different directory. In-place upgrades that keep the existing cluster are unaffected — their data stays by design. For non-interactive runs pass --reuse-data, --wipe-data, or --data-dir=<path> (with no choice and no terminal the install aborts rather than adopting). This is the RFC-0003 offboard-hygiene guard (#376).

Helm install

For existing Kubernetes clusters:

helm repo add tracebloc https://tracebloc.github.io/client
helm repo update
helm install my-tracebloc tracebloc/client \
--namespace tracebloc --create-namespace \
-f my-values.yaml

Full deployment guide → docs/INSTALL.md (prerequisites, required values, upgrade & rollback, air-gapped install).

Ingest a dataset

Once the client is running, get a dataset into your cluster's local MySQL with ~8 lines of YAML and a single helm install. No Dockerfile, no Python script — the platform owns the official image, you describe what you want ingested.

The flow is two steps. First, stage your raw files on the cluster's shared PVC (client-pvc by default, mounted at /data/shared/ inside the ingestor Pod). The chart doesn't transport data into the cluster — it points at data the cluster can already see. The simplest pattern is a throwaway kubectl cp Pod that mounts the PVC; the chart README links the manifest.

Second, describe the dataset and install:

# my-cats-dogs.yamlapiVersion: tracebloc.io/v1kind: IngestConfigcategory: image_classificationtable: cats_dogs_trainintent: traincsv: /data/shared/cats-dogs/labels.csvimages: /data/shared/cats-dogs/images/label: label

helm install my-cats-dogs tracebloc/ingestor \
--namespace tracebloc \
--set-file ingestConfig=./my-cats-dogs.yaml

The ingestor runs once, validates the data, copies files into the destination directory on the PVC, inserts rows into the cluster's MySQL, sends metadata to the tracebloc backend — then exits. The chart artifacts (ConfigMap + post-install hook Job) become inert; nothing keeps running. Repeat per dataset.

Full ingestor docs → ingestor/README.md (data staging patterns, every supported category, the schema, the update model, verification, override knobs).

Topic	Where to look
Production install + required values	docs/INSTALL.md
Ingest a dataset (declarative YAML)	ingestor/README.md
Available ingestion categories + example YAMLs	tracebloc/data-ingestors templates
Threat model & operator responsibilities	docs/SECURITY.md
Migrating from eks-1.0.x / aks-* charts to client-1.x	docs/MIGRATIONS.md
Per-tenant migration runbook	docs/migration-tools/README.md
Per-platform value mapping	client/MIGRATION.md

Platform-specific walkthroughs: Linux · macOS · EKS · Azure / AKS

NetworkPolicy required. The chart's training-pod egress lockdown only takes effect on a CNI that enforces NetworkPolicy. See SECURITY.md § Per-platform caveats.

Links

Platform · Docs · Discord

License

Apache 2.0 — see LICENSE.

Deployment help?support@tracebloc.io or open an issue.

Pre-commit

Optional but recommended: pip install pre-commit && pre-commit install sets up the git hooks from .pre-commit-config.yaml. The hooks run automatically on each commit and are lint-only (ShellCheck at the same error-severity gate as CI) — nothing rewrites files, so scripts/manifest.sha256 always stays true to scripts/. They are a fast local guard — CI remains the guarantee.