feat(chart): perIngestionTables — RFC-0003 D16 enablement knob (backend#1205)

[LukasWodka]

Summary

The enablement knob for RFC-0003 D16 per-ingestion tables — the one wire the merged D-series (tracebloc/data-ingestors#408 → tracebloc/backend#1300 → tracebloc/tracebloc-engine#543 → cli#424 + tracebloc/client-runtime#222) was still missing: nothing ever set PER_INGESTION_TABLES on an ingestion Job.

• values.perIngestionTables (default false, schema-typed, heavily documented in values.yaml): renders PER_INGESTION_TABLES="1" onto the jobs-manager api container; the jobs-manager forwards it into every ingestion Job it spawns (companion: tracebloc/client-runtime#225).
• Conditional block — default installs render byte-identically (helm-unittest pins both sides: absent by default, "1" when set; 298/298 green).

Enablement sequence (dev first)

1. Merge this + the client-runtime companion; let dev images/deploys roll (dev pulls floating tags).
2. Flip perIngestionTables: true on the dev fleet values, helm upgrade.
3. Validate end-to-end with the ready-made tabular sample dataset: tb data ingest → confirm the summary carries the handle and the backend row has physical_table set → run a tabular training experiment (e2e harness) → confirm data_info.tables reaches the pod and training reads ds_<hex> → tb data delete → confirm table + salt + journal rows are gone and the dataset tombstones.
4. Staging → prod per fleet, same checklist. File-bearing categories stay refused under the flag until client-runtime#203 phase 2.

Epic: tracebloc/backend#1151 · Design: tracebloc/backend#1204

🤖 Generated with Claude Code

---

Note

Low Risk
Opt-in feature flag with default false and unittest guards; no behavior change until operators flip the value per environment.

Overview
Adds an opt-in Helm valueperIngestionTables (default false) to turn on RFC-0003 D16 per-ingestion physical tables. When set to true, the jobs-manager api container gets PER_INGESTION_TABLES=1, which jobs-manager is expected to forward into spawned ingestion Jobs so ingestors write immutable ds_<hex> tables instead of legacy shared label-named tables.

The env block is conditional so default installs stay unchanged. values.yaml, values.schema.json, and helm-unittest cases lock in absent-by-default vs PER_INGESTION_TABLES=1 when enabled.

^{Reviewed by Cursor Bugbot for commit 0362930. Bugbot is set up for automated code reviews on this repo. Configure here.}

[LukasWodka]

👋 Heads-up — Code review queue is at 36 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#222 — fix(sklearn): unwrap Pipeline to terminal estimator before merge dispatch (Prod: Set kubeconfig context namespace after Helm install for tracebloc  #209) · author: @divyasinghds · no reviewer assigned
• averaging-service#224 — chore: clear the ruff backlog (E4,E7,E9,F) · author: @LukasWodka · no reviewer assigned
• backend#1201 — docs(rfc): Data spaces & federated dataset composition (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1217 — docs(rfc): Dataset integrity & score reproducibility (draft) · author: @LukasWodka · reviewer: @saadqbal
• backend#1261 — RFC-BACKEND-0008: board & process reset (decision RFC — for markup) · author: @LukasWodka · reviewer: @saadqbal
• backend#1284 — docs(deploy): backend deploy runbook skeleton (#1263) · author: @LukasWodka · no reviewer assigned
• backend#1293 — fix(heartbeat): tolerate unknown experiment_key in training_info (#1287) · author: @divyasinghds · no reviewer assigned
• backend#1320 — feat(blob_store): migrate ModelCodeView to blob_store (#1162 PR3) · author: @saadqbal · no reviewer assigned
• backend#1321 — chore: clear house-rules findings · author: @LukasWodka · no reviewer assigned
• backend#1322 — fix(errors): replace bare except clauses with scoped handling (release-train: develop -> staging #614) · author: @saqlainsyed007 · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[LukasWodka]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit cdeb2e0. Configure here.}

[saqlainsyed007]

Review: perIngestionTables enablement knob

Overview

Small, well-scoped chart change adding an opt-in value perIngestionTables (default false). When enabled, it renders PER_INGESTION_TABLES="1" onto the jobs-manager api container, which forwards it into spawned ingestion Jobs (RFC-0003 D16). The env block is conditional, so default installs render byte-identically. Four files: template, unittest, schema, values.yaml docs.

Correctness ✅

• The conditional {{- if .Values.perIngestionTables }} with trim markers is placed correctly between sibling env entries at 8-space indent — no stray blank lines, clean YAML output.
• --reuse-values safe: a release predating this key resolves .Values.perIngestionTables to nil → falsy → env omitted. No nil-guard needed for a top-level scalar (unlike the (default dict ...) guards used for the nested images.ingestor reads nearby), so this is consistent, not a gap.
• Default-off is enforced in three places (values.yaml default, schema type, the "off" test) — the right belt-and-suspenders for a cross-stack flag.
• The notContains/contains assertions match the full {name, value} map element, so they can't be fooled by an unrelated env var — correct negative and positive coverage.

Conventions ✅

• Env entry, quoting (value: "1"), and inline comment style all match the existing block.
• Schema addition is a correct top-level sibling of ingestionAuthz. Since the schema sets no additionalProperties: false, it's documentation rather than a hard gate — consistent with the rest of the file.
• Docs are thorough and honest about the ordering constraint (backend/engine/jobs-manager must carry the D-series first).

Suggestions / nits

• values.yaml placement (minor readability).perIngestionTables: false is inserted directly under the # ==== Ingestion endpoint authorization (client-runtime#21) ==== banner, between that section's commented example and the real ingestionAuthz: key. It now visually reads as if it belongs to the authz section, which it doesn't — it's a separate D16 feature. Consider moving it above the authz banner, or giving it its own banner, so the authz doc block stays adjacent to the ingestionAuthz: key it documents.
• String contract is cross-repo (can't verify here). The exact "1" value is the wire contract consumed by jobs-manager / client-runtime#225. Worth confirming the companion reads == "1" (not a generic truthy parse) before the coordinated flip. Already sequenced in your merge plan — checklist item, not a code issue.
• Cosmetic: the test addition introduces a double blank line before the first new - it:. Harmless — drop one line if repushing anyway.

Risk

Low, and accurately self-described. No secrets, no new surface, no behavior change unless an operator explicitly flips the flag on a stack that already supports it. The byte-identical-default claim is backed by the passing unittest guard.

Verdict: Approve. Only the values.yaml placement is worth addressing before merge, and it's non-blocking.

🤖 Generated with Claude Code

[LukasWodka]

Both nits taken (0362930): perIngestionTables now sits above the authz section under its own ==== banner — the authz doc block is adjacent to its key again — and the double blank line in the test file is gone. On the string contract: confirmed on the companion (tracebloc/client-runtime#225) — jobs-manager forwards the raw value only when non-empty, and the ingestor's Config() parses truthiness (1/true/yes/on), so the contract is presence + truthy, with this chart emitting exactly "1" when enabled and omitting the var entirely when disabled (the conditional block — never a zeroed value). 298/298 helm tests green. Thanks for the approve!

bugbot run