feat(data ingest): wire the 5 text tasks (#182)

[LukasWodka]

Summary

Wires the five CLI-pending text tasks into data ingest — token classification, sentence-pair classification, causal language modeling, seq2seq, and embeddings — so the whole text family is pushable, not just text classification and masked language modeling. Discovery, staging, spec-building, the interactive picker, and preflight all learn these tasks by reading the vendored layout contract (layout.v1.json), never by forking the ingestor's rules (RFC-0002 Principle 6). Data stays on-prem throughout; only metadata syncs.

This PR also folds in the verified review findings on the branch.

What changed

• layout contract vendored — internal/schema/layout.v1.json + layout_contract.go mirror data-ingestors' per-task on-disk layout. sync-schema.sh now vendors both ingest.v1.json and layout.v1.json from the pinned ref, drift-checked in CI.
• enforced record formats — sentence-pair (text_a<TAB>text_b) and embeddings (anchor<TAB>positive[<TAB>negative]) get a local structural check before staging, so a malformed file fails with a clear message instead of after the upload. The check is scoped to the files labels.csv actually references — mirroring the ingestor's TabSeparatedRecordValidator manifest walk — so a stray unreferenced .txt never blocks a layout the cluster would accept.
• label preflight parity — the text-family label-column and diversity previews now cover every supervised text task, gated on registry flags (!SelfSupervisedText for the label-column existence check, IsClassification for diversity) rather than a hardcoded text_classification. Token classification correctly skips the diversity check — the ingestor runs BIOLabelValidator on it, not LabelDiversityValidator.
• self-supervised vs supervised split — buildText emits label for exactly the supervised text tasks, keyed on the registry's SelfSupervised flag. TextSidecarDir is now read from the contract's primary_subdir instead of a hardcoded MLM special-case.
• sync-schema.sh hardening — signal-safe temp-file cleanup (EXIT/INT/TERM), and a failed write now returns non-zero instead of a false "wrote" + exit 0.

Test plan

• go build ./... — clean
• go test ./... — all green (added TestDiscoverText_EnforcedRecordFormat_IgnoresUnreferenced, TestPreflightDataset_TextLabelParity, plus the existing phase-4 discovery/record-format/spec tests)
• gofmt -l . — empty
• go vet ./... — clean
• scripts/sync-schema.sh --check — no drift against the pinned ref; write-failure path verified to exit non-zero

Layout-contract fields consumed

The CLI reads these from layout.v1.json so discovery/staging stays a verified mirror:

• primary_subdir — the text-file subdir to stage into (texts/, or sequences/ for MLM), via TextSidecarDir
• record_format (fields, min_fields, separator, enforced) — the per-.txt structure and whether the ingestor rejects a malformed file, via RecordFormatFor / ValidateTextRecord
• manifest.has_label_column — pinned against the registry's supervised/self-supervised split in layout_contract_test.go

Stacking

Was stacked on the #180b data-first-inversion work (#198). That has since squash-merged into develop, so this targets develop directly and is standalone — no ordering constraint left.

Deferred: semantic segmentation

Semantic segmentation stays CLI-pending on purpose. di#136 shipped the mask sidecar, but the ingestor doesn't yet populate the mask_id link column the contract requires, and the training-side sign-off is tracked in backend#816. The registry keeps it schema-recognized-but-unsupported with that note, so the push gate explains why. Wire it once #816 lands.

Part of #182.

---

Note

Medium Risk
Broad changes to ingest discovery, spec emission, and preflight across the text family; correctness depends on staying in sync with the vendored layout contract, though tests and CI drift checks mitigate that.

Overview
Enables data ingest for the five previously CLI-pending text tasks — token classification, sentence-pair classification, causal LM, seq2seq, and embeddings — so the full text family is pushable except semantic segmentation (still blocked on ingestor mask_id + backend#816).

Discovery, spec building (buildText labels via SelfSupervised), preflight, and the interactive picker all follow the vendored layout.v1.json contract (synced with ingest.v1.json in scripts/sync-schema.sh) instead of hardcoded per-task rules. TextSidecarDir now comes from the contract’s primary_subdir; a new IsClassification registry flag drives label-diversity preflight so token classification skips diversity (BIO tags) while supervised class tasks do not.

Local validation before upload: enforced tab-separated shapes for sentence-pair and embeddings mirror the ingestor’s TabSeparatedRecordValidator, only on files referenced in labels.csv (stray .txt files do not fail discovery). Shared CSV helpers (openCSVReader, matchColumnIndex) dedupe BOM-aware reads and column matching.

Tests and registry expectations were updated (14 supported categories; image picker still shows semantic segmentation as “not yet in the CLI”).

^{Reviewed by Cursor Bugbot for commit a98e598. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit bcea12f. Configure here.}

[saadqbal]

Careful PR — the registry/contract/flag refactor mirrors the ingestor faithfully (verified SelfSupervised↔has_label_column, IsClassification, TextSidecarDir, buildText against data-ingestors). Findings below are mostly in the new record-format machinery, where the enforced check can silently no-op. Two candidates I checked turned out fine: text tasks resolve the filename by a namedfilename column (not positional like the image path), and token_classification correctly requires --label-column (BIOLabelValidator). #1 and #2 are the ones worth acting on.

[LukasWodka]

Thanks for the careful review, @saadqbal — all 11 addressed in a98e598. The core fix reworks the enforced text-record check to match the actual discovered files (mirroring the ingestor's manifest walk) instead of reconstructing <stem>.txt, which closes all four fail-open paths at once.

Correctness (the record-format cluster):

1. Configured extension — validateTextRecords now indexes discovered files by lowercased basename + stem and resolves each manifest value against them; row a matches texts/a.text. No more hardcoded .txt.
2. Missing filename column — now a local hard error for the enforced tasks ("labels.csv has no filename column …"), mirroring the ingestor's rejection instead of silently validating nothing.
3. Case-insensitive basename — folded via Phase 1: embed ingest.v1.json + tracebloc ingest validate #1 (both maps + lookups lowercase); A.txt ↔ a.txt validates.
4. Tolerant manifest read — LazyQuotes=true (+ FieldsPerRecord=-1) so a pandas-tolerable row Go's strict reader would reject is read + validated, not dropped.

Robustness / convention:
5. layout_contract field-count message is now contract-driven (sepLabel()/shape()), no hardcoded "tab-separated"; pinning test updated.
6. sync-schema.sh captures curl's exit explicitly — a 404 now says "failed to fetch … (curl exited N)" instead of misdiagnosing "not valid JSON".
7. --tlsv1.2 added to the fetch (matches install.sh).
8. TextSidecarDir's unreachable fallback now panics with a drift message instead of silently returning "texts" (guarded by the mirror test + sync-schema --check).

Reuse:
9. Shared matchColumnIndex (exact → case-insensitive-trimmed) — one copy, used by CheckLabelColumn + the filename resolve.
10. Shared openCSVReader (open + BOM-strip + FieldsPerRecord=-1) — deduped across the 5 copies (ReadCSVHeader, CheckHasDataRows, CrossCheckLabels, readLabelColumnValues, and the new manifest reader). LazyQuotes stays opt-in per caller.
11. slices.Contains replaces containsInt.

New tests: TestDiscoverText_ConfiguredExtension / _MissingFilenameColumn / _CaseMismatchedBasename / _TolerantManifestRow. go build/test/vet/gofmt/sync-schema --check all green.

One thing I did NOT change (flagging for transparency): there's no text-family missing-file cross-check — a text row naming a file absent from disk is caught by the ingestor ("text file not found") but not in CLI preflight, whereas the image family has CrossCheckLabels. That's pre-existing (not introduced here) and out of this PR's scope; it likely belongs with the #1009 parity-harness work. Happy to file a follow-up if you'd like it tracked.