fix(push): reject image datasets with no "filename" column in dry-run (#371)

[LukasWodka]

Summary

Closes#371.

tb data ingest for image tasks green-lit a labels.csv whose file column wasn't named filename (e.g. image_id): the dry-run printed ✔ Dry-run complete, then the real ingest failed cluster-side with exit 9 (file_transfer.py:182 | No filename found in record, 0 records ingested). An accept-locally / fail-after-upload parity gap (epic tracebloc/backend#1142 Theme D; sibling of #352).

Root cause: image preflight resolved the file column via imageFileColIndex, which fell back to column 0 when neither filename nor data_id was present — reading the label values as filenames and passing. The cluster reads each image via a case-sensitive record.get("filename") at transfer time and accepts exactly one key, filename (it never aliases another column), so image_id/data_id resolve to nothing. The ingestor's own preflight (validate_data) doesn't catch this, which also blinded the golden harness (goldens come from validate_data, which accepts image_id,label).

What changed

• Enforce requires_filename_column up front for the image family (CheckImageFilenameColumn) — reject in the dry-run with an actionable message, mirroring the text family's existing check.
• Drop the data_id false-accept + column-0 fallback in imageFileColIndex (require literally filename, case-insensitive + trimmed, like the ingestor's _match_column). data_id was never a real alias — it fails at transfer identically to image_id.
• Fixtures: corrected all 17 imgc-* fixtures to filename; added imgc-no-filename-col to pin the divergence (ingestor preflight accepts, CLI rejects the doomed upload); regenerated goldens.json against the pin.
• Fixed the unit-test CSVs and the walk.go help text.

Type

Bug fix.

Test plan

• go test ./... — green.
• go test ./internal/push/ -run TestValidatorParity — green, incl. the new imgc-no-filename-col case.
• PYTHON=<di-venv> DATA_INGESTORS_DIR=<pinned> scripts/sync-validator-goldens.sh --check → "validator goldens in sync". The regeneration diff was surgical: only the new case added + imgc-label-missing's golden error text corrected (image_id, target → filename, target); no other verdict/value changed.
• End-to-end with the built binary:
  • image_id,label dataset → dry-run now rejects at "Step 1/3 Check your data" (exit 3), before touching the workspace: "labels.csv has no "filename" column (columns: image_id, label) — image tasks match each row to its file by that column … Rename your file column to "filename" and re-run."
  • filename,label dataset → dry-run passes (exit 0), "✔ Dry-run complete".

Checklist

• Tests added/updated (CheckImageFilenameColumn, imageFileColIndex, CrossCheckLabels, new parity case)
• Goldens regenerated against the pin (--check in sync)
• gofmt/go vet clean
• Verified end-to-end with the real binary

Related

• Follow-up data ingest: "filename" column resolved case-insensitively at preflight but case-sensitively at transfer (#340-class) #372 — the filename column is matched case-insensitively at preflight but read case-sensitively at transfer (chore: back-merge main → develop (adopt cobra 1.10.2 + reconcile drift) #340-class asymmetry), so Filename passes preflight yet fails transfer. Best fixed ingestor-side; out of scope here.
• Unrelated to feat(doctor): plain-language redesign (connected + ready, concrete fixes, --diagnose) #365 (doctor redesign). Does not touch data ingest: preflight parity gaps (semseg masks, DataValidator types) + stale pin — accept-then-reject #352's scope.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes image preflight validation and dry-run behavior for all image tasks; stricter than before but prevents known post-upload failures. No auth or cluster execution path changes beyond earlier local rejection.

Overview
Fixes #371: image ingest dry-runs could pass when labels.csv used image_id or other file columns, then fail cluster-side at transfer with exit 9.

CheckImageFilenameColumn enforces the layout contract (requires_filename_column): manifests must have an exact lowercase filename column (whitespace-trimmed). Wrong names (image_id, data_id) and case variants (Filename) are rejected with actionable errors before upload.

imageFileColIndex no longer falls back to column 0 or treats data_id as an alias—only filename matches, aligned with case-sensitive record.get("filename") at transfer. PreflightDataset runs manifest/header checks (including the new one) before per-image decode for fail-fast behavior.

Tests and parity fixtures use filename headers; new parity cases imgc-no-filename-col and imgc-uppercase-filename-col document deliberate CLI-reject / ingestor-accept gaps.

^{Reviewed by Cursor Bugbot for commit 8ea2cdb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[saadqbal]

Good, careful fix and the parity-pin approach is nice. A few things worth checking before merge — flagged inline. Main one: this reverses the data_id handling from #207, so I'd want the ingestor source confirming data_id-only really fails transfer, otherwise it's a new false-reject. Also the case-insensitive match leaves the Filename variant still failing after upload (the same bug class), and we decode every image before this cheap header check runs.

[saadqbal]

/fr-pass