fix(dataset push): make live ingestion work end-to-end

[LukasWodka]

Summary

The first real-cluster run of tracebloc dataset push (against a live tracebloc/client release) surfaced three issues the mock-only test suite couldn't catch. This PR fixes all three. End-to-end validated: image_classification, 6/6 records, 100% success, rows confirmed in MySQL, metadata sent to the backend API.

Fixes

1. Staging collided with the ingestor's DEST_PATH (the blocker)

The CLI staged source files into /data/shared/<table> — exactly the ingestor's DEST_PATH (data-ingestors config.DEST_PATH = STORAGE_PATH/TABLE_NAME). The ingestor's DuplicateValidator rejects a pre-existing, non-empty destination, so the CLI's own staging tripped the check on every run (Destination directory '/data/shared/<table>' already exists).

→ StagedPrefix now stages under /data/shared/.tracebloc-staging/<table>, leaving the destination fresh. Added FinalDestPrefix for the real destination shown in pre-flight.

2. Image target_size had no override

image_classification defaults to 512x512 and the ingestor validates the resolution (it does not resize) — so any other size hard-failed the Image Resolution Validator with no escape from the CLI.

→ Added --target-size WxH, plus auto-detect from the first image when omitted (emitted as spec.file_options.target_size).

3. Watch treated a broken log stream as fatal

A Pod replaced / restarted / deleted mid-follow (e.g. a backoffLimit retry) returned exit 9 even when the Job ultimately succeeded.

→ The Job's terminal status is now the source of truth; a non-ctx stream error is fatal only when the Job outcome can't be determined.

Test plan

• go build ./... && go vet ./... && go test ./... — all green.
• New/updated unit tests: push/detect_test.go (ParseTargetSize, DetectImageSize); spec_test.go (staging≠dest regression + target_size passes schema); watch_test.go (Job status wins on stream break); stream_test.go (derives paths from StagedPrefix).
• Live: built the binary and ran dataset push end-to-end against a real client → staged → submitted → ingestor Job ran → INGESTION SUMMARY 100%; 6 rows confirmed in training_test_datasets.clidemo_clean_train.

Notes / out of scope

• The live run also re-confirmed two environment/chart issues not addressed here (tracked separately): the pinned INGESTOR_IMAGE_DIGEST is amd64-only (arm64 nodes can't pull it), and the jobs-manager SA needs system:auth-delegator for TokenReview. Neither is a CLI bug — the CLI submits and watches correctly.
• The README still says "Pre-alpha, Phase 0"; a docs refresh + v0.1 release is the planned follow-up.

🤖 Generated with Claude Code

[LukasWodka]

👋 Heads-up — Code review queue is at 18 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#98 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• client#171 — Installer: verify readiness & credentials before reporting success; RHEL-family support · author: @LukasWodka · no reviewer assigned
• client-runtime#45 — Staging: Enhance jobs-manager with HTTP proxy, ingestion endpoint, and fixes · author: @saadqbal · no reviewer assigned
• client-runtime#61 — docs: record MySQL credential threat-model decision · author: @saadqbal · no reviewer assigned
• data-ingestors#132 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• data-ingestors#133 — docs: fix declarative-ingest path/column drift (issue R7: adopt-and-backfill the live in-cluster client instead of minting an orphan #131 A-series) · author: @divyasinghds · no reviewer assigned
• data-ingestors#138 — test(e2e): end-to-end ingestion equivalence suite (all modalities, real MySQL) · author: @LukasWodka · no reviewer assigned
• data-ingestors#139 — fix(ingest): MLM tokenizer + relax PascalVOC difficult + reserved-id guard (client create: auto-generate the client name (<firstname>-NN) and make --location optional #137, fix(cli): advertise data ingest/data delete in runtime + help strings #135) · author: @LukasWodka · no reviewer assigned
• design-system#19 — fix: un-track coverage/ and node_modules/ from git · author: @LukasWodka · no reviewer assigned
• design-system#22 — ci: add Vitest test workflow · author: @LukasWodka · reviewer: @aptracebloc

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

[aptracebloc]

Approve ✅ — live-validated end-to-end

Tested the whole stack from this branch's binary against a real tracebloc/client release (EKS dev, chart 1.4.2, amd64).

This PR's three fixes, verified:

• Staging path — files land under /data/shared/.tracebloc-staging/<table> (confirmed on the PVC); the ingestor's DEST_PATH stays fresh, so DuplicateValidator passes on a real run. No collision.
• --target-size + auto-detect — auto-detected 64x64 from the first image; explicit --target-size/W,H parsing validated, bad values → exit 2.
• Watch resilience — a live run hit Retrying (attempt 1) and the watch still reported success from the Job's terminal status → exit 0, 240/240 ingested.

Also confirmed: multi-file tar stream, SIGINT-safe stage-Pod cleanup (no orphans), and the full exit-code contract (2/3 guards all fire correctly).

Note (non-blocking): the Schema drift check is red here only because the embedded-schema re-sync lands in #15 (this branch doesn't touch ingest.v1.json). It's an UNSTABLE/non-required check and goes green once #15 merges. Merging in stack order #12→#13→#14→#15.

[LukasWodka]

👋 Heads-up — Code review queue is at 26 / 8

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

• averaging-service#98 — ci: add fr-gate caller workflow · author: @LukasWodka · no reviewer assigned
• cli#13 — feat(dataset push): tabular / time-series modality family · author: @LukasWodka · no reviewer assigned
• cli#14 — feat(dataset push): text family + generic sidecar staging · author: @LukasWodka · no reviewer assigned
• cli#15 — feat(dataset push): object_detection + keypoint_detection · author: @LukasWodka · no reviewer assigned
• cli#16 — test(cli): coverage wins (preflight/progress/errors) + smoke-test hardening · author: @LukasWodka · no reviewer assigned
• cli#17 — test(cli): integration harness for the real-I/O seams (kind e2e) · author: @LukasWodka · no reviewer assigned
• client-runtime#45 — Staging: Enhance jobs-manager with HTTP proxy, ingestion endpoint, and fixes · author: @saadqbal · no reviewer assigned
• client-runtime#61 — docs: record MySQL credential threat-model decision · author: @saadqbal · no reviewer assigned
• client-runtime#65 — fix(release: promote develop → main (toward v0.3.1 — install.sh PATH fix) #64): re-sync jobs-manager ingest schema (accept masked_language_modeling) + anti-drift · author: @LukasWodka · no reviewer assigned
• client-runtime#67 — ci: publish jobs-manager images on merge (closes the deploy-delivery gap) · author: @LukasWodka · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)