Skip to content

refactor(cli): extract resolveLocalInput + connectIngestTarget from runDataIngest - #309

Closed
LukasWodka wants to merge 1 commit into
refactor/282-data-go-file-splitfrom
refactor/283-rundataingest-extract
Closed

refactor(cli): extract resolveLocalInput + connectIngestTarget from runDataIngest#309
LukasWodka wants to merge 1 commit into
refactor/282-data-go-file-splitfrom
refactor/283-rundataingest-extract

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Summary

The risky one of the WS-B chain: runDataIngest was 586 lines; its two big stanzas are now named functions, moved verbatim:

  • resolveLocalInput (→ data_ingest_local.go) — steps 0–4 + the P3 content preflight: the --overwrite/--idempotency-key guard, intro banner, guided prompts, ~-expansion + the existence-first check, table-name/category/misapplied-flag validation, the layout walk, per-category spec resolution, spec synthesis + schema validation, and the local summary. Takes *runDataIngestArgs and mutates it in place, so the --output-json error defer and the cluster steps see the resolved spec exactly as the inline code did.
  • connectIngestTarget (→ data_ingest_cluster.go) — steps 5–8a: kubeconfig resolve, release + shared-PVC discovery, the verbose cluster summary, and the destination-table guard including the folded interactive replace decision (still flips a.Overwrite).

The jsonEmitted/named-return defer dance stays inside runDataIngest, next to the err it reads — both extracted functions feed that named return, and the dry-run stop (which sets jsonEmitted) also stays inside. The error-JSON contract is unchanged.

The only in-stanza edits are what the new signatures force: multi-value returns on error paths, :== where a named result already exists, &aa where a is already a pointer. Step order, output order, and exit codes untouched.

Targets hit:data.go 693 → 240 lines (<250 ✓); runDataIngest 586 → 138 lines (<150 ✓).

Fixes#283
Part of tracebloc/backend#1106.

Type of change

  • Bug fix
  • New feature
  • Refactor (no functional change)
  • Docs
  • CI/build

Test plan

Ran locally:

Checklist

  • Tests pass locally
  • gofmt/vet clean
  • No behavior change intended; no test file touched

Stacked PR 2/4 of the WS-B data.go chain — merge order: cli#303 (#282) → this → cli#284 → cli#285. Branch is based on refactor/282-data-go-file-split; until #303 merges, this diff shows its commit too. Review only the top commit (a2464db).

🤖 Generated with Claude Code


Note

Low Risk
Mechanical move-only refactor with explicit parity claims and unchanged tests; risk is limited to subtle wiring mistakes (named returns, defer, cancelled/overwrite), not new product behavior.

Overview
Splits the oversized runDataIngest flow into two named helpers so data.go stays under the line budget, with no intended behavior change.

resolveLocalInput (data_ingest_local.go) now owns all pre-cluster work: guards, banner, interactive prompts, validation, layout discovery, spec build/schema check, local preflight, and summary. It still mutates *runDataIngestArgs and returns cancelled for clean interactive aborts.

connectIngestTarget (data_ingest_cluster.go) owns kube discovery, verbose cluster summary, and the existing-table guard (including setting a.Overwrite from the replace prompt). It returns cancelled when the user declines replace.

runDataIngest keeps the --output-json error defer, dry-run, staging, and runIngestionRun orchestration; it only calls the two helpers and handles their cancelled paths.

Reviewed by Cursor Bugbot for commit a2464db. Bugbot is set up for automated code reviews on this repo. Configure here.

…unDataIngest
runDataIngest was 586 lines; every pre-cluster step and the cluster
pre-flight now live in two named stanza functions, moved verbatim:
- resolveLocalInput (data_ingest_local.go) — steps 0–4 + the P3 content
preflight: flag guard, banner, guided prompts, ~-expansion +
existence-first check, table-name/category/misapplied-flag validation,
layout walk, per-category spec resolution, spec synthesis + schema
validation, local summary. Mutates a in place (via *runDataIngestArgs)
so the --output-json defer and the cluster steps see the resolved spec
exactly as before.
- connectIngestTarget (data_ingest_cluster.go) — steps 5–8a: kubeconfig
resolve, release + PVC discovery, verbose cluster summary, and the
destination-table guard (incl. the folded interactive replace decision,
which still flips a.Overwrite).
The jsonEmitted/named-return defer dance stays INSIDE runDataIngest,
next to the err it reads — both extracted functions feed that named
return, so the error-JSON contract is unchanged. The dry-run stop (which
sets jsonEmitted) also stays inside.
Only in-stanza edits are what the new signatures force: multi-value
returns on error paths, := to = where a named result already exists, and
&a -> a where a is already a pointer. Step order, output order, and exit
codes are untouched.
data.go: 693 -> 240 lines; runDataIngest: 586 -> 138.
Safety net: the #187 outcome-matrix table test + #264 mutation pins pass
untouched; full suite green under -race; coverage floors hold (82.7%).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@LukasWodka
LukasWodka requested a review from saadqbalJuly 14, 2026 12:30
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

👋 Heads-up — Code review queue is at 48 / 30

Above the WIP limit. The team convention is to review existing PRs before opening new work.

Open PRs currently in Code review (oldest first):

  • averaging-service#181 — feat(weights): normalize-on-read — mixed cycles average instead of rejecting (SEC-03 §8 step 2) · author: @shujaatTracebloc · no reviewer assigned
  • averaging-service#182 — feat(weights): averaging writes SafeTensors (TF + PyTorch) — SEC-03 §8 step 4 · author: @shujaatTracebloc · no reviewer assigned
  • backend#1079 — feat(global_meta): edge dataset_meta exposure + attributes contract at ingest (#924 G4a) · author: @divyasinghds · no reviewer assigned
  • backend#1086 — docs(rfc): SafeTensors weight-format migration — SEC-03 Phase 1 (RFC 0004) · author: @shujaatTracebloc · no reviewer assigned
  • backend#1093 — chore(deps): bump django from 5.2.14 to 5.2.15 · author: @dependabot · no reviewer assigned
  • backend#1095 — feat(experiment): configurable preprocessing knobs incl. tabular scaler — RFC 0003 L1 + L1b (#1094) · author: @LukasWodka · no reviewer assigned
  • backend#1100 — fix(boot): pin SDK install to tracebloc==0.11.2, drop 404 dev line (#1098) · author: @LukasWodka · reviewer: @saqlainsyed007
  • backend#1105 — perf(api): query micro-fixes — notifications N+1, cached data_scientist, composite index, sampling (#975) · author: @aptracebloc · no reviewer assigned
  • backend#1107 — perf(experiments): prefetch ExperimentListSerializer relations + cache carbon intensity (#977) · author: @aptracebloc · reviewer: @saadqbal
  • cli#266 — main - > enhance CLI features and tests · author: @saadqbal · no reviewer assigned

Pull from review before opening new work. (This is a nudge from the kanban WIP check, not a block.)

@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

@BugBot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a2464db. Configure here.

@LukasWodka
LukasWodka changed the base branch from develop to refactor/282-data-go-file-splitJuly 14, 2026 12:49
@LukasWodkaLukasWodka self-assigned this Jul 14, 2026
@LukasWodka

Copy link
Copy Markdown
ContributorAuthor

@BugBot run

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a2464db. Configure here.

@saadqbalsaadqbal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Careful extraction — passing *runDataIngestArgs correctly threads the in-place a.Spec/a.Overwrite mutations, the explicit (cancelled, err) returns preserve the interactive-cancel + error paths, and err feeding the named return keeps the reclaim defer firing. Test + all builds green = behavior preserved. (Stacked on #303.)

@saadqbal
saadqbal deleted the branch refactor/282-data-go-file-splitJuly 14, 2026 14:30
@saadqbal

Copy link
Copy Markdown
Collaborator

Heads-up: this got auto-closed when I squash-merged its base #303 with branch deletion — GitHub closes a PR when its base branch is deleted, and it can't be reopened (base ref gone). Your work is safe on refactor/283-rundataingest-extract. To recover: rebase that branch onto develop (drop the now-duplicated #303 commits) and open a fresh PR. #315/#316 stack on this. Apologies for the disruption.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@LukasWodka@saadqbal