Skip to content

feat(data ingest): flexible file-or-folder input + path fixes (#181) - #202

Merged
saadqbal merged 3 commits into
developfrom
feat/181-flexible-input
Jul 10, 2026
Merged

feat(data ingest): flexible file-or-folder input + path fixes (#181)#202
saadqbal merged 3 commits into
developfrom
feat/181-flexible-input

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary

RFC-0002 phase 3: let data ingest take a bare .csv as easily as a folder, make the per-family layout instructions concrete, and fix two ~/path-handling bugs. Data stays on-prem throughout — this only changes what the CLI accepts as input and how cleanly it fails on a typo; the ingestor contract is untouched.

Closes#181.

What changed

  • Bare .csv input for tabular / time-series.DiscoverTabular now accepts either a directory (its existing exactly-one-CSV rule) or a bare .csv file. Both resolve to the same staged layout — the CSV staged as the one labels.csv under the dataset — so the ingestor sees an identical contract. Media/label families (image, text) stay directory-only and reject a bare file with a clear message.
  • Sniff + copy restored.SniffFamily is confident-tabular for a bare .csv again (mirroring the now file-or-folder-capable walk), and the interactive prompt is back to "Where is your data? (file or folder)".
  • Per-family layout help. The ingest Long help and the data group help now spell out tabular (a .csv or a folder with one .csv), image (labels.csv + images/), and text (labels.csv + texts/) instead of only image. Positional arg renamed <local-path> to <dataset>.
  • ~user expansion + shared expander.expandHome resolves ~user / ~user/… via user.Lookup, not just ~ / ~/…. Both this and cluster.expandPath now delegate to a new leaf internal/pathutil.ExpandHome, so ~-expansion (incl. ~alice/.kube/config) is identical across every subcommand.
  • Path existence checked first. A typo'd path fails on the path (exit 3, "no such file or directory") before any spec/schema/family work — on the flag route and the guided route.

Review-finding fixes (second commit)

Three verified findings from review of the feature commit:

  • expandHome no longer mangles a ~user whose passwd home field is blank (user.Lookup succeeds with an empty HomeDir) — it returns the literal path, matching the documented contract and the current-user branch's existing empty-$HOME guard.
  • The "path existence FIRST" invariant now holds in guided mode too. Previously runInteractive sniffed the family and previewed the label header at a non-existent path before the guard fired, so the user answered the whole questionnaire against a bad path. The guard is extracted to statDatasetPath and runs before the family sniff; a typed exitError from a guided step propagates unwrapped (clean message, not "interactive setup: …").
  • expandHome and cluster.expandPath had drifted (~user on only one) — consolidated into pathutil.ExpandHome, the promote-to-pathutil plan the old comments kept deferring.

Test plan

  • go build ./... && go test ./... && gofmt -l . && go vet ./... — all green locally.
  • New/updated unit tests: bare .csv discovers + stages identically to a one-CSV dir; bare non-.csv and image/text bare files rejected; sniff confident-tabular on a bare .csv; "file or folder" copy restored; ~user expansion; nonexistent path beats the task gate with exit 3 on both routes; <dataset> arg name; new internal/pathutil contract tests.
  • Coverage floor holds: internal/cli 73.6% ≥ 68%, internal/submit 74.9% ≥ 72%.
  • Binary smoke: tracebloc data ingest /nope/typoed-path.csv --no-input … exits 3 with no such file or directory: "/nope/typoed-path.csv" — check the path to your dataset.

Stacked on #198 — merge after it

Base is feat/180b-data-first-inversion (PR #198), not develop. Merge #198 first, then this retargets to develop automatically.

Decisions to ratify

  • Bare-file staging approach. A bare .csv is staged as the one labels.csv under the dataset — byte-identical to the one-CSV-directory case — so no ingestor-side change is needed. The pre-flight "root" stays a directory (the file's parent) so the summary reads the same for both shapes. Confirm this CLI-side-convenience framing is what we want long-term (vs. teaching the ingestor about bare files).
  • ~user resolution is best-effort. An unknown/unlookupable user (incl. a static CGO-less build that can't read /etc/passwd) or a blank passwd home leaves the literal path so the existence check surfaces it plainly. Confirm we're fine with ~user silently not-expanding on a CGO-less binary rather than erroring.
  • Shared pathutil.ExpandHome.cluster.expandPath now also resolves ~user (previously it treated ~alice/… as a subdir of the current home). This is a behavior change for --kubeconfig ~alice/… — intended, but worth a nod.

Note

Low Risk
Changes are localized to CLI input validation, path expansion, and tabular discovery/staging parity; ingestor wire format is unchanged, with broad test coverage for ordering and layout edge cases.

Overview
data ingest now accepts tabular/time-series data as a bare .csv file or a folder with exactly one CSV. Staging still produces the same labels.csv layout as before, so the ingestor contract is unchanged; image and text tasks remain directory-only.

Path handling is tightened: missing paths fail first with exit 3 (statDatasetPath) before task validation or guided sniffing, and interactive exitErrors from that guard propagate without an interactive setup: wrapper. ~user paths resolve via shared internal/pathutil.ExpandHome, used by both data ingest and --kubeconfig (replacing duplicate expanders in cli and cluster).

UX/docs: positional arg renamed to <dataset>, help describes tabular vs image vs text layouts, and the guided path prompt again says “file or folder.”SniffFamily treats a lone .csv as confident tabular (but not symlinked CSVs), aligned with DiscoverTabular and shared isCSV.

Reviewed by Cursor Bugbot for commit b0d0244. Bugbot is set up for automated code reviews on this repo. Configure here.

Base automatically changed from feat/180b-data-first-inversion to developJuly 9, 2026 15:48
LukasWodkaand others added 2 commits July 9, 2026 21:09
…181)
RFC-0002 phase 3: flexible file-or-folder input, clearer instructions, and
two path bugs.
- Accept a bare .csv for the tabular / time-series family. DiscoverTabular
now takes either a directory (its existing exactly-one-CSV rule) or a
bare .csv file — both resolve to the SAME staged layout (the CSV staged
as the one labels.csv under the dataset), so the ingestor's contract is
unchanged. This is a CLI-side input convenience only; no data-ingestors
change. Media/label families (image, text) stay directory-only and error
clearly ("… is not a directory").
- Restore the sniff + copy #180b softened: SniffFamily is confident-tabular
for a bare .csv again (mirroring the now file-or-folder-capable walk), and
the interactive path prompt goes back to "Where is your data? (file or
folder)" with a matching hint.
- Per-family layout help: the ingest Long help + the data group help now
show tabular (a .csv OR a folder with one .csv), image (labels.csv +
images/), and text (labels.csv + texts/) instead of only image.
- Rename the positional arg <local-path> → <dataset> in Use/help.
- expandHome now expands "~user" / "~user/…" via user.Lookup, not just
"~" / "~/…"; an unknown/unlookupable user is left literal so the path
check reports it plainly.
- Check path existence BEFORE spec/schema/family validation, so a typo'd
path fails on the path (exit 3, "no such file or directory") instead of a
confusing downstream error (e.g. the task gate).
Tests: bare .csv discovers + stages identically to a one-CSV dir; bare
non-.csv and image/text bare files rejected; sniff confident-tabular on a
bare .csv; "file or folder" copy restored; ~user expansion; nonexistent
path beats the task gate with exit 3; <dataset> arg name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s + path-check ordering
Three verified review findings on the #181 flexible-input work:
- expandHome mangled a ~user whose passwd home field is blank: user.Lookup
succeeded with an empty HomeDir, so filepath.Join("", rest) produced a
relative path instead of the documented clear literal-path error. Guard
u.HomeDir == "" the same way the current-user branch already guards an
empty $HOME.
- The "path existence FIRST" invariant held only on the flag-only route.
In guided mode runInteractive sniffed the family and previewed the label
header at a non-existent path before the 0b guard ever fired, so the user
answered the whole questionnaire against a bad path. Extract the guard
into statDatasetPath and run it in runInteractive before the family
sniff; a typed exitError from a guided step now propagates unwrapped
(clean message, not "interactive setup: ...").
- expandHome (data ingest) and cluster.expandPath had drifted: #181 added
~user resolution to only one, so `--kubeconfig ~alice/.kube/config`
silently treated alice as a subdir of the current user's home. Consolidate
both into a shared internal/pathutil.ExpandHome (the promote-to-pathutil
plan the old comments kept deferring) so ~-expansion is identical across
every subcommand.
Coverage floor unaffected (internal/cli 73.6% >= 68%). New pathutil package
carries its own contract tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodka
LukasWodkaforce-pushed the feat/181-flexible-input branch from e01f011 to c946085CompareJuly 9, 2026 19:09

@cursorcursorBot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit c946085. Configure here.

Comment threadinternal/cli/interactive_test.go
…st key
Review polish on the #181 flexible-input work:
- SniffFamily no longer confidently sniffs a symlinked .csv as tabular.
os.Stat follows the link, but DiscoverTabular Lstats + rejectSymlinks
the CSV, so a symlinked .csv is a layout the walk REFUSES. The sniff
would lock the guided flow to tabular and then hard-fail on the walk's
symlink guard — breaking SniffFamily's "never claims more than the
matching Discover* would accept" contract. The bare-file branch now
Lstats and stays ambiguous for a symlink, so sniff and walk agree.
- Extract isCSV(name) and route all four .csv-extension checks through
it (findSingleCSV, DiscoverTabular's bare-file branch, and both
SniffFamily sites). The sniff and walk bare-file rules were copy-pasted
case-fold checks the comments insist must stay in lockstep; one helper
removes the drift risk.
- Fix a stale fakePrompter key in TestRunInteractive_TrimsPath: the path
prompt was renamed to "(file or folder)" everywhere except this one
answer-map key, which still read "(the folder holding it)", so the fake
returned "" and the test failed with "a dataset path is required".
Tests: new SniffFamily case asserts sniff + walk both refuse a symlinked
.csv. go build ./... && go test ./... && gofmt -l . && go vet ./... green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@saadqbal
saadqbal merged commit d57a6d5 into developJul 10, 2026
16 checks passed
@saadqbal
saadqbal deleted the feat/181-flexible-input branch July 10, 2026 07:01
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@LukasWodka@saadqbal