Uh oh!
There was an error while loading. Please reload this page.
feat(data ingest): data-first flow inversion + family-scoped task picker (#180b) - #198
Merged
Merged
Conversation
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
LukasWodkaforce-pushed
the
feat/180a-task-split-name-renames
branch
from
July 9, 2026 13:29
ba01f9e to
b4e5678CompareLukasWodkaforce-pushed
the
feat/180b-data-first-inversion
branch
from
July 9, 2026 13:30
68fe49d to
e21a42dCompare…ker (#180b)
Invert the guided ingest flow to data-first (RFC-0002 §12.1) and rework
task selection so the user is never shown the flat 15-task wall.
- Prompt order is now intent → name → path → task → task-specific →
review. Name is asked with an example in the hint (not auto-filled).
- After the path, the family is sniffed from the layout and echoed
("Found a CSV table — this is tabular data."). The sniff is a HINT, not
a lock (§5.1): an ambiguous layout asks the family plainly, an explicit
--task skips the sniff entirely, and non-interactive runs never sniff.
SniffFamily mirrors the Discover* layout markers — it validates nothing,
so it can't fork or replace the real walk.
- Family-scoped task picker (§7): only that family's tasks are offered,
rendered "Display — one-liner · task_id", split into Available now vs a
greyed "Not yet in the CLI" (with the reason). The registry Label field
is finally wired up, plus the locked glosses (time_to_event_prediction →
"Survival analysis", masked_language_modeling → "fill-mask", seq2seq →
"translation / summarization") and a per-task blurb.
- Label question (§8) now SELECTs over the real CSV header row (exact
match kills the data-ingestors#340 case-mismatch silent-null-label
class), worded per task (class vs value-to-predict), skipped for
self-supervised text; falls back to free text if the header can't be
read. --number-of-keypoints / --time-column unchanged.
- Promptable overwrite (folded): a pre-existing destination table is a
y/N prompt in interactive mode (yes → replace, no → clean exit 0), still
a hard exit 6 non-interactively; a reused --idempotency-key falls
through to exit 6 to preserve the data-loss guard.
- Copy polish (folded from #179): data/ingest help says "workspace" and
"ingest", not "client" and "stage". Aliases unchanged.
Exit codes / classify / reclaim matrix and --output-json keys unchanged.
Closes#180
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>The ingest command's Long help described the normal successful flow in
Kubernetes terms ("submits an ingestion run to jobs-manager, and watches
the ingestor Job") - the exact ceremony RFC-0002 section 6 keeps off the
happy path. Reword it to plain language plus the on-prem reassurance. The
exit-code section and the --detach reconnect hint still name
jobs-manager/kubectl: those are failure/detach diagnostics, which section 6
explicitly permits.
Also fix the exit-6 duplicate-table error to say "in this workspace", not
"in this client" - matching the workspace wording used everywhere else in
the flow.
Copy-only; no behavior change. Tests + gofmt + vet green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>LukasWodkaforce-pushed
the
feat/180b-data-first-inversion
branch
from
July 9, 2026 14:32
e21a42d to
8de4ccbCompareUh oh!
There was an error while loading. Please reload this page.
…80b)
SniffFamily + previewLabelCSVPath were an over-permissive fork of the real
discovery walk. Align them to what Discover / DiscoverText / DiscoverTabular
actually accept so a confident sniff can never promise a layout the walk
refuses:
- Match the marker dirs (images/, texts/, sequences/) and labels.csv with
the literal, case-sensitive names the walk joins + Lstats. A mis-cased
"Images/" is no longer read as confident image. Only the .csv extension
match stays case-insensitive (mirrors DiscoverTabular's EqualFold).
- Require BOTH labels.csv AND the subdir before claiming confident image /
text (mirror Discover / DiscoverText); the echo now only names files that
were actually found.
- Require a directory for tabular: a bare .csv file is no longer confident
(DiscoverTabular rejects bare files; that's cli#181).
- previewLabelCSVPath now reuses DiscoverTabular's single-CSV rule via a
shared findSingleCSV helper, so a multi-CSV directory errors (caller falls
back to free text) instead of silently reading the alphabetical first.
Interactive flow:
- Path prompt copy is folder-oriented ("the folder holding it") — the walk
can't take a bare file yet, so don't promise one.
- Path prompt gets a validator that rejects an empty/whitespace answer, so a
bare Enter can't sniff (and ingest) the current working directory.
- Drop the dead famPrompted store in resolveFamily (pickTask sets prompted
unconditionally right after).
Also removes the unused FamilyOf accessor.
Tests pin: mis-cased dir not confident, missing labels.csv not confident
image/text, bare .csv not confident tabular, empty path rejected, multi-CSV
matches the walk.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>…80b)
The family sniff claimed confident tabular whenever a directory held at
least one CSV, but DiscoverTabular's findSingleCSV requires exactly one —
so a two-CSV directory got echoed "this is tabular data" and then the walk
rejected it. That made the sniff more permissive than the walk it mirrors,
breaking the sniff's own contract ("never claims more than the matching
Discover* would accept"). Gate the tabular case on csvCount == 1 so
sniff-confident-tabular holds only for directories the walk would accept
structurally; multi-CSV directories now fall through to the plain family
question. Adds a regression test asserting both the ambiguous sniff and the
walk rejection on the same input.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>…200) The Family<->noun mapping had three hand-maintained copies that had to stay in sync by hand: push.FamilyNoun (Family->noun), interactive.go's familyFromNoun (noun->Family), and resolveFamily's literal []string{"tabular","image","text"} picker options + "tabular" default. Consolidate to one ordered familyNounTable in internal/push. FamilyNoun, the new exported FamilyFromNoun, and FamilyNouns() (picker options + default) all derive from it. resolveFamily now takes its Select options and default from push.FamilyNouns() and its reverse lookup from push.FamilyFromNoun; the local familyFromNoun is gone. Behavior identical: same order (tabular, image, text), same default (tabular). build/vet/tests green. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…flag, honest text sniff echo (#201) - Trim the interactive path answer before storing it. validateDatasetPath only trims to check for emptiness, so a pasted " ~/data" (stray space) otherwise survived, defeated expandHome (first char isn't '~'), and made filepath.Abs prepend cwd — the family sniff / label-header preview then read a path that doesn't exist (and silently fell back to free-text label entry, the data-ingestors#340 class this feature exists to prevent). - Make SelfSupervised a CategorySpec registry field (set on MLM + CLM) and have SelfSupervisedText read it, so a new self-supervised task can't be added without deciding whether it needs a label column — was a hardcoded two-id switch decoupled from the registry. - Soften the confident text sniff echo to "looks like": texts/ and sequences/ map to DIFFERENT tasks (TextSidecarDir), so a family-level echo must not imply the task the user then picks will load. The walk stays the authoritative check. - Drop the unreachable empty-slice branch in defaultLabelChoice (its only caller already guards len(headers) > 0). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit dc2cfe2. Configure here.
Uh oh!
There was an error while loading. Please reload this page.
…s confident tabular (#203) (#204) A directory with labels.csv plus a mis-cased media folder (Images/, Texts/, Sequences/) fell through SniffFamily's confident-tabular branch: the subdir switch is case-sensitive so hasImages/hasText stayed false, labels.csv counted as the one CSV, and the flow confidently echoed "tabular", skipped the family question, and offered only tabular tasks. DiscoverTabular then ignores the subdir, reads labels.csv as the single CSV, and SUCCEEDS — silently ingesting an image/text dataset as a standalone table with the media files dropped and no error. Detect a subdir whose name matches a marker (images/texts/sequences) case-insensitively but not exactly, and bail to ambiguous so the flow asks the family plainly. Narrow: an unrelated subdir (backup/, raw/) still sniffs confident tabular, matching DiscoverTabular which ignores it. The confident image/text branches still require an EXACT match, mirroring the walk's literal os.Lstat. Strengthened the two mis-cased tests to assert the sniff is not confident AT ALL (they previously only checked "not confident image/text" and so missed the tabular masquerade), plus a case pinning that an unrelated subdir stays confident tabular. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal
approved these changes
Jul 9, 2026
Uh oh!
There was an error while loading. Please reload this page.
saadqbal added a commit
that referenced
this pull request
Jul 9, 2026
#180b (#198) landed the data-first flow inversion + family-scoped task picker on develop, which supersedes most of this branch: - The flat Select("Task", ...) picker — and its empty-default crash fix (#197 finding 1/2) — is gone, replaced by resolveFamily + pickTask. Resolved interactive.go to develop's new flow. - The "Destination table name" → "Dataset name" rename is superseded by develop's "What should we call this dataset?" prompt. Resolved the test conflicts to develop's rewritten new-flow suite (my old-flow tests, incl. the seed test, are obsolete). Kept the still-relevant copy fixes: - README quickstart --split → --intent (auto-merged). - --name flag help states the leading-letter rule (auto-merged), and I folded the same wording into develop's interactive name prompt so the prompt and the flag help agree (finding 4 drift, otherwise reintroduced by the merge). gofmt -l clean; go build ./..., go vet, go test ./internal/cli/... ./internal/push/... green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
saadqbal pushed a commit
that referenced
this pull request
Jul 10, 2026
…iguous, not confident tabular (#203) (#211) * fix(data ingest): warn instead of silently ingesting a mis-cased media folder as a table (#203) A dataset with labels.csv plus a mis-cased media folder (Images/, Texts/, Sequences/) already stays ambiguous rather than sniffing confident tabular (#198's miscasedMarker guard) — but the ambiguous path gave no clue why. So the user saw only a blind "which is it?" prompt with no hint that their folder was just mis-cased. Carry a plain-language hint on the ambiguous FamilySniff for that case, naming the folder and the lowercase form it should have (e.g. "Images" → "images"), and surface it via ui.Printer.Warnf in resolveFamily before the family question. Reuses the same marker set the case-sensitive walk keys on (markerFold, formerly isMarkerFold, now returns the canonical marker). Extend TestSniffFamily to cover all three mis-cased markers (adds the Sequences/ case), assert each stays ambiguous AND carries a hint naming the rename, and pin that an unrelated subdir stays confident tabular with no hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(data ingest): mis-cased media sniff tracks the walk per-filesystem (#203) Address the code-review findings on the mis-cased-media guard. - Cross-platform false positive: the guard flagged a mis-cased "Images/" next to labels.csv as ambiguous and told the user to rename it — but on a case-insensitive filesystem (macOS APFS, Windows) the walk's own os.Lstat(<dir>/images) resolves that folder, so Discover already accepts the layout. The CLI preflight runs on the user's own machine, so a valid layout was reported broken with an unnecessary rename instruction. Now the sniff probes the literal lowercase marker path the walk keys on (markerResolves): if it resolves, treat the folder as the real marker (confident media, no hint); only when it doesn't resolve (case-sensitive FS, e.g. Linux) is it the genuine #203 footgun — stay ambiguous + hint. - Test coverage: add TestResolveFamily_SurfacesMiscasedHint, pinning that resolveFamily surfaces the advisory hint through the printer before asking the family plainly (the PR's headline behavior, previously exercised by no test). FS-aware, so on case-sensitive CI deleting the Warnf branch fails it. - Make the SniffFamily mis-cased test FS-aware to match the new behavior, and drop the dead `&& s.Confident` assertion (unreachable after the preceding not-confident guard). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Stacked on #197 / #180a — merge after those. Base is
feat/180a-task-split-name-renames; this is the second half of cli#180 (the data-first flow inversion of RFC-0002). The--task/--namerenames shipped in #180a;--intentstayed as--intent.Closes#180
What changed
Inverts the guided
data ingestflow to data-first and reworks task selection so the user never faces the flat 15-task wall.intent → name → path → task → task-specific → review. Name is prompted with an example in the hint, not auto-filled.Found a CSV table — this is tabular data.). The sniff is a hint, not a lock — ambiguous layout asks plainly, an explicit--taskskips the sniff, non-interactive never sniffs.SniffFamilymirrors theDiscover*layout markers and validates nothing (Principle 6 — no forked validation).Display — one-liner · task_id, split into Available now vs greyed Not yet in the CLI (with reason). Wires the registryLabel(was dead code) + the locked glosses (time_to_event_prediction → Survival analysis,masked_language_modeling → fill-mask,seq2seq → translation / summarization) + a per-task blurb.--number-of-keypoints/--time-columnunchanged.y/Nprompt interactively (yes → replace, no → clean exit 0), still a hard exit 6 non-interactively; a reused--idempotency-keyfalls through to exit 6 (preserves the data-loss guard).data/ingesthelp says "workspace"/"ingest", not "client"/"stage". Aliases unchanged.Guardrails held
Exit-code / classify / reclaim matrix unchanged except the deliberate promptable-overwrite. No NULL-label path.
--output-jsonkeys unchanged. All output viainternal/uiPrinter methods.Tests
go build ./... && go test ./... && gofmt -l . (empty) && go vet ./...— all green. New coverage: prompt order, sniff echo + hint-not-lock + explicit-task-skips-sniff, family-scoped picker (excludes other families, shows pending, DS names + glosses), label-select-from-headers + per-task wording + self-supervised skip + free-text fallback, promptable overwrite (interactive y/n + non-interactive exit-6 + idempotency-key fall-through), and push-levelSniffFamily/PreviewLabelHeaders/DisplayNameglosses.🤖 Generated with Claude Code
Note
Medium Risk
Changes core ingest UX and overwrite semantics for interactive runs; preview/sniff logic must stay aligned with Discover* to avoid misleading users before the real walk fails.
Overview
Implements RFC-0002 data-first guided
data ingest: prompts run intent → name → path → task (then task-specific fields and review), with workspace/ingest copy updates in help text.After the path,
SniffFamilypreviews layout (read-only, aligned withDiscover*) and echoes confident cases; ambiguous layouts ask the family plainly.--taskskips sniff and the flat task list. The picker usesCategoriesByFamilywith display names, glosses, blurbs, and Available now vs Not yet in the CLI.Interactive label columns come from a SELECT on real CSV headers (
PreviewLabelHeaders), with regression vs classification wording and no label prompt for self-supervised text (SelfSupervisedon the registry).When a destination table already exists without
--overwrite, interactive TTY users get a replace y/N (no → exit 0); non-interactive /--output-jsonstill exit 6. Reused--idempotency-keynever gets the prompt (data-loss guard).findSingleCSVis shared so tabular sniff, header preview, andDiscoverTabularstay on the same single-CSV rule.Reviewed by Cursor Bugbot for commit 98789e7. Bugbot is set up for automated code reviews on this repo. Configure here.