Skip to content

fix(data ingest): mis-cased media folder next to labels.csv stays ambiguous, not confident tabular (#203) - #211

Merged
saadqbal merged 2 commits into
developfrom
feat/203-miscased-media-sniff
Jul 10, 2026
Merged

fix(data ingest): mis-cased media folder next to labels.csv stays ambiguous, not confident tabular (#203)#211
saadqbal merged 2 commits into
developfrom
feat/203-miscased-media-sniff

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Problem

When a dataset folder holds labels.csv next to a mis-cased media folder (e.g. Images/ instead of images/), the family sniff used to fall through to confident tabular and the guided flow silently ingested the tree as a table — dropping the images/texts. #203.

The first pass added a guard that flagged any mis-cased marker folder as ambiguous with a rename hint. Code review surfaced that this over-corrected on case-insensitive filesystems: on macOS (APFS) and Windows the walk's own os.Lstat(<dir>/images) resolves Images/, so Discover already accepts the layout. Since the CLI preflight runs on the user's own machine, a user with a perfectly valid layout was told it was broken and instructed to perform an unnecessary rename.

Fix

Make the sniff track the walk on the filesystem it runs on instead of guessing from the on-disk casing:

  • When a subdir matches a marker name only case-insensitively, probe the literal lowercase path the walk keys on (markerResolvesos.Lstat(filepath.Join(dir, "images")), mirroring Discover / DiscoverText).
  • If it resolves (case-insensitive FS): treat it as the real marker — confident media, no hint — exactly as the walk will accept it.
  • If it does not resolve (case-sensitive FS, e.g. Linux): this is the genuine data ingest: mis-cased media folder + labels.csv sniffs as confident tabular → silent mis-ingestion #203 footgun. Stay ambiguous and emit the rename hint so the flow guides the fix rather than silently ingesting a table.

The heuristic stays narrow and consistent with Discover* (RFC-0002 §5.1 Principle 6): the sniff never claims more than the matching walk would accept.

Acceptance

  • macOS/Windows, labels.csv + Images/: sniff is confident image, no rename hint, and Discover accepts the same tree.
  • Linux, labels.csv + Images/: sniff is ambiguous, carries a rename hint naming Imagesimages, and the family question is still asked.
  • A single CSV next to an unrelated subdir (e.g. backup/) stays confident tabular, no hint.
  • The advisory hint reaches the user through the printer before the family question.

Test plan

  • go build ./... && go test ./... && gofmt -l . && go vet ./... — all green locally (case-insensitive branch).
  • TestSniffFamily mis-cased cases are now FS-aware: they assert confident media on a case-insensitive FS and ambiguous + rename hint on a case-sensitive one, tied to the walk's own probe. Linux CI exercises the footgun branch.
  • New TestResolveFamily_SurfacesMiscasedHint pins that resolveFamily surfaces the hint via the printer before asking the family plainly — the PR's headline behavior, previously untested. Deleting the Warnf branch fails it on case-sensitive CI.
  • Dropped a dead && s.Confident assertion (unreachable after the preceding not-confident guard).

Closes#203

🤖 Generated with Claude Code


Note

Low Risk
Changes are limited to interactive family sniff/preview and CLI messaging; behavior aligns with existing Discover* walks rather than introducing new ingest rules.

Overview
Fixes #203, where labels.csv beside a mis-cased media folder (e.g. Images/) could sniff as confident tabular and silently drop media. The sniff now mirrors the real walk via markerResolves (os.Lstat on the lowercase marker path): on case-insensitive FS it treats the folder as a real marker (confident image/text, no hint); on case-sensitive FS it stays ambiguous, blocks the tabular masquerade, and sets FamilySniff.Hint with a rename message.

resolveFamily prints that hint with Warnf before the family question (hint stays advisory). Tests are FS-aware for mis-cased Images/Texts/Sequences, plus TestResolveFamily_SurfacesMiscasedHint for the CLI path.

Reviewed by Cursor Bugbot for commit 847ac11. Bugbot is set up for automated code reviews on this repo. Configure here.

LukasWodkaand others added 2 commits July 10, 2026 11:04
…a folder as a table (#203)
A dataset with labels.csv plus a mis-cased media folder (Images/, Texts/,
Sequences/) already stays ambiguous rather than sniffing confident tabular
(#198's miscasedMarker guard) — but the ambiguous path gave no clue why. So
the user saw only a blind "which is it?" prompt with no hint that their
folder was just mis-cased.
Carry a plain-language hint on the ambiguous FamilySniff for that case,
naming the folder and the lowercase form it should have (e.g. "Images" →
"images"), and surface it via ui.Printer.Warnf in resolveFamily before the
family question. Reuses the same marker set the case-sensitive walk keys on
(markerFold, formerly isMarkerFold, now returns the canonical marker).
Extend TestSniffFamily to cover all three mis-cased markers (adds the
Sequences/ case), assert each stays ambiguous AND carries a hint naming the
rename, and pin that an unrelated subdir stays confident tabular with no hint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
#203)
Address the code-review findings on the mis-cased-media guard.
- Cross-platform false positive: the guard flagged a mis-cased "Images/"
next to labels.csv as ambiguous and told the user to rename it — but on
a case-insensitive filesystem (macOS APFS, Windows) the walk's own
os.Lstat(<dir>/images) resolves that folder, so Discover already accepts
the layout. The CLI preflight runs on the user's own machine, so a valid
layout was reported broken with an unnecessary rename instruction. Now
the sniff probes the literal lowercase marker path the walk keys on
(markerResolves): if it resolves, treat the folder as the real marker
(confident media, no hint); only when it doesn't resolve (case-sensitive
FS, e.g. Linux) is it the genuine #203 footgun — stay ambiguous + hint.
- Test coverage: add TestResolveFamily_SurfacesMiscasedHint, pinning that
resolveFamily surfaces the advisory hint through the printer before
asking the family plainly (the PR's headline behavior, previously
exercised by no test). FS-aware, so on case-sensitive CI deleting the
Warnf branch fails it.
- Make the SniffFamily mis-cased test FS-aware to match the new behavior,
and drop the dead `&& s.Confident` assertion (unreachable after the
preceding not-confident guard).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@saadqbal
saadqbal self-requested a review July 10, 2026 10:06
@saadqbal

Copy link
Copy Markdown
Collaborator

Reviewed at high effort — clean PR 👍

Checked the one thing that matters here: markerResolves faithfully mirrors Discover's os.Lstat(<dir>/images) + IsDir() probe (symlink diff is a no-op since Lstat gives IsDir()==false), so the sniff never claims more than the walk accepts. Switch ordering keeps the !miscasedMarker guard on the tabular case, so the #203 masquerade stays ambiguous+hint on case-sensitive FS and confident on case-insensitive — matching what the walk will actually do. FS-aware tests pin both branches, including the Linux-CI footgun. Right depth, not a bandaid.

@saadqbal
saadqbal merged commit 10e6d89 into developJul 10, 2026
20 checks passed
@saadqbal
saadqbal deleted the feat/203-miscased-media-sniff branch July 10, 2026 10:36
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@LukasWodka@saadqbal