fix(data): drop the tokenizer.json ingest requirement for MLM (#184)

[LukasWodka]

Summary

masked_language_modeling ingestion required a tokenizer.json at the dataset root, staged it, and showed it in the local summary. This was a false gate: the ingestor never read it, the ingest.v1 schema never required it, and the training side deliberately stopped staging a dataset tokenizer in #805 (a dataset-staged tokenizer diverged the vocab and broke weight averaging). The tokenizer is the collaborator's, supplied at model upload.

Closes#184. Design-of-record: RFC-0002 §12.11 (#174).

What changed

• internal/push/text.go — DiscoverText no longer requires/stages tokenizer.json for MLM. An MLM dataset with just labels.csv + sequences/*.txt is accepted. A tokenizer.json left in the directory is simply not walked, so it is ignored gracefully (never an error). Fixed the false "the ingestor reads it" doc comment.
• internal/push/walk.go — removed LocalLayout.ExtraFiles (struct field + FileCount) and its stale comments.
• internal/push/stream.go — removed the now-unreachable ExtraFiles packaging loop; fixed the sortedKeys comment.
• internal/cli/data.go — removed the dead tokenizer row from the local-summary panel.
• internal/push/text_test.go — replaced TestDiscoverText_MLM_RequiresTokenizer with TestDiscoverText_MLM_NoTokenizer (accepted without a tokenizer; a stray one is ignored, not counted, not an error). Dropped the dead ExtraFiles assertion in the classification test.

Design decision for the reviewer to ratify

ExtraFiles (a generic "single root-level file beyond labels.csv" staging map on LocalLayout) was populated solely by the MLM tokenizer branch — no other category used it, and its only documented example was the tokenizer. Rather than leave an always-empty map iterated in the stream hot path plus a dead summary branch, I removed the whole mechanism. If a future category needs root-level extras, re-adding it is trivial. If you would rather keep it as an extension point, I can instead keep the field and only drop the tokenizer branch — say the word.

Out of scope / deferred: RFC-0002's own tables (docs/rfcs/0002-data-ingest-flow.md lines ~90/147/341/407) still describe the old "tokenizer.json required" behavior. I left the RFC untouched because it is under active revision on its own branch (docs/rfc-0002-data-ingest-flow); those tables should be reconciled there to avoid a merge collision. Also note: whether MLM should still require labels.csv at all (the RFC says MLM has "no labels") is a separate question, not touched here.

Test plan

go build ./... # ok
go test ./... # all packages ok (internal/push, internal/cli included)
gofmt -l . # empty
go vet ./... # clean

🤖 Generated with Claude Code

---

Note

Low Risk
Behavior change relaxes a false CLI gate and stops uploading an unused file; no auth or cluster security paths are touched, and existing MLM layouts without tokenizer.json become valid.

Overview
Masked language modeling no longer requires or uploads tokenizer.json at the dataset root. DiscoverText only validates labels.csv plus the text sidecar (texts/ or sequences/); a leftover tokenizer.json in the folder is ignored (not staged, not counted, not an error), matching ingest behavior after #805.

The LocalLayout.ExtraFiles mechanism is removed end-to-end: struct field and FileCount in walk.go, the tar packaging loop in stream.go, and the tokenizer row in the ingest local summary (data.go). Tests now assert MLM works without a tokenizer and that stray tokenizers do not change FileCount.

^{Reviewed by Cursor Bugbot for commit f131a67. Bugbot is set up for automated code reviews on this repo. Configure here.}