RFC-0002 §12.11. Remove the tokenizer.json ingest requirement for masked_language_modeling.
Verified: the ingestor never reads it, the schema never required it, and the training side deliberately removed the dataset-staged tokenizer in #805 (it caused vocab divergence that broke weight averaging). The tokenizer is the collaborator's, supplied at model upload.
- Delete the MLM tokenizer branch in
DiscoverText (stop requiring/staging it), drop the manifest tokenizer field, fix the false "the ingestor reads it" comment, update the tests. CLI-only; low risk.
Design-of-record: RFC-0002 (#174). Part of the data-ingest redesign epic.
RFC-0002 §12.11. Remove the
tokenizer.jsoningest requirement formasked_language_modeling.Verified: the ingestor never reads it, the schema never required it, and the training side deliberately removed the dataset-staged tokenizer in #805 (it caused vocab divergence that broke weight averaging). The tokenizer is the collaborator's, supplied at model upload.
DiscoverText(stop requiring/staging it), drop the manifesttokenizerfield, fix the false "the ingestor reads it" comment, update the tests. CLI-only; low risk.Design-of-record: RFC-0002 (#174). Part of the data-ingest redesign epic.