Problem
The Schema drift check CI job (scripts/sync-schema.sh --check) is failing on every open PR (including #160), because internal/schema/ingest.v1.json has drifted from upstream tracebloc/data-ingestors@master. Upstream added:
- two new task categories:
sentence_pair_classification, embeddings - a
content_hashdata_id strategy - related description updates
The drift predates and is unrelated to #160 (it's stale on develop too).
Why it's not just a schema bump
Running scripts/sync-schema.sh alone breaks TestRegistryCoversSchemaCategories: the CLI's authoritative categoryRegistry (internal/push/category.go) must cover every schema category, and the two new ones aren't wired in.
Plan
scripts/sync-schema.sh to vendor the current upstream schema.- Add
sentence_pair_classification + embeddings to categoryRegistry as FamilyText (both stage raw .txt from texts/ per data-ingestors conventions.pyTEXT_CATEGORIES). - Mark both
CLISupported: false with an UnsupportedNote (conservative default, mirroring causal_language_modeling/seq2seq/token_classification) until the tracebloc ingest discover/build staging path for them is confirmed. Reviewer: flip to true if the push path already supports them. - Update the pinned lists in
category_registry_test.go.
Acceptance
scripts/sync-schema.sh --check passes.go build/vet/test ./... green.- Unblocks the schema-drift check across open PRs once merged to
develop.
Problem
The
Schema drift checkCI job (scripts/sync-schema.sh --check) is failing on every open PR (including #160), becauseinternal/schema/ingest.v1.jsonhas drifted from upstreamtracebloc/data-ingestors@master. Upstream added:sentence_pair_classification,embeddingscontent_hashdata_idstrategyThe drift predates and is unrelated to #160 (it's stale on
developtoo).Why it's not just a schema bump
Running
scripts/sync-schema.shalone breaksTestRegistryCoversSchemaCategories: the CLI's authoritativecategoryRegistry(internal/push/category.go) must cover every schema category, and the two new ones aren't wired in.Plan
scripts/sync-schema.shto vendor the current upstream schema.sentence_pair_classification+embeddingstocategoryRegistryasFamilyText(both stage raw.txtfromtexts/per data-ingestorsconventions.pyTEXT_CATEGORIES).CLISupported: falsewith anUnsupportedNote(conservative default, mirroringcausal_language_modeling/seq2seq/token_classification) until thetracebloc ingestdiscover/build staging path for them is confirmed. Reviewer: flip totrueif the push path already supports them.category_registry_test.go.Acceptance
scripts/sync-schema.sh --checkpasses.go build/vet/test ./...green.develop.