Skip to content

Sync ingest schema + wire sentence_pair_classification & embeddings into CLI category registry #162

Description

@saadqbal

Problem

The Schema drift check CI job (scripts/sync-schema.sh --check) is failing on every open PR (including #160), because internal/schema/ingest.v1.json has drifted from upstream tracebloc/data-ingestors@master. Upstream added:

  • two new task categories: sentence_pair_classification, embeddings
  • a content_hashdata_id strategy
  • related description updates

The drift predates and is unrelated to #160 (it's stale on develop too).

Why it's not just a schema bump

Running scripts/sync-schema.sh alone breaks TestRegistryCoversSchemaCategories: the CLI's authoritative categoryRegistry (internal/push/category.go) must cover every schema category, and the two new ones aren't wired in.

Plan

  • scripts/sync-schema.sh to vendor the current upstream schema.
  • Add sentence_pair_classification + embeddings to categoryRegistry as FamilyText (both stage raw .txt from texts/ per data-ingestors conventions.pyTEXT_CATEGORIES).
  • Mark both CLISupported: false with an UnsupportedNote (conservative default, mirroring causal_language_modeling/seq2seq/token_classification) until the tracebloc ingest discover/build staging path for them is confirmed. Reviewer: flip to true if the push path already supports them.
  • Update the pinned lists in category_registry_test.go.

Acceptance

  • scripts/sync-schema.sh --check passes.
  • go build/vet/test ./... green.
  • Unblocks the schema-drift check across open PRs once merged to develop.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions