Skip to content

feat(push): add time_series_classification category (backend#1054 WS2) - #216

Merged
LukasWodka merged 1 commit into
developfrom
feat/tsc-category
Jul 10, 2026
Merged

feat(push): add time_series_classification category (backend#1054 WS2)#216
LukasWodka merged 1 commit into
developfrom
feat/tsc-category

Conversation

@LukasWodka

@LukasWodkaLukasWodka commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What / Why

CLI workstream (WS2) of the time-series-classification epic (tracebloc/backend#1054): make data ingest --task time_series_classification work end to end against the sequence-grouped contract WS1 implements in the ingestor.

time_series_classification is the [redacted]-sepsis shape: a single CSV with fixed sequence_id / timestamp columns (Decision-2), where each sequence_id groups the timestep rows of ONE sequence and the label is constant within it — one class per whole sequence, and the platform counts the dataset in sequences, not rows (Decision-3).

Scope

  • internal/push/category.go — one registry line: {ID: "time_series_classification", Family: FamilyTabular, Label: "Time-series classification", CLISupported: true, IsClassification: true}. Not RegressionClass → plain string label form (no label.policy). --task help, the interactive picker, and the accept-gate all derive from the registry, so no other listing edits were needed.
  • Re-vendored internal/schema/ingest.v1.json + layout.v1.json via the scripts/sync-schema.sh mechanism: scripts/.data-ingestors-ref bumped to the WS1 branch head (c38c8ad). The registry↔schema parity tests (TestRegistryCoversSchemaCategories / TestRegistryWithinSchema) pin the new enum both ways and are green.
  • internal/push/layout_contract.go — parse the contract's new grouping trait (Decision-4) and expose GroupingFor; all grouped preflight behaviour gates on the trait, never on a category id.
  • internal/push/preflight.go
    • label-diversity gate extended from the hardcoded tabular_classification to IsClassification(category) (mirrors the ingestor's is_classification wiring; covers TSC).
    • CheckSequenceSchemaColumns: sequence_id + timestamp must be declared in the schema (previews the ingest.v1 sequence-grouped conditional).
    • CheckSequenceRows: every timestep row must carry a non-empty sequence id (previews SequenceGroupValidator's null-id rule; NA sentinels count as null, pandas parity). Also yields the distinct-sequence count, echoed as an advisory note ("N sequence(s) … counts this dataset in sequences, not rows").
  • Parity harness — 6 new tsc-* cases; goldens regenerated from the REAL WS1 validators (DATA_INGESTORS_DIR=/tmp/tsc_ws1, branch of data-ingestors#359). All pre-existing golden verdicts/values byte-identical. Two deliberate divergences documented in the manifest: tsc-label-flip (mid-sequence label flip — LabelConstantWithinGroupValidator rejects in-cluster, no local whole-group preview yet) and tsc-unsorted-timestamp (PerGroupTimeOrderedValidator, same gap). Candidate follow-up if burned uploads show up.
  • README — category listing updated (also folds in the 5 text tasks the line had fallen behind on: 15 of 16 supported).
  • Deliberately untouched (fixed-names Decision-2): spec.go, flags, interactive prompts — buildTabular already emits the correct string-label tabular spec for TSC.

Coupling

Requires tracebloc/data-ingestors PR tracebloc/data-ingestors#359 to land in the same release window (vendored schema comes from that branch — T16).scripts/.data-ingestors-ref pins that PR's head SHA; once #359 merges, bump the ref to the merge commit and re-run scripts/sync-schema.sh (the PR-branch SHA stays fetchable after a squash merge, so CI stays green either way).

Test plan

Verified locally:

  • go build ./..., go vet ./..., gofmt -l clean, go test -race -cover ./... — all packages pass; scripts/coverage-floor.sh ok (push 84.6%).
  • scripts/sync-schema.sh --check green against the new pinned ref (the exact CI invocation).
  • Registry/parity/layout-contract pins: TestRegistryKnownCategories, TestSupportedCategories (now 15), TestRegistryCoversSchemaCategories, TestRegistryWithinSchema, TestRegistryMirrorsLayoutContract, new TestGroupingForMirrorsContract.
  • New unit tests: TestCheckSequenceSchemaColumns, TestCheckSequenceRows, TestPreflightDataset_SequenceGrouped (dispatch-level: accept + sequence-count note, schema-missing reject, null-id reject, single-class reject, and time_series_forecasting unaffected), TestBuild_Tabular_PassesSchema/time_series_classification, TestBuild_TSC_SchemaConditionalRequiresSequenceColumns (proves the vendored conditional actually enforces Decision-2).
  • TestValidatorParity: goldens regenerated from the real WS1 validators; all 33 cases pass; zero drift in pre-existing verdicts/values.
  • Done-contract check: drove the real pipeline (DiscoverTabular → InferSchema → PreflightDataset → SpecArgs.Build → embedded-schema validation) over WS1's sample CSV (templates/time_series_classification/time_series_classification_sample_in_csv_format.csv, 6 patients / 30 rows / nullable lactate). Preflight accepts with the "6 sequence(s)" note; the produced ingest.yaml passes BOTH the WS1 branch's canonical ingest.v1.json (python jsonschema) and WS1's full real validator chain (map_validators("time_series_classification", …)) — schema + sequence-group + label-constancy + per-group order + numeric-columns + label-diversity all green.

Deferred to CI / dev:

  • The kind-cluster integration e2e suite (nightly / e2e label) — not run locally.
  • An actual in-cluster data ingest push against a live workspace — needs a dev cluster with the WS1 ingestor image; belongs to the epic's FR-on-dev stage.

Part of tracebloc/backend#1057 (epic tracebloc/backend#1054).

🤖 Generated with Claude Code


Note

Medium Risk
Touches ingestion validation and couples release to data-ingestors#359; two parity divergences mean some bad TSC CSVs can pass local preflight but fail after upload.

Overview
Adds time_series_classification to data ingest so sequence-grouped tabular CSVs (fixed sequence_id / timestamp, one label per sequence) can be pushed end to end. The category registry marks it CLI-supported and classification-shaped; README now reflects 15 of 16 supported tasks.

Vendored contracts (ingest.v1.json, layout.v1.json) and scripts/.data-ingestors-ref are re-synced from the upstream WS1 branch: new enum value, schema conditional requiring sequence_id + timestamp, and a grouping trait on layout v2. GroupingFor reads that trait so grouped behavior is not hardcoded to a category id.

Preflight gains CheckSequenceSchemaColumns and CheckSequenceRows (null/NA sequence ids, sequence count note). Tabular label-diversity now gates on IsClassification (covers TSC, not only tabular_classification). Six tsc-* parity cases and goldens document two known gaps: mid-sequence label flips and per-group timestamp order are still accepted locally but rejected in-cluster.

Reviewed by Cursor Bugbot for commit 8c24142. Bugbot is set up for automated code reviews on this repo. Configure here.

Wire the sequence-grouped time-series task end to end on the CLI side:
- category registry: time_series_classification, FamilyTabular,
CLISupported, IsClassification (mirrors the ingestor registry's
is_classification=True). Not RegressionClass: real class labels use
the plain string label form, no label.policy. --task help and the
interactive picker derive from the registry, so both pick it up
without further edits. No spec.go / flag / prompt changes — the
sequence_id / timestamp column names are fixed (Decision-2).
- re-vendor ingest.v1.json + layout.v1.json from the WS1 branch
(data-ingestors#359) and bump scripts/.data-ingestors-ref to its
head SHA so sync-schema.sh --check stays green. The registry<->schema
parity tests pin the new enum both ways.
- layout contract: parse the new grouping trait (Decision-4) and expose
GroupingFor; grouped preflight behaviour gates on the trait, never on
a category id.
- preflight: extend the label-diversity gate from a hardcoded
tabular_classification to IsClassification, and add the cheap local
sequence checks — CheckSequenceSchemaColumns (fixed sequence_id /
timestamp columns declared in the schema, previewing the ingest.v1
conditional) and CheckSequenceRows (no null/empty sequence ids,
previewing SequenceGroupValidator; also yields the sequence count,
echoed as a note since the platform counts sequences, not rows —
Decision-3).
- parity harness: 6 new tsc-* cases with goldens generated from the
REAL WS1 validators (existing goldens unchanged). tsc-label-flip and
tsc-unsorted-timestamp document the two deliberate divergences (no
local whole-group label-constancy / per-group order preview yet).
- README: category listing updated (15 of 16 supported).
Part of backend#1054 (WS2, backend#1057). Requires data-ingestors#359
in the same release window (T16) — the vendored schema comes from that
branch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@LukasWodkaLukasWodka self-assigned this Jul 10, 2026
@LukasWodka
LukasWodka requested a review from saadqbalJuly 10, 2026 12:20
@LukasWodka
LukasWodka merged commit ebe36b9 into developJul 10, 2026
20 checks passed
@LukasWodka
LukasWodka deleted the feat/tsc-category branch July 10, 2026 12:54
LukasWodka added a commit that referenced this pull request Jul 10, 2026
Closes the T16 coupled release opened by #216: the vendored schema was
pinned to the WS1 PR-branch head pre-merge; data-ingestors#359 is now
merged, so the pin moves to the merge commit on develop
(7b4ecac21ee491998ea4252daace9a5af6a1cb4a). sync-schema.sh confirms
both vendored files (ingest.v1.json, layout.v1.json) are byte-identical
to the merge commit - no vendored content changes, ref-only.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@LukasWodka@divyasinghds@saadqbal