Skip to content

Add schema migration to supervisor-child comm - #67235

Merged
uranusjr merged 1 commit into
apache:mainfrom
astronomer:supervisor-schema-migration
May 22, 2026
Merged

Add schema migration to supervisor-child comm#67235
uranusjr merged 1 commit into
apache:mainfrom
astronomer:supervisor-schema-migration

Conversation

@uranusjr

@uranusjruranusjr commented May 20, 2026

Copy link
Copy Markdown
Member

With foreign language SDKs, it may be possible the two sides of supervisor comm have different versions. This adds a migration layer at the supervisor (server) side, so an SDK (client) using a lower version of the schema may be able to communicate to the server.

This adds: (subpoints describe what each component is for, but those are not covered in this PR)

  1. A JSON schema file to describe the models used by the supervisor. This is automatically kept up-to-date with pre-commit hooks.
    • This should be published with versions for SDKs to use.
    • An SDK should use a version of this schema to talk to the supervisor.
  2. A structure to host Cadwyn migration files inside execution_time.
    • Whenever the supervisor schema changes, a migration should be added to the SDK.
  3. The supervisor use the migrations to upgrade the SDK’s message to match the schema version used by the supervisor, and downgrade messages before they are sent to the SDK to match the schema version used by the SDK.

@uranusjruranusjr mentioned this pull request May 20, 2026
Merged
1 task
@uranusjr
uranusjrforce-pushed the supervisor-schema-migration branch from 7cd6fc3 to ec65643CompareMay 20, 2026 09:36
@uranusjruranusjr added area:coordinator Coordinator: The interface to spawn Lang-SDK subprocesses and removed backport-to-v3-2-test labels May 20, 2026
ashb
ashb reviewed May 20, 2026
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/schema.json
ashb
ashb reviewed May 20, 2026
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/versions/__init__.py
@uranusjr
uranusjrforce-pushed the supervisor-schema-migration branch from ec65643 to d057c20CompareMay 20, 2026 10:52
jason810496
jason810496 approved these changes May 20, 2026
View reviewed changes

@jason810496jason810496 left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM to introduce the automated mechanism to snapshot the schema between supervisor and the subprocesses and the migrator to the main branch first.

kaxil
kaxil reviewed May 20, 2026
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/supervisor.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/supervisor.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/migrator.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/migrator.py Outdated
Comment threadtask-sdk/tests/task_sdk/execution_time/schema/test_integration.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/migrator.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/migrator.py Outdated
Comment threadtask-sdk/src/airflow/sdk/execution_time/schema/migrator.py Outdated
Comment threadtask-sdk/.pre-commit-config.yaml Outdated
Comment threaduv.lock
@github-actions

This comment was marked as resolved.

Sign in to view
@uranusjr
uranusjrforce-pushed the supervisor-schema-migration branch 5 times, most recently from 82f666b to c971bf6CompareMay 22, 2026 06:21
@uranusjr
Add schema migration to supervisor-child comm
e28a191
With foreign language SDKs, it may be possible the two sides of
supervisor comm have different versions. This adds a migration layer at
the supervisor (server) side, so an SDK (client) using a lower version
of the schema may be able to communicate to the server.
@uranusjr
uranusjrforce-pushed the supervisor-schema-migration branch from c971bf6 to e28a191CompareMay 22, 2026 06:54
@jason810496jason810496 moved this to In review in Java-SDKMay 22, 2026
@uranusjr
uranusjr requested review from ashb and kaxilMay 22, 2026 10:33
@uranusjr
uranusjr merged commit 0920c77 into apache:mainMay 22, 2026
143 checks passed
@uranusjr
uranusjr deleted the supervisor-schema-migration branch May 22, 2026 19:53
@github-project-automationgithub-project-automationBot moved this from In review to Done in Java-SDKMay 22, 2026
shivaam added a commit to shivaam/airflow that referenced this pull request May 24, 2026
@shivaam
ts-sdk: generate wire types from supervisor schema (PRapache#67235)
f54fd43
Adopt the upstream wire schema from the supervisor's Cadwyn migration
framework. Vendor schema.json into ts-sdk/schema/, generate type atoms
via json-schema-to-typescript, and wrap them in protocol.ts with the
narrowing patterns the generated types can't express on their own
(optional discriminators, fields the supervisor passes through but
the schema snapshot doesn't yet list, string-typed enums, and
optional-per-schema-but-required-per-API-validator mismatches).
Expose SUPERVISOR_API_VERSION so callers can read which Cadwyn version
the SDK was generated against. Logged at runtime startup for operator
visibility; not sent on the wire — the supervisor learns the SDK's
schema version out-of-band (bundle metadata) and runs its migrator
accordingly.
shivaam added a commit to shivaam/airflow that referenced this pull request May 24, 2026
@shivaam
docs: ts-sdk roadmap specs — CI workflow, wire-contract tracking, e2e…
6bb80ba
… tests
Three design docs for the next three ts-sdk PRs, written before any
implementation so the structure and acceptance criteria are agreed up
front:
- 2026-05-23-ts-sdk-ci-workflow-design.md (Phase B, first to ship)
Standalone .github/workflows/ts-sdk.yml runs typecheck + vitest +
build on ts-sdk/** changes. Sets the polyglot-SDK CI precedent that
a future java-sdk.yml can mirror.
- 2026-05-23-ts-sdk-wire-contract-tracking-design.md (Phase A)
Builds on commit f54fd43 (schema vendoring + codegen +
SUPERVISOR_API_VERSION exposure). Remaining: a prek hook that detects
drift between our vendored schema and upstream PR apache#67235's
task-sdk/.../schema.json; an airflow-metadata.yaml convention TS
bundles ship to surface their schema version to the
TypescriptCoordinator; a separate upstream PR teaching the
TypescriptCoordinator to read it.
- 2026-05-23-ts-sdk-e2e-integration-tests-design.md (Phase C)
Mirrors PR apache#65959's java_sdk_tests/ layout exactly under
airflow-e2e-tests/tests/airflow_e2e_tests/ts_sdk_tests/. Five
pytest scenarios (happy path, client APIs, handler failure, unknown
task, schema-version handshake). Bundle built in-container by
ts_sdk_build.sh; breeze gets --sdk ts flag analogous to --sdk java.
shivaam added a commit to shivaam/airflow that referenced this pull request May 24, 2026
@shivaam
ts-sdk: refresh doc set for current state (post-apache#67235, post-co…
51b0c19
…ordinator)
Update README, COORDINATOR, REFACTOR_NOTES, REVIEW_OBSERVATIONS,
SDK_COMPARISON, and TESTING to reflect:
- Cadwyn supervisor-schema framework merged upstream as PR apache#67235;
ts-sdk pinned to api_version 2026-06-16 via SUPERVISOR_API_VERSION.
- TypescriptCoordinator (PR apache#65958, the coordinator-mode launcher) is
the production model — replaces the earlier "companion operator
package" framing.
- README leads with both Edge and Coordinator modes side by side, since
both are first-class and share the TaskClient interface.
- SDK_COMPARISON corrects out-of-date entries (getConnection, DAG
parser status); adds Cadwyn migration row to the cross-SDK table.
- REVIEW_OBSERVATIONS fix list reconciled against current code.
shivaam added a commit to shivaam/airflow that referenced this pull request Jun 2, 2026
@shivaam
Add TypeScript Task SDK for Apache Airflow
a70ee21
Adds `ts-sdk/` — a TypeScript SDK for authoring and running Airflow tasks
in Node.js. Supports two execution modes:
- **Coordinator mode** (AIP-108): spawned as a subprocess by
`TypescriptCoordinator`; communicates over TCP with the Airflow supervisor
using the same 4-byte length-prefixed msgpack protocol as the Python and
Java SDKs. Supports all 13 `TaskClient` APIs (Variables, XCom,
Connections, Dag operations, secret masking) and DAG serialization.
- **Edge Worker mode** (AIP-69/72): long-lived HTTPS worker using JWT
authentication; picks up tasks from the edge executor queue.
Wire types are generated from the supervisor schema (introduced in
PR apache#67235) at `ts-sdk/schema/supervisor-schema.json`, pinned to
`SUPERVISOR_API_VERSION = "2026-06-16"`.
Includes unit tests (180 tests) and integration test workers.
@shivaamshivaam mentioned this pull request Jun 2, 2026
Merged
1 task
@phanikumvphanikumv added this to the Airflow 3.3.0 milestone Jun 5, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner

Assignees

No one assigned

Labels

area:coordinatorCoordinator: The interface to spawn Lang-SDK subprocessesarea:dev-toolsarea:task-sdk

Projects

Java-SDK
Status: Done

Milestone

Java SDK 1.0 GA

Development

Successfully merging this pull request may close these issues.

6 participants

@uranusjr@ashb@kaxil@jason810496@vatsrahul1001@phanikumv