Add TypeScript coordinator runtime

[shivaam]

Adds the TypeScript coordinator runtime to @apache-airflow/ts-sdk, the
follow-up to #67908 (TypeScript task SDK public interface). This lets Airflow
execute registered TypeScript task handlers through a bundled Node.js entrypoint.

• Adds the coordinator runtime that speaks the supervisor wire protocol
  (length-prefixed msgpack frames) over the --comm and --logs sockets:
  connects, receives the task startup message, dispatches to the registered
  handler for the Dag/task pair, and reports the terminal task state.
• Adds TaskClient-backed task-time data access: Variables (get / getOrThrow),
  XCom (get / set, including the automatic return_value push for non-undefined
  handler returns), and Connections.
• Adds cooperative cancellation via ctx.signal (AbortSignal); SIGTERM/SIGINT
  abort the signal with a grace period before force-exit.
• Adds generated supervisor wire types (src/generated/supervisor.ts) plus a
  generate:supervisor script, and a ./coordinator package export subpath.
• Adds an example/ showing a Python stub Dag + TypeScript handlers, bundled
  with esbuild, and documents the bundle layout and Airflow coordinator config.

This is intentionally still Python-stub-Dag mode: TypeScript Dag declaration is
not introduced here. TypeScript registers handlers for Dag/task IDs declared by
Python stub tasks. The example bundles with esbuild and writes
airflow-metadata.yaml manually; folding metadata generation into the build
script is left to a separate follow-up.

Testing performed:

• Unit tests and full e2e test with the airflow coordinator. Readme to run tests is available under examples.

Next steps

• Combine airflow-metadata.yml with the .mjs file and introduce a ts pack tool.

[shivaam]

@uranusjr@jason810496 PR for TS sdk runtime.

[uranusjr]

Need to fix the static check failure obviously but otherwise very nice work

[shivaam]

Need to fix the static check failure obviously but otherwise very nice work

Thank you. I have been spending the last couple of months understanding the Task SDK execution path, edge worker work, and coordinator model, so it’s really encouraging to see this coming together.

I’ve pushed a follow-up for the static check failure.

[jason810496]

Nice! LGTM overall, thanks.
Btw, does it mean that with this PR, the TS-SDK "works" (at least able to execute the Multi-Lang Dag e2e in breeze)?
I didn't test it manually.

[shivaam]

Nice! LGTM overall, thanks. Btw, does it mean that with this PR, the TS-SDK "works" (at least able to execute the Multi-Lang Dag e2e in breeze)? I didn't test it manually.

Yes, the multi-lang dag will work after this PR. Just the bundler and packer needs to be implemented so you will have to create airflow-metadata.yml file. The readme in the example describes how to run it

[jason810496]

Nice! Here're all the follow-up issues that I saw during the review so far. I prefer to have real issues to keep track of them instead of having follow-up statements in the review comments that might be missed in the long run.

• #69286 - TypeScript SDK: add a ts pack bundle build tool
• #69287 - TypeScript SDK: Dag/task-aware multi-bundle selection in NodeCoordinator
• #69288 - TypeScript SDK: native TypeScript Dag declaration (remove Python stub requirement)
• #69289 - TypeScript SDK: coordinator schema-mapping tests (camelCase to snake_case)
• #69290 - TypeScript SDK: Airflow-level integration tests for coordinator-mode tasks
• #69291 - TypeScript SDK: refactor shared logic between NodeCoordinator and ExecutableCoordinator
• #69292 - TypeScript SDK: log socket buffering and reconnect hardening

[uranusjr]

(Note that native dag declaration should wait on AIP-82 Dag Importers)

[jason810496]

(Note that native dag declaration should wait on AIP-82 Dag Importers)

Yes, I just added at the top of the issue.

[shivaam]

Nice! Here're all the follow-up issues that I saw during the review so far. I prefer to have real issues to keep track of them instead of having follow-up statements in the review comments that might be missed in the long run.

• TypeScript SDK: add a ts pack bundle build tool #69286 - TypeScript SDK: add a ts pack bundle build tool
• TypeScript SDK: Dag/task-aware multi-bundle selection in NodeCoordinator #69287 - TypeScript SDK: Dag/task-aware multi-bundle selection in NodeCoordinator
• TypeScript SDK: native TypeScript Dag declaration #69288 - TypeScript SDK: native TypeScript Dag declaration (remove Python stub requirement)
• TypeScript SDK: coordinator schema-mapping tests (camelCase to snake_case) #69289 - TypeScript SDK: coordinator schema-mapping tests (camelCase to snake_case)
• TypeScript SDK: Airflow-level integration tests for coordinator-mode tasks #69290 - TypeScript SDK: Airflow-level integration tests for coordinator-mode tasks
• TypeScript SDK: refactor shared logic between NodeCoordinator and ExecutableCoordinator #69291 - TypeScript SDK: refactor shared logic between NodeCoordinator and ExecutableCoordinator
• TypeScript SDK: log socket buffering and reconnect hardening #69292 - TypeScript SDK: log socket buffering and reconnect hardening

Thank you @jason810496 . I will get to working on these tasks one by one. Curious, is there a task that you think is higher priority out of these?