Skip to content

Add TypeScript task SDK public interface - #67908

Merged
uranusjr merged 8 commits into
apache:mainfrom
shivaam:feat/add-ts-sdk
Jun 28, 2026
Merged

Add TypeScript task SDK public interface#67908
uranusjr merged 8 commits into
apache:mainfrom
shivaam:feat/add-ts-sdk

Conversation

@shivaam

@shivaamshivaam commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

Adds a minimal @apache-airflow/ts-sdk package that defines the public TypeScript task-handler contract.

Summary:

  • Add explicit Dag-scoped task registration with separate dagId and taskId fields.
  • Add task context, task handler, and TaskClient interfaces.
  • Keep the first public surface aligned with the Go and Java SDKs: Variables, Connections, and XCom only.
  • Expose idiomatic TypeScript names such as dagId, taskId, runId, mapIndex, and connId; coordinator/runtime code will map those names to Airflow schema fields.
  • Add standalone package config, lint/format/test/build scripts, README, and focused Vitest coverage, including type-level assertions for the public API shape.

This PR has intentionally been narrowed to the user-facing SDK interface only. It does not include coordinator-mode execution, edge-worker execution, generated supervisor schema types, or Python coordinator registration. Those are intended as follow-up PRs built on this public interface.

Example intent:

fromairflow.sdkimportdag, task@dagdefsales_pipeline():
@task.stub(queue="typescript")defextract(): ...
@task.stub(queue="typescript")deftransform(extracted): ...
transform(extract())
sales_pipeline()
import{registerTask}from"@apache-airflow/ts-sdk";registerTask({dagId: "sales_pipeline",taskId: "extract"},async({ client })=>{constconnection=awaitclient.getConnection("sales_db");constrowCount=Number((awaitclient.getVariable("daily_row_count"))??"0");return{connectionId: connection?.connId??null,
rowCount,};});registerTask({dagId: "sales_pipeline",taskId: "transform"},async({ client })=>{constextracted=awaitclient.getXCom<{rowCount: number}>({key: "return_value",taskId: "extract",});return{transformedRows: extracted?.rowCount??0,};});

The Python stub defines the Dag dependency graph. The TypeScript handler does the work and uses TaskClient for task-time Airflow data access. The follow-up coordinator runtime will launch Node.js, find the registered handler for that Dag/task pair, and run it.

Next steps:

  • Add coordinator-mode execution in a follow-up PR: TypeScript runtime, coordinator TaskClient implementation, supervisor protocol/schema wiring, logging, cancellation handling, and Python TypescriptCoordinator registration.
  • Add coordinator schema-mapping tests that validate camelCase public options serialize to the supervisor schema's snake_case fields.
  • Add Airflow-level integration coverage for coordinator-mode TypeScript tasks, including the daemon-based end-to-end path we used locally.
  • Keep edge-worker execution separate from coordinator mode so the public SDK, coordinator runtime, and future edge runtime can evolve independently.
  • Revisit higher-level TypeScript Dag authoring and bundle ergonomics after coordinator-mode execution is reviewed, instead of expanding this first interface PR.

Testing:

  • pnpm -C ts-sdk format:check
  • pnpm -C ts-sdk lint
  • pnpm -C ts-sdk typecheck
  • pnpm -C ts-sdk test
  • pnpm -C ts-sdk run build
  • pnpm -C ts-sdk pack --dry-run
  • ~/.local/bin/prek run --from-ref upstream/main --stage pre-commit
  • ~/.local/bin/prek run --from-ref upstream/main --stage manual
Was generative AI tooling used to co-author this PR?
  • Yes - Codex (GPT-5)

Generated-by: Codex (GPT-5) following the guidelines


Drafted-by: Codex (GPT-5) (no human review before posting)

@shivaam

Copy link
Copy Markdown
ContributorAuthor

@jason810496@uranusjr Appreciate your feedback

@shivaamshivaam changed the title Add TypeScript Task SDKAdd TypeScript task SDK public interfaceJun 4, 2026
@uranusjr
uranusjr self-requested a review June 5, 2026 05:20
shivaam added 2 commits June 5, 2026 00:20
Require TypeScript task handlers to register with separate dagId and taskId fields instead of a flat task id string. Store handlers in a Dag-scoped registry so task ids only need to be unique within a Dag, matching the Java and Go SDK registration model.
Update README examples and public API type tests to use the explicit registration shape.
@shivaam

Copy link
Copy Markdown
ContributorAuthor

Addressed feedback from @ashb offline just on the user interface.

@eladkaleladkal added this to the Airflow 3.3.0 milestone Jun 7, 2026
@potiukpotiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 8, 2026
@eladkaleladkal modified the milestones: Airflow 3.3.0, Airflow 3.4.0, Typescript SDK 0.0.1Jun 11, 2026
@shivaamshivaam mentioned this pull request Jun 15, 2026
1 task
@jason810496
jason810496 self-requested a review June 15, 2026 04:07

@pierrejeambrunpierrejeambrun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really nice first slice.

Also +1 on the ts-sdk naming — it's consistent with the Java SDK naming itself after the author's language (it's actually Kotlin under the hood), so no change needed if anyone questions it.

A few small consistency notes inline

Comment threadts-sdk/src/task.ts Outdated
Comment threadts-sdk/src/client-types.ts Outdated
Comment threadts-sdk/src/registry.ts Outdated
@uranusjr

Copy link
Copy Markdown
Member

I think in general this is good. We should add a pre-commit hook to lint ts files in here.

@shivaam

Copy link
Copy Markdown
ContributorAuthor

I think in general this is good. We should add a pre-commit hook to lint ts files in here.

@uranusjr I have added a root prek hook for the TS SDK, following the existing Airflow TS/UI hooks. It runs the SDK’s local lint/format/typecheck scripts and auto-fixes lint/format issues where possible.

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for being patient and continue one TS-SDK, and sorry for pushing back on the user interface.

Comment thread.pre-commit-config.yaml
Comment threadts-sdk/src/sdk/client.ts
Comment threadts-sdk/README.md Outdated

@shivaamshivaam left a comment

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed all 3 comments form @jason810496

Comment threadts-sdk/src/sdk/client.ts
Comment threadts-sdk/README.md Outdated
Comment thread.pre-commit-config.yaml

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. LGTM.

@shivaam

shivaam commented Jun 27, 2026

Copy link
Copy Markdown
ContributorAuthor

@uranusjr Thanks for the reviews. Would it be possible to merge this PR so I can build upon it and add the protocol PR.

@uranusjr
uranusjr merged commit f57229b into apache:mainJun 28, 2026
77 checks passed
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jun 30, 2026
@jason810496jason810496 moved this to Done in TS-SDKJul 20, 2026
@jason810496jason810496 modified the milestones: Typescript SDK 0.0.1, TS SDK 1.0 BetaJul 20, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for maintainer reviewSet after triaging when all criteria pass.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants

@shivaam@uranusjr@pierrejeambrun@jason810496@potiuk@eladkal