Uh oh!
There was an error while loading. Please reload this page.
Support TaskFlow Dag definitions natively in the Go-SDK - #70158
Draft
jason810496 wants to merge 8 commits into
Draft
Support TaskFlow Dag definitions natively in the Go-SDK#70158jason810496 wants to merge 8 commits into
jason810496 wants to merge 8 commits into
Conversation
jason810496force-pushed
the
feature/go-sdk/taskflow-syntax
branch
3 times, most recently
from
July 21, 2026 07:15
04c6c1b to
0ea0905Compare
This was referenced Jul 22, 2026
jason810496force-pushed
the
feature/go-sdk/taskflow-syntax
branch
2 times, most recently
from
August 20, 2026 09:08
8ec3ab9 to
ecff8f0CompareNative bundle registration bypasses Python-side validation, so invalid scheduling, identifiers, and task execution flags must be enforced in the Go runtime.
jason810496force-pushed
the
feature/go-sdk/taskflow-syntax
branch
from
August 21, 2026 03:15
ecff8f0 to
0d9ad95CompareDependency-only task declarations should not need an empty TaskSpec value.
jason810496force-pushed
the
feature/go-sdk/taskflow-syntax
branch
from
August 21, 2026 07:23
0d9ad95 to
3d074b6CompareDag.Task(fn, opts...) replaces AddTask/AddTaskWithName and returns a
*TaskRef handle. Passing handles via v1.Inputs(...) wires the dependency
edge and feeds each upstream return value into the matching data
parameter of the task function at run time: the runtime pulls the
upstream return-value XCom and strictly decodes it into the declared
parameter type, so a Go dag reads like plain function composition:
d := reg.AddDag(v1.DagSpec{DagId: "etl", Schedule: "@daily"})
extracted := d.Task(extract)
d.Task(transform, v1.Inputs(extracted))
v1.After(...) declares ordering-only edges. Task ids move into the
specs: TaskSpec.TaskId (generated from schema.json as an identity
field) and DagSpec.DagId replace positional ids and AddTaskWithName.
Mismatched input counts or types, refs from another dag, and
non-decodable parameters all panic at registration, i.e. dag-parse
time.jason810496force-pushed
the
feature/go-sdk/taskflow-syntax
branch
from
August 21, 2026 12:29
3d074b6 to
35f50c4Comparejason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Aug 24, 2026
Review found serde.go's serializeTaskGroup described as shipped present-tense fact; it only exists on the unmerged apache#67155/apache#70158 branches, same class of mistake ADR 7 already had to fix. Also split the ShortCircuit/Branch examples' func declarations out of the registration-statement fences to match ADR 7's presentation, since a package-level func decl mixed with := statements isn't valid Go as written.
jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Aug 24, 2026
…s current shape Review found apache#70158's own registry.go still exposes After(refs...); "Inputs is the only way to wire a dependency" read as a factual claim about that branch rather than what it actually is, a simplification this ADR proposes on top of it. Also cited serialized_objects.py instead of schema.json for where task_type/_task_module actually get written, since schema.json only declares the fields as required.
jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Aug 25, 2026
Add ADRs for the Mixed Lang Dag interface already shipped via apache#70209, the proposed Native Dag interface (apache#67155/apache#70158), and the common task constructs a native Go author will need next (TaskGroup, ShortCircuit, Branch, TriggerDagRun). Recording the rationale here gives reviewers and future contributors a single reference for these tradeoffs instead of reconstructing them from scattered PR discussions.
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The base Go SDK authoring API wires dependencies with string task IDs, which is typo-prone and does not carry upstream values. This PR makes dependencies follow typed data flow while retaining an explicit ordering-only option.
Example
Before this PR, native task dependencies are optional string slices:
With this PR, task references express both data and ordering relationships:
How
AddTaskandAddTaskWithNamewithDag.Task(fn, opts...), which returns aTaskRefand acceptsTaskSpec,Inputs, andAfteroptions.DagSpec.DagIdandTaskSpec.TaskId, with function names remaining the default task IDs.native_dagto showcase typed native authoring.DagImporteris wired up.Was generative AI tooling used to co-author this PR?