Uh oh!
There was an error while loading. Please reload this page.
Support TaskFlow call syntax on stub tasks for the Lang SDK - #69757
Conversation
7b528b3 to
bec0e8dComparefca5aa1 to
0b464a1Compare6c609d9 to
c21137fCompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
82a0aa5 to
7e2d0f2Compare
jason810496
left a comment
There was a problem hiding this comment.
I will wait until next review then address my own comments to avoid CI-rerun.
Uh oh!
There was an error while loading. Please reload this page.
ashb
left a comment
There was a problem hiding this comment.
Overall I like the direction. Almost all of my comments I can be challenged on, don't just make the changes if you think the current way is better/more correct
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
jason810496
left a comment
There was a problem hiding this comment.
Thanks you Ash for the review. I will address the comments shortly.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
jason810496
left a comment
There was a problem hiding this comment.
Thanks all for the review, I just resolved all the comments and the PR is ready for the next review.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Stub tasks silently ignored TaskFlow call arguments, so a Dag author could not hand literals or upstream XCom results to a lang-SDK runtime. The decorator now binds the call to the stub's signature at parse time and captures an ordered arg spec (literal values and direct upstream XCom references, with pydantic-derived JSON value schemas) that serializes with the Dag, while rejecting what cannot cross the language boundary: custom XCom keys, aggregated mapped outputs, non-JSON literals, and stubs with arguments inside mapped task groups. Mapped (.expand()) stubs capture no spec and keep the legacy behavior until a follow-up delivers per-map-index bindings.
TIRunContext gains an arg_bindings field so a lang-SDK runtime receives the stub task's TaskFlow arg spec at startup. ti_run derives it from the serialized Dag only for stub operators, so regular tasks never pay for the lookup, and only for clients on the new API version -- gated on the Cadwyn VersionChangeWithSideEffects.is_applied check rather than a date comparison -- so stub Dags that predate arg bindings keep running against older clients, for which the version migration strips the field.
StartupDetails in the supervisor wire schema carries the new arg_bindings so foreign runtimes receive the spec at task startup, with a version migration that strips it for runtimes pinned to the previous schema. The Go and TS SDKs regenerate against the new schema version; the Go arg-binding runtime itself lands in a stacked follow-up PR.
An XComArg buried in a list or dict literal fell through to the JSON check, whose "pass it in its JSON form instead" advice is impossible to follow for a task output. Detect nested references up front and point the author at the working alternative: pass the upstream output as its own argument.
Review feedback flagged two issues with keeping this in the provider: the execution API gated on @task.stub's operator class name, which any future lang-SDK operator would have had to duplicate, and generating JSON-schema fragments from Python type hints is tightly coupled to the execution API and didn't belong in the standard provider's decorator. A generic inherits_from_stub_operator flag (mirroring EmptyOperator's inherits_from_empty_operator) now drives arg-binding materialization from OperatorSerialization._serialize_node, so the provider's _StubOperator shrinks to just its own structural checks and any future foreign-runtime operator gets the same treatment for free.
inherits_from_stub_operator mirrored EmptyOperator's private-field-plus- property pattern, but that indirection exists there to let a live EmptyOperator instance override an inherited property; @task.stub has no such need for a settable default. A plain public is_stub attribute is simpler and matches the sibling lang-SDK marker landing in parallel. Propagating it onto MappedOperator too (not just the non-mapped stub) keeps the flag meaningful for any future consumer that needs to know a task is stub-backed regardless of whether it's mapped; arg-binding materialization itself still skips mapped operators, which have no op_args/op_kwargs to bind against.
serialized_objects.py was carrying the entire TaskFlow arg-binding builder inline, pulling in pydantic schema generation and several SDK-only imports for logic that only fires for @task.stub tasks. Splitting it into stub_arg_bindings.py keeps the core serialization module's import surface focused, and a dedicated arg_bindings property replaces direct access to the private _arg_bindings attribute from outside SerializedBaseOperator. The builder is imported lazily too, so Python-only deployments never pay for pydantic's JSON-schema machinery just to serialize a Dag. Review also caught that schema.json described the binding in the {__type, __var} encoded form the serializer never emits, and that the test asserting it validated nothing -- schema.json's tasks array never reaches the operator subschema, so the definition is now asserted directly instead.
f0c6664 to
3030981Compare
uranusjr
left a comment
There was a problem hiding this comment.
Do we need an ADR for this decisions made in this PR? Otherwise I think this is good.
jason810496
commented
Aug 12, 2026
Yes, I will add an ADR as follow-up to avoid the heavy full test CI run again. |
Uh oh!
There was an error while loading. Please reload this page.
The design settled in apache#69757 -- where the arg-binding spec is materialized, why types travel as JSON Schema rather than an Airflow-specific enum, why the spec is delivered at ti_run, and which TaskFlow forms deliberately do not cross the boundary -- was argued out in the review thread and recorded nowhere a future reader would find it. Without it the same questions get re-litigated on the follow-up PRs that build on the contract.
jason810496
commented
Aug 12, 2026
Hi @ashb, please feel free to get any suggestion for follow-up when you have a moment, thanks. |
* Add ADR for TaskFlow across the language boundary The design settled in #69757 -- where the arg-binding spec is materialized, why types travel as JSON Schema rather than an Airflow-specific enum, why the spec is delivered at ti_run, and which TaskFlow forms deliberately do not cross the boundary -- was argued out in the review thread and recorded nowhere a future reader would find it. Without it the same questions get re-litigated on the follow-up PRs that build on the contract. * Correct ADR-0007 claims about the arg-binding extension contract Review of the ADR caught that it overstated how far the is_stub flag generalizes: materialization also requires a DecoratedOperator, because the builder works from a python_callable signature. It likewise implied the signature and literal checks run for every stub, when they engage only once a call passes an argument, and that _arg_bindings decodes as strictly as is_stub. An ADR that misdescribes its own extension point is worse than no ADR, since the next foreign-runtime operator would be designed against it. * Drop a false union-annotation gap from ADR-0007 The appendix claimed union annotations were unhandled, carried over from a mid-review comment that predated the move from a data_type enum to JSON Schema. Unions are handled, and thoroughly: test_stub_arg_bindings.py covers dict | bool, PEP 604 and legacy optionals, three-way unions, temporals inside unions, and member dedup. They now serve as evidence for the JSON Schema decision they were wrongly listed as a gap in. Also drop a Consequences bullet that restated the motivation instead of a consequence.
A Go task could only reach an upstream task's output by hand-writing a GetXCom call against a hard-coded task id, duplicating wiring the Dag file already owns and breaking silently whenever that upstream was renamed. apache#69757 ships the Python half: a `@task.stub` TaskFlow call is captured at Dag serialization as an ordered arg-binding spec and returned by ti_run. Consuming it here lets a Go task function take the Dag's literals and upstream XComs as ordinary typed parameters. A function declares either flat positional parameters or a single struct whose fields bind by name -- kwarg-style, so an unmatched field keeps its zero value while an argument no field claims fails the task. Signature problems are caught once at registration; per-execution arity, type and spec errors fail the task before its body runs, replacing a silent reflect.Zero fill.
A Go task could only reach an upstream task's output by hand-writing a GetXCom call against a hard-coded task id, duplicating wiring the Dag file already owns and breaking silently whenever that upstream was renamed. apache#69757 ships the Python half: a `@task.stub` TaskFlow call is captured at Dag serialization as an ordered arg-binding spec and returned by ti_run. Consuming it here lets a Go task function take the Dag's literals and upstream XComs as ordinary typed parameters. A function declares either flat positional parameters or a single struct whose fields bind by name -- kwarg-style, so an unmatched field keeps its zero value while an argument no field claims fails the task. Signature problems are caught once at registration; per-execution arity, type and spec errors fail the task before its body runs, replacing a silent reflect.Zero fill.
…9757) * Support TaskFlow call syntax on @task.stub tasks Stub tasks silently ignored TaskFlow call arguments, so a Dag author could not hand literals or upstream XCom results to a lang-SDK runtime. The decorator now binds the call to the stub's signature at parse time and captures an ordered arg spec (literal values and direct upstream XCom references, with pydantic-derived JSON value schemas) that serializes with the Dag, while rejecting what cannot cross the language boundary: custom XCom keys, aggregated mapped outputs, non-JSON literals, and stubs with arguments inside mapped task groups. Mapped (.expand()) stubs capture no spec and keep the legacy behavior until a follow-up delivers per-map-index bindings. * Ship stub arg_bindings in a new execution API version TIRunContext gains an arg_bindings field so a lang-SDK runtime receives the stub task's TaskFlow arg spec at startup. ti_run derives it from the serialized Dag only for stub operators, so regular tasks never pay for the lookup, and only for clients on the new API version -- gated on the Cadwyn VersionChangeWithSideEffects.is_applied check rather than a date comparison -- so stub Dags that predate arg bindings keep running against older clients, for which the version migration strips the field. * Deliver stub arg bindings to SDK runtimes via the supervisor schema StartupDetails in the supervisor wire schema carries the new arg_bindings so foreign runtimes receive the spec at task startup, with a version migration that strips it for runtimes pinned to the previous schema. The Go and TS SDKs regenerate against the new schema version; the Go arg-binding runtime itself lands in a stacked follow-up PR. * Reject upstream outputs nested inside stub literal collections An XComArg buried in a list or dict literal fell through to the JSON check, whose "pass it in its JSON form instead" advice is impossible to follow for a task output. Detect nested references up front and point the author at the working alternative: pass the upstream output as its own argument. * Materialize stub TaskFlow arg bindings in core Dag serialization Review feedback flagged two issues with keeping this in the provider: the execution API gated on @task.stub's operator class name, which any future lang-SDK operator would have had to duplicate, and generating JSON-schema fragments from Python type hints is tightly coupled to the execution API and didn't belong in the standard provider's decorator. A generic inherits_from_stub_operator flag (mirroring EmptyOperator's inherits_from_empty_operator) now drives arg-binding materialization from OperatorSerialization._serialize_node, so the provider's _StubOperator shrinks to just its own structural checks and any future foreign-runtime operator gets the same treatment for free. * Simplify the stub-task marker to a plain is_stub attribute inherits_from_stub_operator mirrored EmptyOperator's private-field-plus- property pattern, but that indirection exists there to let a live EmptyOperator instance override an inherited property; @task.stub has no such need for a settable default. A plain public is_stub attribute is simpler and matches the sibling lang-SDK marker landing in parallel. Propagating it onto MappedOperator too (not just the non-mapped stub) keeps the flag meaningful for any future consumer that needs to know a task is stub-backed regardless of whether it's mapped; arg-binding materialization itself still skips mapped operators, which have no op_args/op_kwargs to bind against. * Extract stub arg-binding materialization into its own module serialized_objects.py was carrying the entire TaskFlow arg-binding builder inline, pulling in pydantic schema generation and several SDK-only imports for logic that only fires for @task.stub tasks. Splitting it into stub_arg_bindings.py keeps the core serialization module's import surface focused, and a dedicated arg_bindings property replaces direct access to the private _arg_bindings attribute from outside SerializedBaseOperator. The builder is imported lazily too, so Python-only deployments never pay for pydantic's JSON-schema machinery just to serialize a Dag. Review also caught that schema.json described the binding in the {__type, __var} encoded form the serializer never emits, and that the test asserting it validated nothing -- schema.json's tasks array never reaches the operator subschema, so the definition is now asserted directly instead.
* Add ADR for TaskFlow across the language boundary The design settled in apache#69757 -- where the arg-binding spec is materialized, why types travel as JSON Schema rather than an Airflow-specific enum, why the spec is delivered at ti_run, and which TaskFlow forms deliberately do not cross the boundary -- was argued out in the review thread and recorded nowhere a future reader would find it. Without it the same questions get re-litigated on the follow-up PRs that build on the contract. * Correct ADR-0007 claims about the arg-binding extension contract Review of the ADR caught that it overstated how far the is_stub flag generalizes: materialization also requires a DecoratedOperator, because the builder works from a python_callable signature. It likewise implied the signature and literal checks run for every stub, when they engage only once a call passes an argument, and that _arg_bindings decodes as strictly as is_stub. An ADR that misdescribes its own extension point is worse than no ADR, since the next foreign-runtime operator would be designed against it. * Drop a false union-annotation gap from ADR-0007 The appendix claimed union annotations were unhandled, carried over from a mid-review comment that predated the move from a data_type enum to JSON Schema. Unions are handled, and thoroughly: test_stub_arg_bindings.py covers dict | bool, PEP 604 and legacy optionals, three-way unions, temporals inside unions, and member dedup. They now serve as evidence for the JSON Schema decision they were wrongly listed as a gap in. Also drop a Consequences bullet that restated the motivation instead of a consequence.
next:
arg_bindings)Scope
This PR ships the Python side only of the contract: parse-time capture of the TaskFlow call into a serialized arg-binding spec, the wire model, and its delivery to SDK runtimes through the Execution API and supervisor schema.
Nothing in this PR binds arguments inside a task runtime — the Go snippet below illustrates the consumer and lives in the stacked follow-up.
Why
@task.stubtasks can only be declared argless today, so a Go task that needs an upstream's output has to hand-writeGetXComcalls (with the upstreamtask_idhard-coded in Go, duplicating the wiring the Dag file already expresses). This PR ships the Python side of making the natural TaskFlow call work across the language boundary:Supported TaskFlow syntax
Every form below parses, serializes, and round-trips through the execution API (capture matrix in
test_stub.py; the Dag shown is #70209'staskflow_binding_dagexample):value_schemais the JSON-schema fragment pydantic derives from the parameter annotation (plus OpenAPI'sint64/doublenumeric formats); annotations pydantic cannot schema, and untyped/Anyparameters, omit it entirely and bind decode-only.@taskor another@task.stub— which also wires the dependency edge (transform("uk", extract())impliesextract >> transform).from_default: true, letting keyword-style consumers leave them unclaimed..expand()) capture no spec and keep today's ignored-args behavior; per-map-index binding lands in Support dynamic task mapping on stub tasks #70570.Rejected loudly when the Dag is serialized including:
map/zip/concatXComArgs, a mapped upstream's aggregated output/**kwargs` or context-key parameter names, and non-JSON-serializable literals (including NaN/Infinity).How
Implement the
is_stuband_arg_bindingin core serialization side instead of provider side so the Execution API layer can leverage "flag-based" pattern (if task.is_stub) instead of relying on the "operator_name". So no need to bump thecommon.compatprovider anymore (we need to bump it to provide some constants in the last implementation).is_stub, binding the TaskFlow call to the stub's signature and emitting an ordered positional spec (_arg_bindings)kind-discriminated union —XComArgBindingorLiteralArgBinding— carrying the parameter'snameand its annotation-derivedvalue_schema(a JSON schema as typing system for describing the "type" of this argument).ti_runreturns the spec as a new optionalTIRunContext.arg_bindings, resolved through the sharedDBDagBagonly foris_stubtasks and stripped for older clients by a new execution-API version2026-10-30.Was generative AI tooling used to co-author this PR?