Derive a mixed-language marker for Dags with task.stub

[jason810496]

• related: AIP-108, AIP-85
• depends on:Support TaskFlow call syntax on stub tasks for the Lang SDK #69757 -- the is_stub flag was introduced there

Why

A Lang-SDK artifact can play two roles that a Dag processor must treat oppositely, and nothing in the serialized Dag tells them apart:

Role	Where the Dag is defined	What the artifact contributes
Mixed-language Dag	A Python file whose tasks are @task.stub	Only the task implementations
Native Lang-SDK Dag	The Go/Java source itself	The whole Dag

For a native Dag the processor parses the artifact and the result is persisted. For a mixed-language Dag it must persist nothing: the Python Dag carrying the @task.stub tasks already owns that dag_id, so persisting the Lang-SDK side too would give conflicts.

How

The new is_mixed_language_dag flag at Dag level solves the problem.

• Only the Lang SDK Dag can explicitly set the is_mixed_language_dag flag at Dag level argument.
• For the Python Dag side, the is_mixed_language_dag field is a compute attribute that if there's any StubOperator in the Dag, the property will be true.
• The Task SDK DAG gains nothing, a Python Dag can never set the is_mixed_language_dag flag.

The further mixed language Dag processing flow will be like:

• JavaDagImporter don't emit those Dag with is_mixed_language_dag=True by default.
• PythonDagImporter -> for all Dags which is_mixed_language_dag=True (the flag will be on if there's StubOperator in the Dag) -> validate the corresponding Java Dag structure (via TI.queue -> Coordinator -> JavaDagImporter(discover_mixed_language_dags=True) -> Java Dag structure) -> raise import error if there's validation error
• and don't persist the Java side Dag structure.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code Opus 5 following the guidelines

[uranusjr]

I guess this is the best given the restrictions since those names need to be public, and we don’t want to rename the Task protocol.

[uranusjr]

When do we plan to actually use this? I wonder if a property similar to the SDK DAG class’s property is enough.

[uranusjr]

• Only the Lang SDK Dag can explicitly set the is_mixed_language_dag flag at Dag level argument.

This does not seem to be a part of the PR? Why are you putting this here? Even if it’s true, why is it the case?

[uranusjr]

The PR description diverges A LOT from the implementation. In the future, I would suggest not putting this verbose information in the description. Use an issue instead, and when you update the PR, add comments to the issue to record the design changes timeline.

If you really do want to put this much stuff in the PR description, REMEMBER TO UPDATE IT. Tell Claude to do it, I don’t care, but you need to do it.

[jason810496]

When do we plan to actually use this? I wonder if a property similar to the SDK DAG class’s property is enough.

When introducing the native Dag DSL. We have to introduce at "serialization Dag level" so the native Dag is able to set the flag. (native Dag DSL -> Serialized Dag JSON -> DagFileParsingResult -> LazyDeserializedDAG)

• Only the Lang SDK Dag can explicitly set the is_mixed_language_dag flag at Dag level argument.

This does not seem to be a part of the PR? Why are you putting this here? Even if it’s true, why is it the case?

IMO, we need to define the spec first (which is the purpose of this PR) before wiring the native Dag part up.
If I don't explain how does the further native Dag <-> Airflow core Dag serialization works. The further reviewers don't know why do we need to introduce a new field at all.

[uranusjr]

I think there are two issues. First of all, why does a language sdk dag need to define this at dag-level? From Airflow Core’s prespective, such a dag is essentially equivalent to a Python-defined dag that only contains stub tasks. Why can’t the dag processor just emit task-level markers?

Second, even if we accept a dag-level flag for this use case, is_mixed_language_dag is completely backwards. A Java dag that defines only Java tasks is entirely the opposite of “mixed” since there’s only Java.

[jason810496]

I think there are two issues. First of all, why does a language sdk dag need to define this at dag-level? From Airflow Core’s prespective, such a dag is essentially equivalent to a Python-defined dag that only contains stub tasks. Why can’t the dag processor just emit task-level markers?

Second, even if we accept a dag-level flag for this use case, is_mixed_language_dag is completely backwards. A Java dag that defines only Java tasks is entirely the opposite of “mixed” since there’s only Java.

The mixed Lang artifact is parseable once both #70805 (support optional DagBundle based deployment) and the DagImporter interface (JarImporter, able to get Dags from Jars) get merged.

Without the flag, consider the case where we place the Python Dag with stub operator in one DagBundle and the Jar artifact in another DagBundle, the DagModel will jump between mixed Lang Dag and native Dag. Since the Jar artifact for mixed Lang itself is also a valid native Dag Jar.

From my perspective, it's necessary to introduce the flag at the Dag-level to make the purpose explicit, so that the DagImporters are able to distinguish between the mixed Lang Jar and the native Dag Jar.

[Andrushika]

Just want to clarify a higher-level design question... Will the Lang SDK support stub syntax in native Dags in the future? (e.g. a native Java Dag that holds a Python or Go task)
If not, please just ignore me.

If it is planned, I think the current flag may not cover that case. Right now the same boolean means two different things:

1. On a Python Dag it is derived, and it means “this Dag has stub tasks”.
2. On a Lang-SDK Dag the producer sets it, and it means “this Dag is implementation-only, do not persist it” (if I read the planned flow right).

A native Dag with stub tasks would hit both meanings at once. It has stub tasks, but it is also the real definition that should be persisted. Setting the flag to true would get it dropped, so it has no way to mark itself.

If that case is planned, I would suggest a separate field such as definition_role (primary / implementation_only). Then this flag keeps one meaning only (“has stub tasks”), and the persist decision reads the role instead.

Just a small suggestion and I could be completely wrong. Thanks!

[jason810496]

Will the Lang SDK support stub syntax in native Dags in the future? (e.g. a native Java Dag that holds a Python or Go task)

Your understanding is correct! With the native Dag support in foreign Lang SDK, we can define the "primary" Dag with Stub Operator in foreign Lang SDK side and let the Python Dag implement the task body.

If not, please just ignore me.

From my perspective, I don't think we need to support this feature at all. It makes more sense to only support Python as the primary language for the mixed Lang Dag feature to keep the whole idea simple.

Airflow only supports Python DSL in the last decade! Additionally, the Stub Operator itself might be confusing for some of the user, so not to mention defining the Stub Operator in foreign Lang SDK (it will be mind blowing for the users I guess)

[Andrushika]

Thanks for clarifying -- I have no further questions. These changes LGTM!