Uh oh!
There was an error while loading. Please reload this page.
Record the Go SDK's Mixed Lang and Native Dag task interfaces - #72043
Record the Go SDK's Mixed Lang and Native Dag task interfaces#72043jason810496 wants to merge 1 commit into
Conversation
Add ADRs for the Mixed Lang Dag interface already shipped via apache#70209, the proposed Native Dag interface (apache#67155/apache#70158), and the common task constructs a native Go author will need next (TaskGroup, ShortCircuit, Branch, TriggerDagRun). Recording the rationale here gives reviewers and future contributors a single reference for these tradeoffs instead of reconstructing them from scattered PR discussions.
| ### Registering the Dag | ||
| ```go | ||
| dag := registry.AddMixedLangDag("etl") |
There was a problem hiding this comment.
Can this just be
| dag := registry.AddMixedLangDag("etl") | |
| dag := registry.NewDag("etl") |
Or AddDag
From the point of view of go is not mixed language, nor do i think that is being mixed changes anything?
There was a problem hiding this comment.
The reason is that we need a different interface for the mixed Lang Dag at the user authoring level, to avoid further confusion when we support the native Dag.
We need a way to distinguish whether a Go Dag definition is a mixed Lang Dag or a native Dag; otherwise, when the DagImporter is involved, the DagModel will jump between the Python Dag with Stub Operator definition and the Go native Dag for the same DagId. The alternative I had considered is having an is_mixed_language_dag flag at the Dag level to separate the purpose, but IMO it would be better to have an explicit, different interface, as the MixedLangDag is really a special case.
Another direction is to make Dag a package-level function instead of coupling it as a registry method. For example: d1 := v1.Dag("etl") d2 := v1.MixedLangDag("cross_lang") registry.AddDags(d1, d2)
| cleaned := group.Task(cleanRows) | ||
| validated := group.Task(validateRows, v1.Inputs(cleaned)) | ||
| dag.Task(nativeLoad, v1.Inputs(validated)) |
There was a problem hiding this comment.
I think this line was left over from the previous example.
But i think it would be worth showing how to add up or downstream deps to a task grout
There was a problem hiding this comment.
This is where Then comes into play. a.Then(b, c) is equivalent to a >> [b, c] in Python. I will update this example to avoid the confusion.
| ## Question | ||
| - How should the Go SDK implement deferral, so `TriggerDagRunOperator` can support `deferrable=True` together with `wait_for_completion=True`? Python defers via `TaskDeferred`/`DagStateTrigger`, letting the operator yield control and resume once the triggered Dag run finishes; Go has no equivalent mechanism today, so a task function runs to completion in one call with no way to pause mid-run. |
There was a problem hiding this comment.
Go can return special error values similarly to how python raises exceptions.
That said, I'm not sure if that pattern is good.
There was a problem hiding this comment.
Deferal likely matters a lot less in Go as a each task runs in a goroutine, not a process, so the overhead per task is much much smaller and one go worker process could, if we/the user chooses to, run 100s of tasks inside it
There was a problem hiding this comment.
What I thought in the first place is to leverage the existing Python-based Triggerer components. The TriggerDagRunOperator on the Go side only serves the DSL purpose, and the Go runtime + coordinator will proxy the message to the Python world's supervisor for invoking the Execution API.
Deferal likely matters a lot less in Go as a each task runs in a goroutine, not a process, so the overhead per task is much much smaller and one go worker process could, if we/the user chooses to, run 100s of tasks inside it
IIUC, Do you mean to introduce Go-based Triggerer? Then we will need to make sure the existing trigger table with the serialized tuple (classpath, kwargs) still fits Golang (and other Lang SDK runtimes) and it's a bigger picture to support.
I prefer to keep the TriggerDagRunOperator as Golang DSL only and translate it into the existing Execution API call, like how the TriggerDagRunOperator Python operator works, to keep it as simple as possible and make the feature work for now.
There was a problem hiding this comment.
What I thought in the first place is to leverage the existing Python-based Triggerer components. The TriggerDagRunOperator on the Go side only serves the DSL purpose, and the Go runtime + coordinator will proxy the message to the Python world's supervisor for invoking the Execution API.
As we just discussed offline, the Go SDK will only focus on the DSL purpose for this quarter.
| ## Question | ||
| - How should the Go SDK implement deferral, so `TriggerDagRunOperator` can support `deferrable=True` together with `wait_for_completion=True`? Python defers via `TaskDeferred`/`DagStateTrigger`, letting the operator yield control and resume once the triggered Dag run finishes; Go has no equivalent mechanism today, so a task function runs to completion in one call with no way to pause mid-run. | ||
| - Should the `v1` API move entirely to package-level functions instead of `Dag` methods (`v1.Dag(...)`, `v1.Task(dag, fn, opts...)`, `v1.TriggerDagRunOperator(dag, spec)`), rather than the method style adopted here? |
There was a problem hiding this comment.
Dag and Task aside, i think the standard operators (Branch, TriggerDagRun etc) should live in a different package to the "core language"
There was a problem hiding this comment.
Sure, it makes sense to follow the Python prior art convention.
Write the design for TaskFlow and native Dag for the Go-SDK up front, to settle the design early and surface as much context as possible for reviewers on the actual implementation PRs.
Was generative AI tooling used to co-author this PR?