Allow ExternalTaskSensor to wait for taskgroup

[xinbinhuang]

closes: #14563

This PR enables ExternalTaskSensor to also wait for the external task_group.

The implementation is to retrieve the external DAG from the DagBag and then check if the TaskGroup exists. If so, query and wait for the states of all tasks within that TaskGroup during the poking cycle.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

[xinbinhuang]

This is to keep the poking check behavior return count_allowed == len(dttm_filter)

[xinbinhuang]

self.check_existence = check_existence is False by default, which maybe make sense for external_dag or external_task. But external_task_group has to check and get an existing dag in order to get the list of task_ids.

https://github.com/apache/airflow/blob/fce49402461ee4e7a5f6ffd18cee3121f3496a39/airflow/sensors/external_task.py#L174-L180

I wonder if we can change the default to True or even have check_existence enabled required? This can give more useful errors if the external task/dag does not exist as well as having a consistent behavior as external_task_group. Also, what would be use case to have a Sensor waiting for an object that doesn't exist until it times out?

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Backport packages$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[xinbinhuang]

Here I am making the assumption that as long as there is at least one external task failure, then we will want to fail the sensor. Though this changes the original behavior, I think this will be a better.

[turbaszek]

Should we add this comment in code?

[xinbinhuang]

I don't think it's necessary more than an entry in the UPDATING.md. I think the only situation where you will have multiple counts is when the execution_date_fn returns more than one execution date to wait for. However, the original behavior will get you into a weird state when only part of the TIs fail, i.e. one fail and one succeeds, resulting in time out. IMHO, I think this's more like a bug than intended behavior. WDYT?

[xinbinhuang]

The main piece that you retrieve a list of tasks for a TaskGroup. I believe that read_dags_from_db=True is safe to use here because serialized dag is enabled by default in 2.0.

[yuqian90]

The existing task execution code is creating DagBag on its own instead of reading serialized dags from db. For example this line is creating a DagBag. I think we should do the same here. It's important for tasks to get the latest view of the dag during execution.

https://github.com/apache/airflow/blob/f1edc220d3f9cb050016d23246a682276bd09eee/airflow/sensors/external_task.py#L213

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[github-actions]

The Workflow run is cancelling this PR. It has some failed jobs matching ^Pylint$,^Static checks,^Build docs$,^Spell check docs$,^Provider packages,^Checks: Helm tests$,^Test OpenAPI*.

[xinbinhuang]

Test fails on K8S image build job, and I think it's not relevant to this PR

[potiuk]

Just fixed the K8S problem in #15182 - can you please rebase.

[kaxil]

Can you fix the conflicts please @xinbinhuang

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[eladkal]

@xinbinhuang will you have time to complete it?

[xinbinhuang]

@xinbinhuang will you have time to complete it?

Thanks for the nudge! Will try to wrap it up before the holidays hit.

[eladkal]

Thanks for the nudge! Will try to wrap it up before the holidays hit.

Great :) re-opening so it won't be missed

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[russellpierce]

Darn, it's a shame this didn't get through - it is exactly what I was looking for!

[potiuk]

Feel free to open a PR and contribute it on your own. You willl just have to make sure to follow it up and implement it to the quality that we expect @russellpierce - Airflow is created by > 2000 contributors like you, so if you need something, implementing it yourself is the fastest way to get things done @russellpierce.