Skip to content

Allowing task_group Sensor in ExternalTaskSensor - #24902

Merged
eladkal merged 14 commits into
apache:mainfrom
pateash:airflow-14563
Aug 22, 2022
Merged

Allowing task_group Sensor in ExternalTaskSensor#24902
eladkal merged 14 commits into
apache:mainfrom
pateash:airflow-14563

Conversation

@pateash

Copy link
Copy Markdown
Contributor

closes: #14563
related: #14640

Add meaningful description above
Hi All,

This PR proposes to allow users to provide external_task_group_id in ExternalTaskSensor to make tasks on a DAG wait for another task_group on a different DAG for a specific execution_date.
we can also provide options to set if the Task on a remote DAG succeeded or failed via allowed_states and failed_states parameters.

@pateashpateash changed the title allowing task_group Sensor in ExternalTaskSensorAllowing task_group Sensor in ExternalTaskSensorJul 7, 2022
@pateash

pateash commented Jul 7, 2022

Copy link
Copy Markdown
ContributorAuthor

cases

  1. group1_sensor -> wait for task_group from external dag which exists and succeeded ( as per allowed_states )
  2. group2_sensor- > wait for task_group from external dag which doesn't exists ( wait till timeout )
  3. group2_sensor_check_enabled -> wait for task_group from external dag which doesn't exists and with check_enabled=True ( fail with error )

image

@pateash
pateashforce-pushed the airflow-14563 branch 2 times, most recently from 1425c95 to eb29793CompareJuly 9, 2022 22:20
Comment threadairflow/models/dag.py Outdated
Comment threadairflow/models/dag.py Outdated
Comment threadairflow/sensors/external_task.py Outdated
@pateash

Copy link
Copy Markdown
ContributorAuthor

Hi @eladkal, could you please take a look into this PR.

@uranusjr

Copy link
Copy Markdown
Member

Since we are already allowing multiple task IDs, I wonder if we should just go directly with external_group_ids instead of just one single group. Someone’s going to ask for that sooner or later.

@pateash

Copy link
Copy Markdown
ContributorAuthor

@uranusjr, we already allow multiple taskIds which I personally don't think a good idea, because it is not very transparent.

Although For TaskGroup, I think if someone want to depend on multiple of them, they must create different task sensors.

Comment threadtests/sensors/test_external_task_sensor.py Outdated
Comment threadtests/sensors/test_external_task_sensor.py Outdated
@eladkaleladkal added this to the Airflow 2.4.0 milestone Aug 3, 2022
Comment threaddocs/apache-airflow/howto/operator/external_task_sensor.rst
Comment threaddocs/apache-airflow/howto/operator/external_task_sensor.rst

@eladkaleladkal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eladkal
eladkal merged commit bc04c5f into apache:mainAug 22, 2022
@pateash
pateash deleted the airflow-14563 branch August 23, 2022 07:42
@ephraimbuddyephraimbuddy added the type:new-feature Changelog: New Features label Sep 13, 2022
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:documentationtype:new-featureChangelog: New Features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TaskGroup Sensor

4 participants

@pateash@uranusjr@eladkal@ephraimbuddy