Skip to content

Add deferrable param in EmrContainerSensor - #30945

Merged
pankajastro merged 2 commits into
apache:mainfrom
astronomer:async_emr_container_sensor
Jun 19, 2023
Merged

Add deferrable param in EmrContainerSensor#30945
pankajastro merged 2 commits into
apache:mainfrom
astronomer:async_emr_container_sensor

Conversation

@pankajastro

Copy link
Copy Markdown
Member

Add the deferrable param in EmrContainerSensor.
This will allow running EmrContainerSensor in an async way
that means we only submit a job from the worker to run a job
then defer to the trigger for polling and wait for a job the job status
and the worker slot won't be occupied for the whole period of
task execution.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Apr 28, 2023
@pankajastro
pankajastro marked this pull request as ready for review April 28, 2023 20:26
Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threadairflow/providers/amazon/aws/waiters/emr-containers.json

@syedahsnsyedahsn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Just a few minor comments.

Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py
Comment threadairflow/providers/amazon/aws/waiters/emr-containers.json
@pankajastro
pankajastro marked this pull request as draft May 26, 2023 08:49
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch 2 times, most recently from 40c947f to 619ab51CompareMay 26, 2023 09:37
@pankajastro
pankajastro marked this pull request as ready for review May 26, 2023 12:32
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch 2 times, most recently from e25f107 to fd0db14CompareJune 5, 2023 10:00
Comment threadairflow/providers/amazon/provider.yaml Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though I have seen it elsewhere, to me, putting "max_attempts" as a trigger parameter doesn't make sense.

The "right" way to set a time limit on a trigger is the deferral timeout. IMO, if an existing operator has max_attempts param, then we should just calculate the deferral timeout based on that number and use that.

This avoids the added complexity in the trigger, the extra signature param, and it avoids the odd fact that, if max attempts is used, then if a triggerer dies and the trigger is picked up again on another machine, it will start from zero and then get killed by the timeout anyway.

What do you think about this @pankajastro ?

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point. thank you for raising it.

honestly, I didn't think about the possibility that a trigger can restart while writing this. I'll fix this.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a point that was brought up in a previous PR (by @dstandish :D), and the solution we went with was to essentially use both. Set a timeout on the operator level that is computed from the given parameters (as well as a 60 second buffer), but also use the number of attempts as a metric. I think it is beneficial to have the number of attempts used as a metric in the Triggers, but it is definitely a good idea to have the timeout set on the operator level in case of a Trigger restart, as mentioned above.

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adeed the timeout while deferring to handle

if a triggerer dies and the trigger is picked up again on another machine

removed max_attempt to reduce the code maintenance

@Lee-WLee-W mentioned this pull request Jun 6, 2023
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch 5 times, most recently from 1244bd9 to b1edbf7CompareJune 8, 2023 10:17
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch from bda9e41 to 5ed07dfCompareJune 13, 2023 15:05
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch 5 times, most recently from a56de48 to 1bcbf8eCompareJune 19, 2023 10:15
Add the deferrable param in EmrContainerSensor.
This will allow running EmrContainerSensor in an async way
that means we only submit a job from the worker to run a job
then defer to the trigger for polling and wait for a job the job status
and the worker slot won't be occupied for the whole period of
task execution.
@pankajastro
pankajastroforce-pushed the async_emr_container_sensor branch from 1bcbf8e to df24dc2CompareJune 19, 2023 11:19
@pankajastro
pankajastro merged commit f0b91ac into apache:mainJun 19, 2023
@pankajastro
pankajastro deleted the async_emr_container_sensor branch June 19, 2023 19:38
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@pankajastro@ephraimbuddy@pankajkoti@dstandish@vincbeck@syedahsn