Skip to content

Add deferrable mode in EMR operator and sensor - #32029

Merged
pankajastro merged 11 commits into
apache:mainfrom
astronomer:add_emr_async_op
Jun 27, 2023
Merged

Add deferrable mode in EMR operator and sensor#32029
pankajastro merged 11 commits into
apache:mainfrom
astronomer:add_emr_async_op

Conversation

@pankajastro

@pankajastropankajastro commented Jun 20, 2023

Copy link
Copy Markdown
Member

Add the deferrable param in
EmrContainerOperator/EmrStepSensor/EmrJobFlowSensor.
This will allow running Operator/Sensors in an async way
which means we only submit a job from the worker to run a job
then defer to the trigger for polling and wait for a job the job status
and the worker slot won't be occupied for the whole period of task execution.

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Jun 20, 2023
@pankajastro

pankajastro commented Jun 20, 2023

Copy link
Copy Markdown
MemberAuthor

cc: @syedahsn

@pankajastro
pankajastro marked this pull request as ready for review June 20, 2023 12:27
@pankajkoti
pankajkoti requested a review from Lee-WJune 20, 2023 12:35
pankajkoti
pankajkoti reviewed Jun 20, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
Lee-W
Lee-W reviewed Jun 21, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
Comment threadtests/providers/amazon/aws/operators/test_emr_containers.py Outdated
Comment threadtests/providers/amazon/aws/sensors/test_emr_containers.py Outdated
Comment threadtests/providers/amazon/aws/sensors/test_emr_job_flow.py Outdated
phanikumv
phanikumv reviewed Jun 21, 2023
View reviewed changes

@phanikumvphanikumv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add docs about availability of deferrable mode for these operators?

Lee-W
Lee-W reviewed Jun 23, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
@pankajastro
pankajastro requested a review from Lee-WJune 23, 2023 10:16
phanikumv
phanikumv reviewed Jun 26, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadairflow/providers/amazon/aws/sensors/emr.py Outdated
Comment threaddocs/apache-airflow-providers-amazon/operators/emr/emr.rst Outdated
Comment threaddocs/apache-airflow-providers-amazon/operators/emr/emr.rst Outdated
phanikumv
phanikumv reviewed Jun 26, 2023
View reviewed changes
Comment threadtests/providers/amazon/aws/operators/test_emr_containers.py Outdated
Comment threadtests/providers/amazon/aws/sensors/test_emr_job_flow.py Outdated
Comment threadtests/providers/amazon/aws/sensors/test_emr_step.py Outdated
Comment threadtests/providers/amazon/aws/triggers/test_emr.py Outdated
Comment threadtests/providers/amazon/aws/triggers/test_emr.py Outdated
Comment threadtests/providers/amazon/aws/triggers/test_emr.py Outdated
Comment threadtests/providers/amazon/aws/triggers/test_emr.py Outdated
@pankajastro
pankajastro requested review from ephraimbuddy and phanikumv and removed request for syedahsnJune 26, 2023 08:10
phanikumv
phanikumv approved these changes Jun 27, 2023
View reviewed changes

@phanikumvphanikumv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase it from main and we should be good to go!

syedahsn
syedahsn reviewed Jun 27, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/triggers/emr.py Outdated
syedahsn
syedahsn reviewed Jun 27, 2023
View reviewed changes
Comment threaddocs/apache-airflow-providers-amazon/operators/emr/emr.rst Outdated
syedahsn
syedahsn reviewed Jun 27, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/waiters/emr.json Outdated
pankajastroand others added 10 commits June 27, 2023 14:47
@pankajastro
Add deferrable mode in EMR operator and sensor
561b4cd
Add the deferrable param in EmrContainerOperator/EmrStepSensor/EmrJobFlowSensor.
This will allow running Operator/Sensors in an async way
that means we only submit a job from the worker to run a job
then defer to the trigger for polling and wait for a job the job status
and the worker slot won't be occupied for the whole period of task execution.
@pankajastro
Apply review suggestion
422a5c8
@pankajastro
Fix static check
e37fd13
@pankajastro
Add docs
63f5acb
@pankajastro
Apply review suggestions
2d31c78
@pankajastro
Apply review suggestion
d49ce09
@pankajastro
Apply review suggestion
ea65205
@pankajastro
Apply review suggestions
02951bd
@pankajastro
Apply review suggestion
e577283
@pankajastro
Fix args name
58e90bb
ephraimbuddy
ephraimbuddy approved these changes Jun 27, 2023
View reviewed changes
Comment threadairflow/providers/amazon/aws/operators/emr.py Outdated
@pankajastro@ephraimbuddy
Apply review suggestion
9153226
Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com>
@pankajastro
pankajastro merged commit 06b5a1e into apache:mainJun 27, 2023
@pankajastro
pankajastro deleted the add_emr_async_op branch June 27, 2023 10:27
ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 27, 2023
@pankajastro@ferruzzi
Add deferrable mode in EMR operator and sensor (apache#32029)
d5a6347
* Add deferrable mode in EMR operator and sensor
Add the deferrable param in EmrContainerOperator/EmrStepSensor/EmrJobFlowSensor.
This will allow running Operator/Sensors in an async way
that means we only submit a job from the worker to run a job
then defer to the trigger for polling and wait for a job the job status
and the worker slot won't be occupied for the whole period of task execution.
This was referenced Jul 6, 2023
Closed
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ephraimbuddyephraimbuddyephraimbuddy approved these changes
@Lee-WLee-WLee-W approved these changes
@phanikumvphanikumvphanikumv approved these changes
@eladkaleladkalAwaiting requested review from eladkal
@o-nikolaso-nikolasAwaiting requested review from o-nikolas
@pankajkotipankajkotiAwaiting requested review from pankajkoti
+1 more reviewer
@syedahsnsyedahsnsyedahsn left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@pankajastro@ephraimbuddy@Lee-W@pankajkoti@phanikumv@syedahsn