Skip to content

Add wait_policy option to EmrCreateJobFlowOperator. - #44055

Merged
vincbeck merged 1 commit into
apache:mainfrom
adrianlzr:emr-job-flow-operator-add-wait-steps-completion
Nov 25, 2024
Merged

Add wait_policy option to EmrCreateJobFlowOperator.#44055
vincbeck merged 1 commit into
apache:mainfrom
adrianlzr:emr-job-flow-operator-add-wait-steps-completion

Conversation

@adrianlzr

@adrianlzradrianlzr commented Nov 15, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Currently, when using wait_for_complition=True it will trigger the node to only wait until the cluster is successfully launched.
I added the option wait_policy so that one can chose what waiter will be used.

Possible values:

  • None: No wait (default)
  • WaitPolicy.WAIT_FOR_COMPLETION: Previous behaviour when wait_for_completion was True
  • WaitPolicy.WAIT_FOR_STEPS_COMPLETION: New behaviour - wait for the cluster to terminate.

This is useful because it will eliminate the need to have an extra node in your DAG.

@boring-cyborgboring-cyborgBot added area:providers provider:amazon AWS/Amazon - related issues labels Nov 15, 2024
@boring-cyborg

boring-cyborgBot commented Nov 15, 2024

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

vincbeck
vincbeck reviewed Nov 15, 2024
View reviewed changes

@vincbeckvincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you cover this new use case in a unit test?

eladkal
eladkal reviewed Nov 15, 2024
View reviewed changes
Comment threadproviders/src/airflow/providers/amazon/aws/operators/emr.py Outdated
@adrianlzradrianlzr changed the title Add wait_for_steps_completion option to EmrCreateJobFlowOperator.Add wait_policy option to EmrCreateJobFlowOperator.Nov 18, 2024
eladkal
eladkal reviewed Nov 18, 2024
View reviewed changes
Comment threadproviders/src/airflow/providers/amazon/aws/operators/emr.py Outdated
Comment threadproviders/src/airflow/providers/amazon/aws/utils/waiter.py Outdated
@vincbeck

vincbeck commented Nov 18, 2024

Copy link
Copy Markdown
Contributor

Good job! I like that version

eladkal
eladkal reviewed Nov 18, 2024
View reviewed changes
Comment threadproviders/src/airflow/providers/amazon/aws/operators/emr.py
@adrianlzr

adrianlzr commented Nov 21, 2024

Copy link
Copy Markdown
ContributorAuthor

I'm not sure what's the next step here but from my side everything is ready.

vincbeck
vincbeck requested changes Nov 21, 2024
View reviewed changes

@vincbeckvincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! Just one nit and you do not a newsfragment for this change

Comment threaddocs/apache-airflow-providers-amazon/operators/emr/emr.rst Outdated
Comment threadnewsfragments/44055.feature.rst Outdated
vincbeck
vincbeck approved these changes Nov 21, 2024
View reviewed changes

@vincbeckvincbeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect!

@adrianlzr

adrianlzr commented Nov 25, 2024•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Should be ready for merge

Adrian Lazar
Add wait_policy option to EmrCreateJobFlowOperator.
de09376
Possible values:
- None: No wait (default)
- WaitPolicy.WAIT_FOR_COMPLETION: Previous behaviour when wait_for_completion was True
- WaitPolicy.WAIT_FOR_STEPS_COMPLETION: New behaviour - wait for the cluster to terminate.
@vincbeck
vincbeck merged commit b4c4806 into apache:mainNov 25, 2024
@boring-cyborg

boring-cyborgBot commented Nov 25, 2024

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@eladkaleladkal mentioned this pull request Dec 22, 2024
Closed
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
@adrianlzr@got686-yandex
Addwait_policyoption toEmrCreateJobFlowOperator(apache#44055)
f260384
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eladkaleladkaleladkal approved these changes
@vincbeckvincbeckvincbeck approved these changes
@o-nikolaso-nikolasAwaiting requested review from o-nikolas
+1 more reviewer
@vin100bkvin100bkvin100bk approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:providersprovider:amazonAWS/Amazon - related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@adrianlzr@vincbeck@vin100bk@eladkal