Add support in AWS Batch Operator for multinode jobs

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.

[camilleanne]

• Adds support for AWS Batch multinode jobs by allowing a node_overrides json object to be passed through to the boto3 submit_job method.
• Adds support for multinode jobs by properly parsing the output of describe_jobs (which is different for container vs multinode) to extract the log stream name.

closes: #25522

I had a hard time running tests locally, so I'm opening as a draft PR initially although I don't anticipate any changes beyond syncing with main, but I'd like to confirm test success before making available for review.

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Taragolis]

Just a question. Did you check are this arguments mutually exclusive?
AWS API doesn't mention it however everything might possible because even new eksPropertiesOverride not marked as exclusive.

[vandonr-amz]

yes they are, specifying both returns an input validation error.
BTW @Taragolis , could you approve running the workflow to help get this PR ready-to-review ? 🙏

[camilleanne]

Ok I got a handle on all tests finally :) ready for review now.

[vandonr-amz]

Is it possible to have zero element in the array ? i.e. should we add a check on len == 0 and a user-friendly error message ?

[vandonr-amz]

Maybe this could be an error log, considering the user-provided input is invalid for this kind of request ?

The other warning logs in this method are mostly informative (there are several node groups, which is important info for the user to know, but doesn't require any action), this one I think requires user action, and thus more attention.

[vandonr-amz]

beautiful 😄

[vandonr-amz]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class, so:

Suggested change

	"overrides",
	"container_overrides",

Thinking about it, I wonder if this is a breaking change... I don't know too well how this works 😬

[Taragolis]

@camilleanne@vandonr-amz I recommend deprecate old parameter but keep it for a while, so users would have a time for change their code.
It is much easier achieve in subclass of BaseOperator (this PR case), because all arguments are keyword.

Some examples

Use old attribute value only if new attribute not set

airflow/airflow/providers/slack/operators/slack_webhook.py

Lines 95 to 104 in 11f30a8

	http_conn_id=kwargs.pop("http_conn_id", None)
	ifhttp_conn_id:
	warnings.warn(
	"Parameter `http_conn_id` is deprecated. Please use `slack_webhook_conn_id` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifslack_webhook_conn_id:
	raiseAirflowException("You cannot provide both `slack_webhook_conn_id` and `http_conn_id`.")
	slack_webhook_conn_id=http_conn_id

Use old attribute value only if it equal new attribute value or not new attribute not set

airflow/airflow/providers/amazon/aws/operators/athena.py

Lines 91 to 101 in 11f30a8

	ifmax_tries:
	warnings.warn(
	f"Parameter `{self.__class__.__name__}.max_tries` is deprecated and will be removed "
	"in a future release. Please use method `max_polling_attempts` instead.",
	DeprecationWarning,
	stacklevel=2,
	)
	ifmax_polling_attemptsandmax_polling_attempts!=max_tries:
	raiseException("max_polling_attempts must be the same value as max_tries")
	else:
	self.max_polling_attempts=max_tries

[vandonr-amz]

given the fact that it was just a rename for readability, I wonder if it wouldn't be simpler to just keep the old name ?
but then it could be confusing to users...

[Taragolis]

I thought it fine if rename this argument because one day we might also add eks_properties_overrides, so override might confuse users more rather than change attribute name.
And deprecation warning give time to change arguments in end users DAG code.

[ferruzzi]

TIL the names here need to match not the parameter name in the ctor but the name of the attribute in the class

To translate that into ELI5: if you have self.foo = bar in your operator, the template field would be named "foo" to match "self.foo", not "bar". :P

[o-nikolas]

Hey @camilleanne,

Any plans to pick this one up again?

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

[o-nikolas]

@vandonr-amz is taking on this work on in #29522

Closing this PR with that context.