BashOperator: Execute templated bash script as file

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.

[Joffreybvn]

This PR change the behavior of the BashOperator: When executing a .sh file with Jinja templating enabled (without the space -- .sh ), the rendered script is written back into a temporary file, and then executed. Instead of being directly executed as inline command.

Note: No changes / side-effect when executing a inline command or a script without Jinja templating ( .sh with space).

Why ?

When using templated bash command, the rendered string can be longer than the maximum argument length. This causes "Argument list too long" error.

Without this PR, currently, the bash command is passed as argument into ["bash", "-c", command]. The command cannot be longer than the max length of the system.

With this PR, the user can write his command into a bash file. The file is jinja-rendered and executed as a file, which bypass the maximum argument length limit.

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

Backport #43191 warns the user when permissions are missing, and fallback to executing the script as inline command.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[eladkal]

Breaking change

The BashOperator writes the rendered file to the disk. By default, it's in a temporary directory (/tmp) where Airflow has write access (in a common Linux system). However, if the user specify a cwd directory where Airflow cannot write, task will fail.

IMO this will affect only a small portion of users. Should I add a flag to control the behavior of the Operator ? Or maybe another PR with a warning that behavior will change in the future ?

We need newsfragment in the PR that explains the change.
We need a new PR against v 2.10 test branch that adds the warnning and allow users to migrate to the new functionality. We show warning only if user can do something about it

[Joffreybvn]

Created a new PR #43191, which implements this feature with a fallback to the former behaviour in case of permission issue; and a warning for the user to fix it.