Fix CloudwatchTaskHandler not deleting local logs after streaming by sam-dumont · Pull Request #62985 · apache/airflow

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont · 2026-03-06T09:53:35Z

Fix CloudwatchTaskHandler not deleting local log files after streaming to CloudWatch.

In Airflow 3, logs stream to CloudWatch in real-time via structlog processors, so upload() was a no-op. But delete_local_copy was never honoured: local log files kept accumulating on shared storage indefinitely.

We hit this in production on EFS. Storage grew at ~31 GB/day from log accumulation alone. After deploying this fix, net growth dropped to effectively zero. A separate cleanup job removed the ~780 GB backlog that had built up.

What changed:

CloudWatchRemoteLogIO.upload() deletes the local log parent directory when delete_local_copy is True
CloudwatchTaskHandler.close() calls upload() with the rendered log path stored during set_context()
Path traversal guard: upload() refuses to delete anything outside base_log_folder

Was generative AI tooling used to co-author this PR?

Yes — Claude Code (Claude Opus 4.6)

Generated-by: Claude Code (Claude Opus 4.6) following the guidelines

boring-cyborg · 2026-03-06T09:53:38Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`.

Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

boring-cyborg · 2026-03-09T14:27:48Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…ache#62985) * Fix CloudwatchTaskHandler not deleting local logs after streaming In Airflow 3, `CloudwatchTaskHandler` streams logs to CloudWatch in real-time via structlog processors, so `upload()` was a no-op. However, `delete_local_copy` was never honoured, causing local log files to accumulate indefinitely on shared storage (e.g. EFS). Changes: - `CloudWatchRemoteLogIO.upload()` now deletes the local log directory when `delete_local_copy` is True. - `CloudwatchTaskHandler.close()` calls `upload()` with the rendered log path stored during `set_context()`. - Path traversal guard prevents deletion outside `base_log_folder`. * chore: rename newsfragment to PR apache#62985 * Remove provider newsfragment (not used for providers) * fix: align close() guard and tests with S3/GCS/HDFS convention Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont requested a review from o-nikolas as a code owner March 6, 2026 09:53

boring-cyborgBot added area:logging area:providers provider:amazon AWS/Amazon - related issues labels Mar 6, 2026

sam-dumont force-pushed the fix/cloudwatch-delete-local-logs branch from 1096a14 to c21c3ffCompare March 6, 2026 09:54

sam-dumont added a commit to sam-dumont/airflow that referenced this pull request Mar 6, 2026
chore: rename newsfragment to PRapache#62985
c21c3ff
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

eladkal reviewed Mar 6, 2026
View reviewed changes

Comment threadproviders/amazon/newsfragments/62985.bugfix.rst Outdated

eladkal requested a review from vincbeck March 6, 2026 09:59

sam-dumont added 3 commits March 6, 2026 11:50

chore: rename newsfragment to PRapache#62985
5ca818c

Remove provider newsfragment (not used for providers)
56bd864

sam-dumont force-pushed the fix/cloudwatch-delete-local-logs branch from 6c273b4 to b8412a0Compare March 6, 2026 10:51

fix: align close() guard and tests with S3/GCS/HDFS convention
7c95018
Use hasattr(self, "ti") guard in close() like S3/GCS/HDFS handlers. Fix tests to be version-agnostic by asserting against handler's own log_relative_path instead of hardcoded filename format.

sam-dumont force-pushed the fix/cloudwatch-delete-local-logs branch from 8a1134c to 7c95018Compare March 6, 2026 10:56

Merge branch 'main' into fix/cloudwatch-delete-local-logs
80f9d1d

vincbeck approved these changes Mar 9, 2026
View reviewed changes

vincbeck merged commit e478004 into apache:mainMar 9, 2026
92 checks passed

vincbeck mentioned this pull request Mar 10, 2026
Status of testing Providers that were prepared on March 10, 2026 #63279
Closed
81 tasks