Fix remote processor injection happening before dictConfig runs in configure_logging

[korex-f]

Problem

In configure_logging(), getattr(remote, "processors") was called before dictConfig() ran. dictConfig() calls _clearExistingHandlers() which closes every handler registered in logging._handlerList, including the watchtower handler built moments earlier. This caused CloudWatch/Watchtower logs to be silently dropped when using the Task SDK with remote logging.

Fix

Move the remote processor injection to after dictConfig() has run. The handler is now built into a clean logging state and cannot be killed prematurely.

Also gate the injection on not sending_to_supervisor to avoid unmasked events from the task subprocess, and add a None default to getattr to handle third-party RemoteLogIO objects missing the processors attribute.

Related

Provider-side fix: #68779

closes: #66475

---

Important

🛠️ Maintainer triage note for @korex-f · by @potiuk · 2026-07-02 17:46 UTC

Some review feedback from @jason810496 is waiting on you:

• 6 unresolved review thread(s) from @jason810496 need a reply or a fix.

The ball is in your court — you've been assigned to this PR. Reply or push a fix in each thread, then mark them resolved.

_{Automated triage — may be imperfect; a maintainer takes the next look.}

[chris-stetter]

I can confirm that this works in our ECS environment. Thank you!

[korex-f]

You're very much welcome.

[o-nikolas]

I would have expected some new or more updated test cases to ensure this doesn't regress again? Do you not think that's required here @korex-f ?

[korex-f]

I would have expected some new or more updated test cases to ensure this doesn't regress again? Do you not think that's required here @korex-f ?

Good point, I have added tests to test_cloudwatch_task_handler.py covering the self-healing handler property (verifies a new handler is created when shutting_down=True) and the dynamic self.handler access in the processors closure.

[o-nikolas]

Looks reasonable to me now, but I still think @ashb or @amoghrajesh should have a look

[potiuk]

@korex-f — There are 2 unresolved review threads on this PR from @o-nikolas. Could you either push a fix or reply in each thread explaining why the feedback doesn't apply? Once you believe the feedback is addressed, mark the thread as resolved so the reviewer isn't re-pinged needlessly. Thanks!

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[seanghaeli]

Verified end-to-end this fixes remote logging. Thanks @korex-f this issue has been floating around for a while.

[1fanwang]

Thanks for addressing the comments, lgtm!

[potiuk]

@korex-f — A reviewer (@ashb) has requested changes on this PR, so I've removed the ready for maintainer review label — the next step is on your side. Could you address the review comments (push a fix, or reply in-thread explaining why the feedback doesn't apply)? Once addressed, re-request review from @ashb or re-mark the PR ready and it returns to the maintainer queue. Thank you.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[korex-f]

Thanks for the review, @ashb. I have addressed the requested changes, pushed the fixes, and re-requested review. Please do take another look when you have the chance.

[sarvesh371]

Hey, running into this exact issue on Airflow 3.2.2 with KubernetesExecutor — task pods emit WatchtowerWarning: Received message after logging system shutdown and logs never make it to CloudWatch.

Any idea on ETA for merge and which provider version will ship the fix? Also is there any interim workaround while we wait?

Happy to test if there's a pre-release available.

[korex-f]

Addressed all outstanding review comments:

• Reverted unrelated changes in _handle_request (GetXComCount ordering and send_msg parameter style)
• Restored the dropped TODO comment in _configure_logging
• Used walrus operator for load_remote_log_handler() check
• Added hasattr(handler, "close") guard in _close_remote_log_handler to avoid mypy error after reverting the RemoteLogIO protocol change
• Reverted all provider and shared/logging changes per @jason810496 guidance, this PR now touches only supervisor.py and log.py

[eladkal]

Can the provider related changes be split into a dedicated PR?

[korex-f]

Can the provider related changes be split into a dedicated PR?

Done!
All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.

[o-nikolas]

@jason810496 and @ferruzzi, does this look good to approve/merge for y'all?

@ashb it looks like your feedback was addressed, but you "requested changes" which makes me think you saw something very serious, are you happy with this PR now?

[ferruzzi]

Can the provider related changes be split into a dedicated PR?

Done! All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.

You may not have pushed that, there are still provider changes in the current diff:

 diff --git a/providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
diff --git a/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
+ if ti is None:
+ return
diff --git a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
+ if ti is None:
+ return

Edit to add: the whitespace change in supervisor.py that you said was done is also not there.

[korex-f]

Can the provider related changes be split into a dedicated PR?

Done! All provider and shared/logging changes were already reverted in the latest push per @jason810496's guidance. This PR now touches only task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. The provider-side fix is being handled separately in #68779.

You may not have pushed that, there are still provider changes in the current diff:

 diff --git a/providers/amazon/src/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
diff --git a/providers/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
+ if ti is None:
+ return
diff --git a/providers/opensearch/src/airflow/providers/opensearch/log/os_task_handler.py
- def upload(self, path: os.PathLike | str, ti: RuntimeTI):
+ def upload(self, path: os.PathLike | str, ti: RuntimeTI | None = None) -> None:
+ if ti is None:
+ return

Edit to add: the whitespace change in supervisor.py that you said was done is also not there.

Sorry for the confusion, the revert was there but spread across multiple commits, making it look like provider changes were still present in the per-commit view. The net diff against upstream/main now only touches task-sdk/src/airflow/sdk/execution_time/supervisor.py and task-sdk/src/airflow/sdk/log.py. No provider files are modified.

[jason810496]

Here're final nits that would appreciate be addressed before merge, thanks.

[kaxil]

Picking up on @o-nikolas's merge question -- I think this needs a second look at scope before it goes in, because a fair bit of it landed via other PRs while this was open:

• Fix CloudWatch remote logging for ephemeral lifecycle executor #68779 (merged) made the CloudWatch RemoteLogIO self-healing: the handler property rebuilds if dictConfig closed it mid-task, and upload() now calls close() to flush pending events. That covers the Cloudwatch remote logging does not work for ECS Executor #66475 symptom at the provider layer.
• Upload task logs even when final state update fails #67935 (merged) moved _upload_logs() into a finally after update_task_state_if_needed(), so logs upload even when the state update raises.
• Remote logging fix #68370 (merged) fixed the cached-None-connection case so the worker can upload at all.

Given that, the supervisor.py side here (explicit _close_remote_log_handler on teardown) now overlaps with _upload_logs() → upload() → close(), which already runs on the state-update error path after #67935. The log.py reorder (build the handler after dictConfig) is the one piece not on main, but with the self-healing handler it's no longer a live drop -- more of a cleaner-root-cause argument.

@korex-f -- is there a failure mode left that the three merged PRs don't cover? If the log.py reorder is the remaining value, it may be worth narrowing this to just that (with a test); otherwise it looks mostly superseded now. Not trying to discount the work here, just want to make sure we're not re-landing something.

[korex-f]

Picking up on @o-nikolas's merge question -- I think this needs a second look at scope before it goes in, because a fair bit of it landed via other PRs while this was open:

* **[Fix CloudWatch remote logging for ephemeral lifecycle executor #68779](https://github.com/apache/airflow/pull/68779)** (merged) made the CloudWatch `RemoteLogIO` self-healing: the `handler` property rebuilds if `dictConfig` closed it mid-task, and `upload()` now calls `close()` to flush pending events. That covers the [Cloudwatch remote logging does not work for ECS Executor #66475](https://github.com/apache/airflow/issues/66475) symptom at the provider layer.
* **[Upload task logs even when final state update fails #67935](https://github.com/apache/airflow/pull/67935)** (merged) moved `_upload_logs()` into a `finally` after `update_task_state_if_needed()`, so logs upload even when the state update raises.
* **[Remote logging fix #68370](https://github.com/apache/airflow/pull/68370)** (merged) fixed the cached-`None`-connection case so the worker can upload at all.

Given that, the supervisor.py side here (explicit _close_remote_log_handler on teardown) now overlaps with _upload_logs() → upload() → close(), which already runs on the state-update error path after #67935. The log.py reorder (build the handler after dictConfig) is the one piece not on main, but with the self-healing handler it's no longer a live drop -- more of a cleaner-root-cause argument.

@korex-f -- is there a failure mode left that the three merged PRs don't cover? If the log.py reorder is the remaining value, it may be worth narrowing this to just that (with a test); otherwise it looks mostly superseded now. Not trying to discount the work here, just want to make sure we're not re-landing something.

Thanks for the detailed audit @kaxil — really helpful context.

You're right that #68779, #67935, and #68370 together cover the provider-layer symptom. Looking at what's left:

log.py reorder (Bug 1):getattr(remote, "processors") was called before dictConfig ran, which meant dictConfig's _clearExistingHandlers closed the just-built watchtower handler before any task log was emitted. With #68779's self-healing property this is no longer a live drop, the handler rebuilds on next access, but the reorder is still the correct root-cause fix and removes the need for the rebuild in the first place. It's a cleaner argument even if it's no longer strictly necessary for correctness.

supervisor.py _close_remote_log_handler: You're right this now overlaps with upload() → close() from #67935/#68779. I'm happy to drop the supervisor.py changes entirely if you think log.py alone is worth keeping, or close the whole PR if you think it's fully superseded. I'd rather do the right thing for the codebase than land something redundant.

What would you prefer?

[jason810496]

Thank you for the update.

is there a failure mode left that the three merged PRs don't cover?

Yes log.py reorder (Bug 1): getattr(remote, "processors") was called before dictConfig ran, which meant dictConfig's _clearExistingHandlers closed the just-built watchtower handler before any task log was emitted. statement is correct.

I'd like to keep the only the log.py changes as that is the key point to resolve the unexpected .close on logging handler. The supervisors.py change is more "nice to have" but it might introduce further regression. Let's keep the change minimal if possible, thanks.

[korex-f]

Thank you for the update.

is there a failure mode left that the three merged PRs don't cover?

Yes log.py reorder (Bug 1): getattr(remote, "processors") was called before dictConfig ran, which meant dictConfig's _clearExistingHandlers closed the just-built watchtower handler before any task log was emitted. statement is correct.

I'd like to keep the only the log.py changes as that is the key point to resolve the unexpected .close on logging handler. The supervisors.py change is more "nice to have" but it might introduce further regression. Let's keep the change minimal if possible, thanks.

Thanks for the clarity. Reverting all supervisor.py changes now, this PR will only contain the log.py reorder (Bug 1). supervisor.py can be revisited in a follow-up if needed.

[jason810496]

Hi @korex-f,

Thanks for follow-up on this.
We need to do the following items before merge.

• Add unit test to exercise the patch itself
• Fix the CI failure
• Rephrase the PR title and the description

Thanks.

[korex-f]

Hi @korex-f,

Thanks for follow-up on this. We need to do the following items before merge.

* Add unit test to exercise the patch itself
* Fix the CI failure
* Rephrase the PR title and the description

Thanks.

Added a regression test in test_log.py that verifies load_remote_log_handler() is called only after the inner configure_logging() (dictConfig) returns. The CI failure in test_inject_parent_job_info_with_resume_on_retry is unrelated, it's a Glue operator test making a real network call that times out, last touched in #64513. Happy to re-trigger CI to confirm it's a flake.

[jason810496]

#70938 - fix the OpenAI provider CI failure. Rebasing on top of the latest main should fix. Thanks.

[jason810496]

LGTM overall now, thanks.

[korex-f]

@kaxil
Thanks for the detailed review, i have addressed all three points:

• Dropped the three test_warns_* tests and reverted upload_to_remote to the simple version
• Removed the cached_property mention from the log.py comment
• Fixed TestConfigureLogging to save and restore global structlog processor state in a finally block and clear the cache on exit

[jason810496]

Nice!

[korex-f]

Nice!

Thanks for the guidance.