Decouple remote logging config from core by jason810496 · Pull Request #67056 · apache/airflow

jason810496 · 2026-05-17T06:26:47Z

related: Decouple remote logging configuration from core airflow #65281
related: Clarify logging_config_class and deprecate ES/OS implicit REMOTE_TASK_LOG registration #66994

Why

Decouple remote logging config from hardcoded airflow-core/src/airflow/config_templates/airflow_local_settings.py by introducing new remote-logging field for ProvidersManager. Add shared resolve_remote_task_log it, so core and the Task SDK
share one implementation of the precedence rule.

What

Single shared resolver resolve_remote_task_log with an explicit three-tier precedence:

User-defined [logging] logging_config_class module exporting
REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID.
ProvidersManager scheme dispatch on [logging] remote_base_log_folder,
instantiated via the provider class's from_config() classmethod.
Legacy attr-path fallback against
airflow.config_templates.airflow_local_settings (transitional).

airflow.logging_config.load_logging_config is deprecated and split into

_get_logging_config() (logging dict)
_load_logging_config() (remote handler, lazy)
configure_logging no longer eagerly resolves the remote handler.

Next step to deprecate hardcoded `airflow_local_settings.py`

Provider migration: Migrate the logic from airflow_local_settings.py as from_config method for each current supported RemoteIO.

classS3RemoteLogIO(LoggingMixin):
@classmethoddeffrom_config(cls) ->"S3RemoteLogIO":
fromairflow.providers.common.compat.sdkimportconf# provider-specific `conf` validation or custom logicreturncls(
base_log_folder=conf.get("logging", "base_log_folder"),
remote_base=conf.get("logging", "remote_base_log_folder"),
delete_local_copy=conf.getboolean("logging", "delete_local_logs"),
)

Raise deprecation warning for each if/else branch after the corresponding RemoteIO.from_config is added.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code (Opus 4.7) following the guidelines

jscheffl

Looks clean and good to me - except some small comments. Another pair of eyes would be good prior merge.

Fix type annotation

…conn id

…config Decouple importing the [logging] logging_config_class dict from remote-handler resolution so each can run independently. The deprecated load_logging_config wrapper delegates to both and keeps the old return shape. Wire the lazy remote-handler load through the shared resolve_remote_task_log factory.

eladkal

LGTM
but need a 2nd reviewer to check it as well

jscheffl

As I was first reviewer, I think LGTM

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the cloudwatch scheme.

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the s3 scheme as the first adopter.

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the cloudwatch scheme.

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the s3 scheme as the first adopter.

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the cloudwatch scheme.

…69816) Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since #67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the cloudwatch scheme.

Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since apache#67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the s3 scheme as the first adopter.

…69817) Airflow resolves remote task log handlers through provider dispatch on the remote_base_log_folder URL scheme since #67056; providers must expose from_config so core and the Task SDK no longer depend on the hardcoded branches in airflow_local_settings.py. This migrates the s3 scheme as the first adopter.

Core resolves remote log handlers by URL scheme through ProvidersManager dispatch (apache#67056); s3 and cloudwatch already migrated. This moves wasb onto the same path, so Azure Blob remote logging is built by the provider's from_config() instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config failure falls back to the legacy path, so behaviour is unchanged. Part of apache#70265. closesapache#70268.

…me (#70301) Core resolves remote log handlers by URL scheme through ProvidersManager dispatch (#67056); s3 and cloudwatch already migrated. This moves wasb onto the same path, so Azure Blob remote logging is built by the provider's from_config() instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config failure falls back to the legacy path, so behaviour is unchanged. Part of #70265. closes#70268.

…me (apache#70301) Core resolves remote log handlers by URL scheme through ProvidersManager dispatch (apache#67056); s3 and cloudwatch already migrated. This moves wasb onto the same path, so Azure Blob remote logging is built by the provider's from_config() instead of the hardcoded branch in airflow_local_settings.py. Existing wasb:// configs resolve to an equivalent handler, and a from_config failure falls back to the legacy path, so behaviour is unchanged. Part of apache#70265. closesapache#70268.

boring-cyborgBot added area:logging area:task-sdk labels May 17, 2026

jason810496 self-assigned this May 17, 2026

jason810496 marked this pull request as ready for review May 18, 2026 02:15

jason810496 requested review from amoghrajesh, ashb, kaxil and potiuk as code owners May 18, 2026 02:15

jason810496 requested review from Lee-W, jscheffl and shahar1 May 18, 2026 05:16

jscheffl reviewed May 18, 2026
View reviewed changes

Comment threadairflow-core/newsfragments/67056.significant.rst Outdated

jscheffl reviewed May 18, 2026
View reviewed changes

Comment threadairflow-core/src/airflow/providers_manager.py Outdated

jscheffl reviewed May 18, 2026
View reviewed changes

jason810496 added 7 commits May 19, 2026 12:32

Decouple remote logging configuration from core airflow
4b84e61
Fix type annotation

Align get_default_remote_conn_id with load_remote_conn_id
7b8f835

Make _build_remote_task_log_from_provider return RemoteLogIO without …
e8bb6b7
…conn id

Add significant md
61e1d90

Fix: Respct ("logging", "remote_logging") in shared logging factory
f263406

Addressed Jens comments
37939dc

jason810496 force-pushed the refactor/logging/decuple-logging-config-from-core branch from aa513bb to 37939dcCompare May 19, 2026 04:32

jason810496 added this to the Airflow 3.3.0 milestone May 25, 2026

eladkal approved these changes Jun 4, 2026
View reviewed changes

jscheffl approved these changes Jun 4, 2026
View reviewed changes

jscheffl merged commit ab1418f into apache:mainJun 4, 2026
143 checks passed

This was referenced Jul 13, 2026
Add CloudWatchRemoteLogIO.from_config and register cloudwatch scheme #69816
Merged
Add S3RemoteLogIO.from_config and register s3 remote logging scheme #69817
Merged

FrankYang0529 mentioned this pull request Jul 23, 2026
Add OpensearchRemoteLogIO.from_config and register opensearch scheme #70295
Merged
1 task

Andrushika mentioned this pull request Jul 23, 2026
Add WasbRemoteLogIO.from_config and register wasb remote logging scheme #70301
Merged
1 task

shahar1 mentioned this pull request Jul 24, 2026
Status of testing Providers that were prepared on July 22, 2026 #70355
Closed

aaron-y-chen mentioned this pull request Jul 27, 2026
Enable provider-based GCS remote logging resolution #70504
Merged
1 task

guruprasad-boop mentioned this pull request Jul 27, 2026
Register GCS remote logging scheme via provider dispatch (gs://) Shreevable/airflow#1
Open
1 task

yuseok89 mentioned this pull request Jul 27, 2026
Add ElasticsearchRemoteLogIO.from_config and register elasticsearch scheme #70525
Merged
1 task

baha-bouali mentioned this pull request Jul 29, 2026
Add OSSRemoteLogIO.from_config and register oss remote logging scheme #70682
Merged

This was referenced Aug 3, 2026
Status of testing Providers that were prepared on August 01, 2026 #70953
Closed
Status of testing Providers that were prepared on August 06, 2026 #71246
Closed

baha-bouali mentioned this pull request Aug 7, 2026
Add HdfsRemoteLogIO.from_config and register hdfs remote logging scheme #71278
Merged
1 task

potiuk mentioned this pull request Aug 8, 2026
Status of testing Providers that were prepared on August 08, 2026 #71321
Closed
32 tasks

hussein-awala mentioned this pull request Aug 19, 2026
Status of testing Providers that were prepared on August 18, 2026 #71847
Closed
91 tasks

Navigation Menu

Uh oh!

Decouple remote logging config from core - #67056

Decouple remote logging config from core#67056
jscheffl merged 7 commits into
apache:mainfrom
jason810496:refactor/logging/decuple-logging-config-from-core

jason810496 commented May 17, 2026•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

jscheffl left a comment

Uh oh!

eladkal left a comment

Uh oh!

jscheffl left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jason810496 commented May 17, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Next step to deprecate hardcoded airflow_local_settings.py

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

eladkal left a comment

Choose a reason for hiding this comment

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jason810496 commented May 17, 2026•
edited
Loading

Next step to deprecate hardcoded `airflow_local_settings.py`