Skip to content

Add HdfsRemoteLogIO.from_config and register hdfs remote logging scheme - #71278

Merged
jason810496 merged 1 commit into
apache:mainfrom
baha-bouali:feature/logging/hdfs-remote-log-io-from-config
Aug 10, 2026
Merged

Add HdfsRemoteLogIO.from_config and register hdfs remote logging scheme#71278
jason810496 merged 1 commit into
apache:mainfrom
baha-bouali:feature/logging/hdfs-remote-log-io-from-config

Conversation

@baha-bouali

Copy link
Copy Markdown
Contributor

Part of #70265 (related: #67056). Migrates #70270.

Why

#67056 decoupled remote logging from the hardcoded branches in airflow_local_settings.py:
core and the Task SDK now resolve the handler via ProvidersManager dispatch on the
[logging] remote_base_log_folder URL scheme, instantiating the provider class through a
no-arg from_config() classmethod. This migrates the hdfs scheme.

What

  • Add HdfsRemoteLogIO.from_config(), mirroring the legacy airflow_local_settings.py
    branch — including stripping to urlsplit(...).path for remote_base (the legacy branch
    drops the hdfs://host part and keeps only the path), the
    [logging] remote_task_handler_kwargs IO-kwargs merge, and expanduser on
    base_log_folder, so behavior is unchanged for existing configs.
  • Register the hdfs scheme under remote-logging: in providers/apache/hdfs/provider.yaml
    and mirror it in get_provider_info.py.
  • Add tests mirroring TestS3RemoteLogIOFromConfig from
    providers/amazon/tests/unit/amazon/aws/log/test_s3_task_handler.py.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    Assisted by Claude Sonnet 5, Reviewed by the submitter

Signed-off-by: baha-bouali <baha.bouali2020@gmail.com>

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, LGTM for the migration.

@jason810496
jason810496 merged commit 3160334 into apache:mainAug 10, 2026
79 checks passed
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
…me (apache#71278)
Signed-off-by: baha-bouali <baha.bouali2020@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@baha-bouali@jason810496