Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104

Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged

Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom
jason810496:refactor/logging/clarify-logging-config-class-core

Conversation

@jason810496

@jason810496jason810496 commented May 18, 2026•
edited
Loading

Copy link
Copy Markdown
Member

Why

[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.

So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.

How

  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
Merged
Merged
@jason810496jason810496 self-assigned this May 18, 2026
@amoghrajesh

amoghrajesh commented May 18, 2026

Copy link
Copy Markdown
Contributor

@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon

@eladkal

eladkal commented May 20, 2026

Copy link
Copy Markdown
Contributor

I think we will need to bump

airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92 in 6082d6a

"elasticsearch": parse_version("6.5.0"),
"opensearch": parse_version("1.9.0"),

as part of this PR but we will need to wait for providers to be released

@eladkaleladkal added this to the Airflow 3.3.0 milestone May 23, 2026
@eladkal

eladkal commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

@jason810496 can you fix the problem above?

phanikumv
phanikumv reviewed Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml Outdated
@jason810496
jason810496 marked this pull request as draft June 5, 2026 14:22
@jason810496
jason810496force-pushed the refactor/logging/clarify-logging-config-class-core branch 3 times, most recently from 584e906 to eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496 marked this pull request as ready for review June 5, 2026 14:44
jason810496
jason810496 commented Jun 5, 2026
View reviewed changes

@jason810496jason810496 left a comment •
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.

@jason810496
jason810496 marked this pull request as draft June 9, 2026 02:22
@jason810496
jason810496force-pushed the refactor/logging/clarify-logging-config-class-core branch from eef5d24 to 654cbecCompareJune 9, 2026 02:48
jason810496
jason810496 commented Jun 9, 2026
View reviewed changes

@jason810496jason810496 left a comment

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,
I some help to resolve the static check CI failure when you have a moment. Thanks.

If I don't update the root pyproject.toml I will encounter:

Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727

However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:

Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717

@eladkal

eladkal commented Jun 9, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.
I assume release manager @vatsrahul1001 is OK with this

@eladkal
eladkalforce-pushed the refactor/logging/clarify-logging-config-class-core branch from b5b681b to 2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal added the backport-to-v3-3-test Backport to v3-3-test label Jul 2, 2026
@jason810496jason810496 mentioned this pull request Jul 2, 2026
Merged
1 task
@jason810496
jason810496force-pushed the refactor/logging/clarify-logging-config-class-core branch 2 times, most recently from 9079e0a to ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W reviewed Jul 6, 2026
View reviewed changes

@Lee-WLee-W left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major

Comment threadairflow-core/src/airflow/logging_config.py Outdated
Comment threadairflow-core/tests/unit/logging/test_logging_config.py Outdated
Comment threadairflow-core/tests/unit/logging/test_logging_config.py Outdated
jason810496 added a commit to jason810496/airflow that referenced this pull request Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W reviewed Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py Outdated
@jason810496
jason810496 requested a review from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed the refactor/logging/clarify-logging-config-class-core branch from 66b52a6 to 9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023 reviewed Jul 15, 2026
View reviewed changes

@SZL741023SZL741023 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.

Comment threadairflow-core/src/airflow/logging_config.py Outdated
@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001 merged commit d8b8620 into apache:mainJul 28, 2026
303 checks passed
@github-actionsgithub-actionsBot mentioned this pull request Jul 28, 2026
Merged
@github-actions

github-actionsBot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
✅v3-3-testPR Link

github-actionsBot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit that referenced this pull request Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv
+1 more reviewer
@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects

None yet

Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Clarify logging_config_class contract and document REMOTE_TASK_LOG - #67104
Merged
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Jul 28, 2026
Merged
Clarify logging_config_class contract and document REMOTE_TASK_LOG#67104
vatsrahul1001 merged 8 commits into
apache:mainfrom 
jason810496:refactor/logging/clarify-logging-config-class-core
Conversation
@jason810496
@jason810496jason810496
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

Member
Why
[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.
So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.
How
  • Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
  • Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_loggingafterdictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
  • Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
  • Add unit tests for _warn_if_missing_remote_task_log.
Was generative AI tooling used to co-author this PR?

This was referenced May 18, 2026
 Merged

 Merged

@jason810496jason810496
self-assigned this
May 18, 2026
@amoghrajesh
amoghrajesh
commented
May 18, 2026

Copy link


Copy Markdown

Contributor
@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon
@eladkal
eladkal
commented
May 20, 2026

Copy link


Copy Markdown

Contributor
I think we will need to bump
airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92
in
6082d6a
"elasticsearch": parse_version("6.5.0"), 
"opensearch": parse_version("1.9.0"), 


as part of this PR but we will need to wait for providers to be released
@eladkaleladkal
added this to the Airflow 3.3.0 milestone
May 23, 2026
@eladkal
eladkal
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
@jason810496 can you fix the problem above?
phanikumv
phanikumv
reviewed
Jun 4, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/config.yml
Outdated

@jason810496
jason810496
marked this pull request as draft
June 5, 2026 14:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
3 times, most recently
from
584e906 to
eef5d24CompareJune 5, 2026 14:43
@jason810496
jason810496
marked this pull request as ready for review
June 5, 2026 14:44
jason810496
jason810496
commented
Jun 5, 2026
View reviewed changes
@jason810496jason810496
left a comment
•

edited

Loading

Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review Elad, Phani. I addressed both comments in last rebase.
@jason810496
jason810496
marked this pull request as draft
June 9, 2026 02:22
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
eef5d24 to
654cbecCompareJune 9, 2026 02:48
jason810496
jason810496
commented
Jun 9, 2026
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

MemberAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @eladkal,

I some help to resolve the static check CI failure when you have a moment. Thanks.
If I don't update the root pyproject.toml I will encounter:
Update Airflow's meta-package pyproject.toml............................................................Failed
- hook id: update-pyproject-toml
- files were modified by this hook
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-edge3>=1.0.0"
]
"elasticsearch" = [
- "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"exasol" = [
"apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opensearch" = [
- "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
]
"opsgenie" = [
"apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-discord>=3.9.0",
"apache-airflow-providers-docker>=3.14.1",
"apache-airflow-providers-edge3>=1.0.0",
- "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-exasol>=4.6.1",
"apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
"apache-airflow-providers-openai>=1.5.0",
"apache-airflow-providers-openfaas>=3.7.0",
"apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
- "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+ "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
"apache-airflow-providers-opsgenie>=5.8.0",
"apache-airflow-providers-oracle>=3.12.0",
"apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727
However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:
Provider dependency version bumps detected that should only be performed by Release Managers!
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
- pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
- pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717
@eladkal
eladkal
commented
Jun 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.

I assume release manager @vatsrahul1001 is OK with this
@eladkal
eladkalforce-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
b5b681b to
2f5ae76CompareJuly 2, 2026 04:29
@eladkaleladkal
added
the 
backport-to-v3-3-test
Backport to v3-3-test
label
Jul 2, 2026
@jason810496jason810496
mentioned this pull request
Jul 2, 2026
 Merged

1 task
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
2 times, most recently
from
9079e0a to
ba8c945CompareJuly 6, 2026 06:28
Lee-W
Lee-W
reviewed
Jul 6, 2026
View reviewed changes
@Lee-WLee-W
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nits. nothing major
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

Comment threadairflow-core/tests/unit/logging/test_logging_config.py
Outdated

jason810496 added a commit
to jason810496/airflow
that referenced
this pull request
Jul 7, 2026
@jason810496
Clarify user-defined logging config detection and simplify warning tests
65a9d61
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
Lee-W
Lee-W
reviewed
Jul 7, 2026
View reviewed changes
Comment threadairflow-core/src/airflow/logging_config.py
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
jason810496
requested a review
from Lee-WJuly 8, 2026 02:12
@jason810496
Clarify logging_config_class contract and document REMOTE_TASK_LOG
4545abc
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
@jason810496
Fix CI error
c2e8be9
@jason810496
CI: Fix pyproject.toml
741be32
@jason810496
Fix pyproject.toml
55d027b
@jason810496
Drop LOGGING_CONFIG dict from remote logging docs section
daa1c04
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
@jason810496
Clarify user-defined logging config detection and simplify warning tests
d5341a4
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
@jason810496
Default remote_task_log to None and clarify empty logging_config_clas…
9282def
…s handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
@jason810496
jason810496force-pushed
the
refactor/logging/clarify-logging-config-class-core
branch
from
66b52a6 to
9282defCompareJuly 13, 2026 05:54
SZL741023
SZL741023
reviewed
Jul 15, 2026
View reviewed changes
@SZL741023SZL741023
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems necessary to actually obtain the remote task log.
Comment threadairflow-core/src/airflow/logging_config.py
Outdated

@jason810496
Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
4233c6c
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
@vatsrahul1001
vatsrahul1001
merged commit d8b8620
into
apache:mainJul 28, 2026

303 checks passed

@github-actionsgithub-actionsBot
mentioned this pull request
Jul 28, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 28, 2026

Copy link


Copy Markdown

Contributor
Backport successfully created: v3-3-test
Note: As of Merging PRs targeted for Airflow 3.X

the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.
In matter of doubt please ask in #release-management Slack channel.
StatusBranchResult
✅v3-3-testPR Link
github-actionsBot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
5a1a293
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
aws-airflow-bot pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jul 28, 2026
@jason810496Your friendly bot
[v3-3-test] Clarify logging_config_class contract and document ``…
bb9bed6
…REMOTE_TASK_LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
(cherry picked from commit d8b8620)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
jason810496 pushed a commit
that referenced
this pull request
Jul 29, 2026
@github-actions
[v3-3-test] Clarify logging_config_class contract and document ``…
4bc20e6
…REMOTE_TASK_LOG`` (#67104) (#70592)
vatsrahul1001 pushed a commit
that referenced
this pull request
Aug 5, 2026
@github-actions@vatsrahul1001
[v3-3-test] Clarify logging_config_class contract and document ``…
9d271a3
…REMOTE_TASK_LOG`` (#67104) (#70592)
dabla pushed a commit
to dabla/airflow
that referenced
this pull request
Aug 14, 2026
@jason810496@dabla
Clarify logging_config_class contract and document ``REMOTE_TASK_…
8157e60
…LOG`` (apache#67104)
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@Lee-WLee-WLee-W approved these changes
@vatsrahul1001vatsrahul1001vatsrahul1001 approved these changes
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@potiukpotiukAwaiting requested review from potiukpotiuk is a code owner
@kaxilkaxilAwaiting requested review from kaxilkaxil is a code owner
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavangopidesupavan is a code owner
@jscheffljschefflAwaiting requested review from jscheffljscheffl is a code owner
@bugraoz93bugraoz93Awaiting requested review from bugraoz93bugraoz93 is a code owner
@ashbashbAwaiting requested review from ashbashb is a code owner
@phanikumvphanikumvAwaiting requested review from phanikumv

+1 more reviewer

@SZL741023SZL741023SZL741023 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@jason810496jason810496

Labels

allow provider dependency bumparea:ConfigTemplatesarea:loggingarea:task-sdkbackport-to-v3-3-testBackport to v3-3-testkind:documentation

Projects


None yet


Milestone

Airflow 3.3.1

8 participants

@jason810496@amoghrajesh@eladkal@ashb@Lee-W@vatsrahul1001@SZL741023@phanikumv