Skip to content

Fix the write-to-es feature - #53821

Merged
eladkal merged 73 commits into
apache:mainfrom
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged

Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom
Owen-CH-Leung:fix-write-to-es-feature

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Jul 28, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.

This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.

The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.

@Owen-CH-Leung
Owen-CH-Leung marked this pull request as ready for review July 29, 2025 12:48
jason810496
jason810496 reviewed Jul 30, 2025
View reviewed changes

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!

Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.
LGTM overall.

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py Outdated
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb requested changes Jul 30, 2025
View reviewed changes

@ashbashb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py Outdated
ashb
ashb reviewed Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py Outdated
@Owen-CH-Leung

Owen-CH-Leung commented Aug 7, 2025

Copy link
Copy Markdown
ContributorAuthor

@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks

potiuk
potiuk reviewed Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml Outdated
jason810496
jason810496 reviewed Aug 7, 2025
View reviewed changes

@jason810496jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!
It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!

Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py Outdated
Comment threadproviders/elasticsearch/tests/conftest.py Outdated
@Owen-CH-Leung

Owen-CH-Leung commented Aug 8, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!

I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase

No worries I think I found it. Let me expand the testing logic

@jason810496

jason810496 commented Aug 10, 2025•
edited
Loading

Copy link
Copy Markdown
Member

I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase

No worries I think I found it. Let me expand the testing logic

Sorry for the late reply.

The following command will be helpful to setup Airflow with ElasticSearch in Breeze.

breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>

It would be great to test with the following version matrix for testing compatibility.
List of the <version> to be replaced with:

  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )
    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.

Thanks a lot!

@jason810496jason810496 mentioned this pull request Aug 10, 2025
Merged
@Owen-CH-Leung
Owen-CH-Leung marked this pull request as draft February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung mentioned this pull request Feb 18, 2026
Merged
@Owen-CH-Leung
Owen-CH-Leung marked this pull request as ready for review February 19, 2026 15:51
eladkal
eladkal approved these changes Feb 20, 2026
View reviewed changes

@eladkaleladkal left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Great work @Owen-CH-Leung

@eladkal
eladkal dismissed ashb’s stale reviewFebruary 20, 2026 08:29

We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after

@eladkal
eladkal merged commit aeec681 into apache:mainFeb 20, 2026
131 checks passed
@eladkaleladkal mentioned this pull request Feb 20, 2026
Closed
6 tasks
@eladkal

eladkal commented Feb 20, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219

choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung mentioned this pull request Feb 23, 2026
Closed
2 tasks
@potiukpotiuk mentioned this pull request Feb 26, 2026
Closed
@Owen-CH-LeungOwen-CH-Leung mentioned this pull request Feb 27, 2026
Merged
2 tasks
@potiukpotiuk mentioned this pull request Mar 3, 2026
Closed
55 tasks
@jhgoebbert

jhgoebbert commented Mar 3, 2026•
edited
Loading

Copy link
Copy Markdown

Thank you @Owen-CH-Leung
I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!

Subham-KRLX pushed a commit to Subham-KRLX/airflow that referenced this pull request Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit to Ankurdeewan/airflow that referenced this pull request Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496 mentioned this pull request Jul 7, 2026
Closed
1 task
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93
+1 more reviewer
@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Fix the write-to-es feature by Owen-CH-Leung · Pull Request #53821 · apache/airflow · GitHub
Skip to content
Fix the write-to-es feature - #53821
Merged
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Feb 20, 2026
Merged
Fix the write-to-es feature#53821
eladkal merged 73 commits into
apache:mainfrom 
Owen-CH-Leung:fix-write-to-es-feature
Conversation
@Owen-CH-Leung
@Owen-CH-LeungOwen-CH-Leung
commented
Jul 28, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
fixes: #50349
Fixes: #51456
The write_to_es feature of ElasticsearchTaskHandler is broken in Airflow 3 because the remote logging write path was redesigned. In Airflow 2, task log handlers wrote directly to the remote destination. In Airflow 3, this responsibility has moved to a dedicated RemoteLogIO class.
This PR fixes the issue on the provider side by introducing ElasticsearchRemoteLogIO, a new class implementing the RemoteLogIO interface. It handles both writing task logs to Elasticsearch (when write_to_es = True) and reading them back. ElasticsearchTaskHandler is refactored to delegate remote read/write operations to ElasticsearchRemoteLogIO while retaining backward compatibility with existing config options.
The companion PR #62121 makes the corresponding change in airflow-core to wire ElasticsearchRemoteLogIO into the logging config.
@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
July 29, 2025 12:48
jason810496
jason810496
reviewed
Jul 30, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Big thanks for helping out!
Only some backward compact need to be fixed and the unit test for new ElasticsearchRemoteLogIO is required.

LGTM overall.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
ashb
ashb
requested changes
Jul 30, 2025
View reviewed changes
@ashbashb
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @potiuk about a recent Slack discussion we had, this is a prime example of why I'm sad we have separate providers for OpenSearch and ElasticSearch. What we have to do in one we should really do in both.
Comment threadproviders/elasticsearch/src/airflow/providers/elasticsearch/log/es_task_handler.py
Outdated

ashb
ashb
reviewed
Jul 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/config_templates/airflow_local_settings.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 7, 2025

Copy link


Copy Markdown

ContributorAuthor
@jason810496@ashb I added unit test using testcontainer so that remote_read and write can be tested against a real elasticsearch instance. Could you review again ? Thanks
potiuk
potiuk
reviewed
Aug 7, 2025
View reviewed changes
Comment threadproviders/elasticsearch/pyproject.toml
Outdated

jason810496
jason810496
reviewed
Aug 7, 2025
View reviewed changes
@jason810496jason810496
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Thanks for the update!

It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
Comment threadproviders/elasticsearch/tests/unit/elasticsearch/log/test_es_task_handler.py
Outdated

Comment threadproviders/elasticsearch/tests/conftest.py
Outdated

@Owen-CH-Leung
Owen-CH-Leung
commented
Aug 8, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Nice! Thanks for the update! It would be great to verify whether the TaskInstanceLog UI page displays correctly with this change. If so, this PR should be good to go. Thank you!
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
@jason810496
jason810496
commented
Aug 10, 2025•

edited

Loading

Copy link


Copy Markdown

Member
I'd love to verify that as well. Do you know which relevant file / class I should look at ? Not very familiar with the frontend codebase
No worries I think I found it. Let me expand the testing logic
Sorry for the late reply.
The following command will be helpful to setup Airflow with ElasticSearch in Breeze.
breeze start-airflow --python 3.10 --backend postgres --integration elasticsearch --mount-sources providers-and-tests --use-airflow-version <version>
It would be great to test with the following version matrix for testing compatibility.

List of the <version> to be replaced with:
  • 3.0.3
  • 3.0.4
  • main ( for unreleased 3.1.x version, but this one is a bit more complicated )

    1. Switch to main
    2. Run rm -rf dist/*
    3. Run breeze release-management prepare-airflow-distributions
    4. Run breeze release-management prepare-task-sdk-distributions
    5. Switch back to branch of this PR ( fix-write-to-es-feature in this case )
    6. Run breeze start-airflow --use-airflow-version wheel --mount-sources providers-and-tests --integration elasticsearch instead of command above.
Thanks a lot!
@jason810496jason810496
mentioned this pull request
Aug 10, 2025
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as draft
February 17, 2026 11:33
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 18, 2026
 Merged

@Owen-CH-Leung
Owen-CH-Leung
marked this pull request as ready for review
February 19, 2026 15:51
eladkal
eladkal
approved these changes
Feb 20, 2026
View reviewed changes
@eladkaleladkal
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
Great work @Owen-CH-Leung
@eladkal
eladkal
dismissed
ashb’s stale reviewFebruary 20, 2026 08:29
We will evaluate the strategy about elastic and opensearch providers. Since this PR is a bug-fix I'd like to release it as soon as we can and handle the refactor after
@eladkal
eladkal
merged commit aeec681
into
apache:mainFeb 20, 2026

131 checks passed

@eladkaleladkal
mentioned this pull request
Feb 20, 2026

Closed

6 tasks
@eladkal
eladkal
commented
Feb 20, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Kindly asking from all users who waited/asked this fix to test it and report if the issue is resolved in
https://redirect.github.com/apache/airflow/issues/62219
choo121600 pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
@Owen-CH-Leung@jason810496@choo121600
fix the write-to-es feature for Airflow 3 (apache#53821)
850bd00
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 23, 2026

Closed

2 tasks
@potiukpotiuk
mentioned this pull request
Feb 26, 2026

Closed

@Owen-CH-LeungOwen-CH-Leung
mentioned this pull request
Feb 27, 2026
 Merged

2 tasks
@potiukpotiuk
mentioned this pull request
Mar 3, 2026

Closed

55 tasks
@jhgoebbert
jhgoebbert
commented
Mar 3, 2026•

edited

Loading

Copy link


Copy Markdown

Thank you @Owen-CH-Leung

I have tested the new feature with the CeleryExecutor and KubernetesExecutor in combination with #62121 and it works great! Fantastic!!
Subham-KRLX pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
@Owen-CH-Leung@jason810496@Subham-KRLX
fix the write-to-es feature for Airflow 3 (apache#53821)
50f5025
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dominikhei pushed a commit
to dominikhei/airflow
that referenced
this pull request
Mar 11, 2026
@Owen-CH-Leung@jason810496@dominikhei
fix the write-to-es feature for Airflow 3 (apache#53821)
33a8cec
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
Ankurdeewan pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
@Owen-CH-Leung@jason810496@Ankurdeewan
fix the write-to-es feature for Airflow 3 (apache#53821)
11ec23a
* Introduce ElasticsearchRemoteLogIO
* Fix ruff check on testEsTaskhandler
* Add ElasticsearchRemoteIO into TaskHandler. Refactor to handle read/write on RemoteIO class
---------
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
@jason810496jason810496
mentioned this pull request
Jul 7, 2026
 Closed

1 task
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@jason810496jason810496jason810496 approved these changes
@eladkaleladkaleladkal approved these changes
@dstandishdstandishAwaiting requested review from dstandish
@potiukpotiukAwaiting requested review from potiuk
@gopidesupavangopidesupavanAwaiting requested review from gopidesupavan
@ashbashbAwaiting requested review from ashb
@jedcunninghamjedcunninghamAwaiting requested review from jedcunningham
@hussein-awalahussein-awalaAwaiting requested review from hussein-awala
@jscheffljschefflAwaiting requested review from jscheffl
@bugraoz93bugraoz93Awaiting requested review from bugraoz93

+1 more reviewer

@topherinternationaltopherinternationaltopherinternational approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

area:ConfigTemplatesarea:loggingarea:providersprovider:elasticsearch

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch

Logs are not integrating with elasticsearch


13 participants

@Owen-CH-Leung@jason810496@Pad71@potiuk@jhgoebbert@eladkal@topherinternational@Sozik87@mvolodarets-kt@kudodenko@m-filippov@ashb@jscheffl