feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

feat(core): make elasticsearch compatible with remote_task_log - #62121

Merged
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io
Mar 5, 2026
Merged

feat(core): make elasticsearch compatible with remote_task_log#62121
eladkal merged 2 commits into
apache:mainfrom
Owen-CH-Leung:core/make-es-compatible-with-remote-log-io

Conversation

@Owen-CH-Leung

@Owen-CH-LeungOwen-CH-Leung commented Feb 18, 2026

Copy link
Copy Markdown
Contributor

fixes: #50349
Fixes: #51456

This is the airflow-core counterpart to #53821, which together restore the write_to_es Elasticsearch logging feature broken in Airflow 3.

airflow_local_settings.py previously configured Elasticsearch logging by injecting ElasticsearchTaskHandler directly into the logging handler dict. In Airflow 3, remote log writing goes through a REMOTE_TASK_LOG object implementing the RemoteLogIO interface. This PR updates the config template to set REMOTE_TASK_LOG = ElasticsearchRemoteLogIO(...)

The minimum required version of apache-airflow-providers-elasticsearch is also bumped to >=6.5.0 (the release containing ElasticsearchRemoteLogIO) in the dependency constraints.

@jhgoebbert

Copy link
Copy Markdown

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

@eladkaleladkal added this to the Airflow 3.1.9 milestone Mar 5, 2026
@potiuk

Copy link
Copy Markdown
Member

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 1780ead to 53b2dacCompareMarch 5, 2026 10:27
@potiuk
potiuk marked this pull request as ready for review March 5, 2026 10:27
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 53b2dac to ba9451cCompareMarch 5, 2026 10:30
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from ba9451c to 474d8daCompareMarch 5, 2026 10:31
@eladkal

Copy link
Copy Markdown
Contributor

I have successfully tested it with Airflow 3.2.0b1 on k8s - it works like a charm. 👍

Great to hear! We are going to include this in 3.1.8 so users can migrate earlier


@Owen-CH-Leung great work on this task!

@Owen-CH-Leung

Copy link
Copy Markdown
ContributorAuthor

Hey @Owen-CH-Leung -> I know you might not be available/have time but we need that one in, I will take it over from here and merge as we want it for 3.1.8. This is fantastic job from you !

No problem :) Thanks a lot !

@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch 2 times, most recently from f84bee7 to 9ab4aa5CompareMarch 5, 2026 12:58
@potiuk
potiukforce-pushed the core/make-es-compatible-with-remote-log-io branch from 9ab4aa5 to e7968feCompareMarch 5, 2026 13:06
@eladkal
eladkal merged commit 4f1faf6 into apache:mainMar 5, 2026
237 of 238 checks passed
github-actionsBot pushed a commit that referenced this pull request Mar 5, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-1-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-1-testPR Link

eladkal pushed a commit that referenced this pull request Mar 5, 2026
… (#62940)
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
(cherry picked from commit 4f1faf6)
Co-authored-by: Owen Leung <owen.leung2@gmail.com>
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
dominikhei pushed a commit to dominikhei/airflow that referenced this pull request Mar 11, 2026
* Make elasticsearch compatible with remote_task_log and add min version
* Fix pyproject.toml to use elasticsearch>=6.5.0
---------
Co-authored-by: vatsrahul1001 <rah.sharma11@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

airflow no writing logs to Elasticsearch Logs are not integrating with elasticsearch

5 participants

@Owen-CH-Leung@jhgoebbert@potiuk@eladkal@vatsrahul1001