Add log buffering to increase performance when streaming big logs

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test

[dsuhinin]

The current streaming approach does not seem to effectively solve the problem of handling large logs. This PR adds buffering to improve performance when processing them. Previously, loading 320k entries took around 50–60 seconds; with buffering, it now takes about 5 seconds. This change does not break existing functionality and slightly improves overall performance. I've tested next cases:

• logs with the delay. just around 20 logs per several minutes. Scrolling works well.
• 320k logs without delay. Buffering and scrolling also works well.
• initial load for 320K logs also works pretty well and fast. Around 4-6 seconds which I believe is pretty good. Check screenshot.

[jason810496]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

[potiuk]

@dsuhinin This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ Failing CI checks: CI checks are failing (individual check names not available). Run prek run --from-ref main locally to reproduce. See static checks docs and testing docs.

Note: Your branch is 102 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. If you have questions, feel free to ask on the Airflow Slack.

---

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

[pierrejeambrun]

+1 for Jason idea.

[connorlwilkes]

Testing a similar change internally we have seen a very large performance increase for loading of logs on the front end when viewing logs for a task instance.

I think the front end should really either just use the normal non streaming endpoint or parse the streaming endpoint as a streaming endpoint. My understanding is that it polls the streaming endpoint and gets all the logs every call

[dsuhinin]

I kind of reminder why I didn't add the buffering in last refactoring. There also be the case that we just produce 10 or 20 logs per minute. If I remember correctly, the API server will show the logs until all the logs exceed the threshold, which look not that real-time for users. (Prehaps we could have another coroutine that trigger by time interval to solve the above "blocking" problem)

Maybe worthwhile to test the scenario I mention above again when you have a moment, thanks.

hey! I did few tests and it seems like all good. UI just pooling in that case and overall functionality works just fine.

[dsuhinin]

CI need fixing. I also tested a bunch of scenarios and things were working as expected. Even the case mentioned by Jason

from what I can see I can try to fix only -> https://github.com/apache/airflow/actions/runs/23599259101/job/68727372004?pr=63180

everything else just failing by some reason fully randomly. one update before it were 2 failed steps, now more - 5.

[jason810496]

everything else just failing by some reason fully randomly. one update before it were 2 failed step, now 5.

It seems like regression of the following integration tests, the other PR also suffer from the same failure.

providers/elasticsearch/tests/integration/elasticsearch/log/test_es_remote_log_io.py::TestElasticsearchRemoteLogIOIntegration

I will look into it if it haven't been resolved.
UPDATE: The ElasticSearch e2e tests had been resolved in #64276, I just rebased this one on top of it.

[jason810496]

Should be good to merge on my end, the UI e2e tests already been failed for a while on main IIRC.

[connorlwilkes]

Is it possible to have the buffer size configurable?

[xBis7]

I see that the issue is tagged for the 3.2 release but I don't know the exact time that we have for getting it merged.

Because people have different use-cases, it would be helpful if the buffer size was configurable. We have found that the performance improvement increases along side the buffer size and the log size. But people with small number of logs won't need a huge buffer size.

Also, a unit test would be helpful to understand how it's supposed to work without having to deep dive into the code like I did.

If time is pressing, we can address the above in a followup PR.

[dsuhinin]

Is it possible to have the buffer size configurable?

of course. thew new variable has been added to configure the buffer size.

[xBis7]

Thanks for adding the config property. LGTM!

[github-actions]

Backport successfully created: v3-2-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status	Branch	Result
✅	v3-2-test