Skip to content

Add e2e test for remote logging - #56191

Merged
gopidesupavan merged 10 commits into
apache:mainfrom
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged

Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom
gopidesupavan:fix-connection-handling

Conversation

@gopidesupavan

@gopidesupavangopidesupavan commented Sep 28, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Adding Remote logging tests to E2E test.

Why:

Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.

This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@gopidesupavan

gopidesupavan commented Sep 28, 2025

Copy link
Copy Markdown
MemberAuthor

Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔

ashb
ashb reviewed Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py Outdated
@ashb

ashb commented Sep 29, 2025

Copy link
Copy Markdown
Member

Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔

Neither, it's integration between the two

@gopidesupavan

gopidesupavan commented Sep 29, 2025

Copy link
Copy Markdown
MemberAuthor

Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔

Neither, it's integration between the two

hehe correct :)

@ashbashb changed the title Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor
image

@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor

have added a full e2e integration test with local stack,

@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor

I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .

github-advanced-security[bot]
github-advanced-securityAI found potential problems Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py Fixed
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py Fixed
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py Fixed
@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch 2 times, most recently from a698cd0 to 2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor

verified on real env:

image

@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch 4 times, most recently from fadc410 to 89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor

I think this is ready for review 2 tests are failing not related to this, am trying to look them.

@gopidesupavan

gopidesupavan commented Sep 30, 2025

Copy link
Copy Markdown
MemberAuthor

raised PR here for failure tests #56270

@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch from 2b30a66 to 4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk reviewed Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml Outdated
@Tni04Tni04 mentioned this pull request Oct 4, 2025
Closed
2 tasks
@kaxil

kaxil commented Oct 13, 2025•
edited
Loading

Copy link
Copy Markdown
Member

Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)

@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch 2 times, most recently from 785544a to 280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan mentioned this pull request Oct 14, 2025
Merged
@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch 2 times, most recently from 092dff0 to d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496 self-requested a review October 14, 2025 13:43
@gopidesupavangopidesupavan changed the title Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan marked this pull request as draft October 14, 2025 20:24
@gopidesupavan

gopidesupavan commented Oct 14, 2025

Copy link
Copy Markdown
MemberAuthor

Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602

@kaxil

kaxil commented Oct 14, 2025

Copy link
Copy Markdown
Member

Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602

Thanks @gopidesupavan

@gopidesupavan
gopidesupavanforce-pushed the fix-connection-handling branch 2 times, most recently from a202686 to cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan marked this pull request as ready for review October 15, 2025 06:22
potiuk
potiuk approved these changes Oct 15, 2025
View reviewed changes

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)

@potiuk
potiukforce-pushed the fix-connection-handling branch from cb5c9dc to 92f5172CompareOctober 15, 2025 12:56
@potiuk

potiuk commented Oct 15, 2025

Copy link
Copy Markdown
Member

Rebased to account for fixed task-sdk integration tests (just in case).

@gopidesupavan

gopidesupavan commented Oct 15, 2025

Copy link
Copy Markdown
MemberAuthor

Nice. I like the composed docker-compose file :)

yeah :)

@gopidesupavan
gopidesupavan merged commit 66d5e72 into apache:mainOct 15, 2025
153 of 154 checks passed
@gopidesupavan
gopidesupavan deleted the fix-connection-handling branch October 15, 2025 18:10
@kaxil

kaxil commented Oct 15, 2025

Copy link
Copy Markdown
Member

Awesome, thanks @gopidesupavan

snreddygopu pushed a commit to Teradata/airflow that referenced this pull request Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496 mentioned this pull request Oct 18, 2025
Merged
TyrellHaywood pushed a commit to TyrellHaywood/airflow that referenced this pull request Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
Add e2e test for remote logging by gopidesupavan · Pull Request #56191 · apache/airflow · GitHub
Skip to content
Add e2e test for remote logging - #56191
Merged
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Oct 15, 2025
Merged
Add e2e test for remote logging#56191
gopidesupavan merged 10 commits into
apache:mainfrom 
gopidesupavan:fix-connection-handling
Conversation
@gopidesupavan
@gopidesupavangopidesupavan
commented
Sep 28, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Adding Remote logging tests to E2E test.
Why:
Currently we dont have a way to test remote logging, all that we are doing so far mocking handlers and we have seen recent times remote logging broken many times.
This is to add remote logging e2e test that replicates real environment with AWS using localstack. This test is remote logging with S3. for other remote logging it can be extended later.
^ Add meaningful description above

Read the Pull Request Guidelines for more information.

In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.

In case of a new dependency, check compliance with the ASF 3rd Party License Policy.

In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.
@gopidesupavan
gopidesupavan
commented
Sep 28, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
ashb
ashb
reviewed
Sep 29, 2025
View reviewed changes
Comment threadtask-sdk/src/airflow/sdk/execution_time/context.py
Outdated

@ashb
ashb
commented
Sep 29, 2025

Copy link


Copy Markdown

Member
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
@gopidesupavan
gopidesupavan
commented
Sep 29, 2025

Copy link


Copy Markdown

MemberAuthor
Tests needs to be added, not sure its tricky part where to add is it in airflow-core or task-sdk 🤔
Neither, it's integration between the two
hehe correct :)
@ashbashb
changed the title
Use in-process-api-server for connection when task sdk context not availableFix loading Connections in API server in order to read remote logsSep 29, 2025
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
image
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
have added a full e2e integration test with local stack,
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I am not sure if you wanted me to create separate PR for testing update i am happy to do that, i have added it part of this because we are trying to fix the remote log issue, i feel its make sense to add .
github-advanced-security[bot]
github-advanced-securityAI
found potential problems
Sep 30, 2025
View reviewed changes
Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

Comment threadairflow-core/src/airflow/utils/log/connection_manager.py
Fixed

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a698cd0 to
2222c11CompareSeptember 30, 2025 00:46
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
verified on real env:
image
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
4 times, most recently
from
fadc410 to
89570d0CompareSeptember 30, 2025 11:30
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
I think this is ready for review 2 tests are failing not related to this, am trying to look them.
@gopidesupavan
gopidesupavan
commented
Sep 30, 2025

Copy link


Copy Markdown

MemberAuthor
raised PR here for failure tests #56270
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
from
2b30a66 to
4558e68CompareOctober 3, 2025 09:32
potiuk
potiuk
reviewed
Oct 3, 2025
View reviewed changes
Comment threadairflow-e2e-tests/docker/docker-compose-with-localstack.yaml
Outdated

@Tni04Tni04
mentioned this pull request
Oct 4, 2025

Closed

2 tasks
@kaxil
kaxil
commented
Oct 13, 2025•

edited

Loading

Copy link


Copy Markdown

Member
Bump on this one @gopidesupavan :) If you can address the comments, we can release 3.1.1 sooner with it :)
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
785544a to
280a597CompareOctober 14, 2025 08:53
@gopidesupavangopidesupavan
mentioned this pull request
Oct 14, 2025
 Merged

@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
092dff0 to
d0efa42CompareOctober 14, 2025 12:43
@jason810496
jason810496
self-requested a review
October 14, 2025 13:43
@gopidesupavangopidesupavan
changed the title
Fix loading Connections in API server in order to read remote logsAdd e2e test for remote loggingOct 14, 2025
@gopidesupavan
gopidesupavan
marked this pull request as draft
October 14, 2025 20:24
@gopidesupavan
gopidesupavan
commented
Oct 14, 2025

Copy link


Copy Markdown

MemberAuthor
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
@kaxil
kaxil
commented
Oct 14, 2025

Copy link


Copy Markdown

Member
Making changes to this pr only to add e2e tests for remote logging, as the issue is fixed by @kaxil in #56602
Thanks @gopidesupavan
@gopidesupavan
gopidesupavanforce-pushed
the
fix-connection-handling
branch
2 times, most recently
from
a202686 to
cb5c9dcCompareOctober 14, 2025 21:01
@gopidesupavan
gopidesupavan
marked this pull request as ready for review
October 15, 2025 06:22
potiuk
potiuk
approved these changes
Oct 15, 2025
View reviewed changes
@potiukpotiuk
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. I like the composed docker-compose file :)
@potiuk
potiukforce-pushed
the
fix-connection-handling
branch
from
cb5c9dc to
92f5172CompareOctober 15, 2025 12:56
@potiuk
potiuk
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Rebased to account for fixed task-sdk integration tests (just in case).
@gopidesupavan
gopidesupavan
commented
Oct 15, 2025

Copy link


Copy Markdown

MemberAuthor
Nice. I like the composed docker-compose file :)
yeah :)
@gopidesupavan
gopidesupavan
merged commit 66d5e72
into
apache:mainOct 15, 2025

153 of 154 checks passed

@gopidesupavan
gopidesupavan
deleted the

fix-connection-handling

branch
October 15, 2025 18:10
@kaxil
kaxil
commented
Oct 15, 2025

Copy link


Copy Markdown

Member
Awesome, thanks @gopidesupavan
snreddygopu pushed a commit
to Teradata/airflow
that referenced
this pull request
Oct 16, 2025
@gopidesupavan@sc250072
Add e2e test for remote logging (apache#56191)
f1ae943
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
@jason810496jason810496
mentioned this pull request
Oct 18, 2025
 Merged

TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
@gopidesupavan@TyrellHaywood
Add e2e test for remote logging (apache#56191)
804f93c
* Prefetch remote log connection id for api server in order to read remote logs
* fix docker compose file path
* Fixup tests
* Add test with mock_aws
* Fixup test
* Extend quick start docker with localstack
* remove comment
* add test connection
* fix static checks
* Add only e2e tests for remote logging
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@ashbashbashb left review comments
@potiukpotiukpotiuk approved these changes
@kaxilkaxilAwaiting requested review from kaxil
@amoghrajeshamoghrajeshAwaiting requested review from amoghrajeshamoghrajesh is a code owner
@jedcunninghamjedcunninghamAwaiting requested review from jedcunninghamjedcunningham is a code owner
@jason810496jason810496Awaiting requested review from jason810496jason810496 is a code owner

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@gopidesupavan@ashb@kaxil@potiuk@github-advanced-security