Uh oh!
There was an error while loading. Please reload this page.
Fix Connection or Variable access in Server context - #56602
Merged
Conversation
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixesapache#56120Fixesapache#56583
kaxilforce-pushed
the
fix-conn-var-retrieval
branch
3 times, most recently
from
October 14, 2025 02:06
8d21aaf to
0cfdaa5Comparekaxil
marked this pull request as ready for review
October 14, 2025 02:06
kaxil
requested review from
XD-DENG, amoghrajesh, ashb, bugraoz93, dstandish, ephraimbuddy, jason810496, pierrejeambrun, potiuk, rawwar and shubhamraj-git
as code ownersOctober 14, 2025 02:06
2 tasks
kaxilforce-pushed
the
fix-conn-var-retrieval
branch
from
October 14, 2025 03:27
0cfdaa5 to
ade61f4CompareUh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
ashb
reviewed
Oct 14, 2025
ashb
left a comment
Member
There was a problem hiding this comment.
Messier then i'd like but i can't think of another option right now
Uh oh!
There was an error while loading. Please reload this page.
ashb
approved these changes
Oct 14, 2025
gopidesupavan
commented
Oct 14, 2025
Member
Uh oh!
There was an error while loading. Please reload this page.
kaxil
commented
Oct 14, 2025
MemberAuthor
Thanks @gopidesupavan , Please do let us know what you find! |
gopidesupavan
commented
Oct 14, 2025
Member
just in time: works fine with this changes :)   |
kaxil added a commit
that referenced
this pull request
Oct 14, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixes#56120Fixes#56583 (cherry picked from commit ae0a330)
2 tasks
TyrellHaywood pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with `ImportError` for `SUPERVISOR_COMMS` because it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval. Implemented automatic context detection using separate secrets backend chains for client and server processes: - Client contexts (workers, DAG processors, triggerers) are detected via `SUPERVISOR_COMMS` presence and use `ExecutionAPISecretsBackend` to route through the Execution API - Server contexts (API server, scheduler, plugins) are detected when `SUPERVISOR_COMMS` is unavailable and use `MetastoreBackend` for direct database access Fixesapache#56120Fixesapache#56583
kaxil added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568Fixesapache#57145
kaxil added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568Fixesapache#57145
kaxil added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568Fixesapache#57145
kaxil added a commit
to astronomer/airflow
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache#55799 for 3.1.0 but apache#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache#55568Fixesapache#57145
kaxil added a commit
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568Fixes#57145
kaxil added a commit
that referenced
this pull request
Oct 23, 2025
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in #55799 for 3.1.0 but #56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: #55568Fixes#57145 (cherry picked from commit da32b68)
2 tasks
kosteev pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Mar 2, 2026
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache/airflow#55799 for 3.1.0 but apache/airflow#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache/airflow#55568Fixesapache/airflow#57145 (cherry picked from commit da32b682d1b0df5d5e2078392cf8626f8fdb00ff) GitOrigin-RevId: f969e6374daa8469938169be16a28f7c073a5ce9
kosteev pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Jul 31, 2026
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache/airflow#55799 for 3.1.0 but apache/airflow#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache/airflow#55568Fixesapache/airflow#57145 GitOrigin-RevId: da32b682d1b0df5d5e2078392cf8626f8fdb00ff
kosteev pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Aug 2, 2026
When deferrable operators run in the triggerer's async event loop and synchronously access connections (e.g., via @cached_property), the `ExecutionAPISecretsBackend` failed silently. This occurred because `SUPERVISOR_COMMS.send()` uses `async_to_sync`, which raises `RuntimeError` when called within an existing event loop in a greenback portal context. Add specific RuntimeError handling in `ExecutionAPISecretsBackend` that detects this scenario and uses `greenback.await_()` to call the async versions (aget_connection/aget_variable) as a fallback. It was originally fixed in apache/airflow#55799 for 3.1.0 but apache/airflow#56602 introduced a bug. Ideally all providers handle this better and have better written Triggers. Example PR for Databricks: apache/airflow#55568Fixesapache/airflow#57145 (cherry picked from commit da32b682d1b0df5d5e2078392cf8626f8fdb00ff) GitOrigin-RevId: f969e6374daa8469938169be16a28f7c073a5ce9
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hooks used in API server contexts (plugins, middlewares, log handlers) previously failed with
ImportErrorforSUPERVISOR_COMMSbecause it only exists in worker Task execution contexts (Worker, Dag processor, Trigger). This prevented using hooks like GCSHook or S3Hook in plugins and broke log retrieval.Implemented automatic context detection using separate secrets backend chains for client and server processes:
SUPERVISOR_COMMSpresence and useExecutionAPISecretsBackendto route through the Execution APISUPERVISOR_COMMSis unavailable and useMetastoreBackendfor direct database accessFixes#56120
Fixes#56583
PS: I do not love the way we deal with secrets backends in airflow/configuration, even before this PR -- but not going to handle it in this PR.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.