Add async connection testing via workers for security isolation

[anishgirianish]

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Summary

Follows the direction proposed by @potiuk in #59643 to move connection testing off the API server and onto workers.

Connection testing has been disabled by default since Airflow 2.7.0 because executing user-supplied driver code (ODBC/JDBC) on the API server poses security risks, and workers typically have network access to external systems that API servers don't.

This moves the whole thing onto workers. A dedicated TestConnection workload goes through the scheduler, gets dispatched to a supporting executor, and the worker runs test_connection()` with a proper timeout. Results come back through the Execution API. Design was discussed on dev@ : "[DISCUSS] Move connection testing to workers" (Feb 2026).

Demo

breeze-e2e-rundown-compressed.mp4

Overview

• Dedicated workload type : not piggybacking on ExecuteCallback, so connection tests never compete with correctness-critical callbacks
• Scheduler dispatch + reaper: PENDING tests get dispatched to a supporting executor, capped by max_connection_test_concurrency (default 4). A reaper catches stuck tests after timeout + grace period
• Worker-side timeout : signal.alarm enforcement in LocalExecutor, results reported back via Execution API
• Request Buffer Mechanism: <TODO:add details>
• Queue parameter: optional queue field on the API, wired through to scheduler dispatch.
• Fail-fast: supports_connection_test flag on BaseExecutor, immediate FAILED if no executor supports it

Config

• [connection_test] timeout: worker timeout, default 60s
• [connection_test] max_concurrency: dispatch budget, default 4
• [connection_test] reaper_interval: reaper frequency, default 30s

Not in this PR

• UI changes (will create separate pr for this)

References

• Dev mailing list discussion: DISCUSS Move connection testing to workers

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jason810496]

Nice! LGTM overall.

[anishgirianish]

@jason810496 Thanks for the thorough review! Addressed your feedback in the latest push:

• Removed result_status column — state is sufficient
• Moved _ImportPathCallbackDef to connection_test.py with a create_callback() factory method

Could you please take another look when you get a chance? Thanks!

[kaxil]

Approving so this can move, but a few things to follow up on:

1. Newsfragment missing. New config option, new lifecycle, security-model changes -- this needs airflow-core/newsfragments/62343.feature.rst (or .significant.rst). Best done before merge.

2. User docs. Only migrations-ref.rst is touched. The POST + poll lifecycle, the [core] connection_test_max_concurrency knob, the worker token + ct:self scope model, and multi-team behavior all need a narrative page under airflow-core/docs/. Follow-up is fine.

3. Logging + metrics. Reaper warnings drop team / state / connection_id context. Supervisor doesn't log start, duration, or timeout-driven cleanup, and there's no bind_contextvars so concurrent tests in one worker can't be grepped apart. No Stats.timing / Stats.gauge for queue depth, dispatch latency, success/failure rate, or hook execution time. Operators will have nothing to look at when this misbehaves.

4. ct:self scope is untested. mock_require_auth in airflow-core/tests/unit/api_fastapi/execution_api/conftest.py reads task_instance_id from path params, but the new connection-test routes use connection_test_id. So the mock always returns TIToken(id="00000000-...") and the ct:self check in security.py (str(token.id) != ct_self_id -> 403) never executes in tests. Same shape as the ti:self gap on #66073. Fix: either extend the mock to also read connection_test_id, or add one test that bypasses dependency_overrides[require_auth] and asserts 403 on a token/path mismatch.

Items 2-4 can land as follow-ups. Item 1 ideally lands here.

[Vamsi-klu]

I found two blockers in the current head:

1. The worker success path drops team ownership when commit_on_success=True creates a brand-new connection. The public API resolves and authorizes the effective team, then stores it on ConnectionTestRequest, but commit_to_connection_table() creates the new Connection(...) without team_name=self.team_name. In multi-team mode that can turn a team-scoped connection credential into a persisted global connection. Please copy the team on creation and add a regression test for a new team-owned connection with commit_on_success=True.

2. The UI still appears to exercise the old synchronous API-server test path, so the main UI workflow does not get the worker-isolation behavior added by this PR. useTestConnection.ts still calls the generated client for /api/v2/connections/test, and that route still executes conn.test_connection() inside the API server process. If the intent is for UI connection tests to be isolated from the API server, the UI should enqueue the test and poll by token, or the old API-server route should be separately gated/deprecated so enabling connection testing does not keep exposing the synchronous API-server execution path.

---

Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting

[anishgirianish]

I found two blockers in the current head:

1. The worker success path drops team ownership when commit_on_success=True creates a brand-new connection. The public API resolves and authorizes the effective team, then stores it on ConnectionTestRequest, but commit_to_connection_table() creates the new Connection(...) without team_name=self.team_name. In multi-team mode that can turn a team-scoped connection credential into a persisted global connection. Please copy the team on creation and add a regression test for a new team-owned connection with commit_on_success=True.
2. The UI still appears to exercise the old synchronous API-server test path, so the main UI workflow does not get the worker-isolation behavior added by this PR. useTestConnection.ts still calls the generated client for /api/v2/connections/test, and that route still executes conn.test_connection() inside the API server process. If the intent is for UI connection tests to be isolated from the API server, the UI should enqueue the test and poll by token, or the old API-server route should be separately gated/deprecated so enabling connection testing does not keep exposing the synchronous API-server execution path.

Drafted-by: Codex (GPT-5); reviewed by @Vamsi-klu before posting

Hi @Vamsi-klu thank you so much for your review.

1. Addressed thanks

2. UI changes are not part of this pr and will come along in a follow-up as stated in Pr description.

I think this clears the blockers. Thank you

[ferruzzi]

Looks like all concerns are addressed. Approving and merging. Thanks for sticking this one out, it was a major redesign.