Allow Variable.get to reuse the caller's database session

[ferruzzi]

MetastoreBackend.get_variable is decorated with @provide_session. Called without a session, it goes through create_session(), which returns the caller's own session for a scoped session and commits it on exit. So any code that reads a Variable while holding a transaction gets that transaction committed underneath it. That surfaces two ways:

1. Outside the scheduler guard: the commit succeeds early and the caller's ORM objects are detached silently.
2. Under the scheduler's prohibit_commit guard: it raises RuntimeError: UNEXPECTED COMMIT, which is then swallowed per-backend and surfaces as a missing Variable.

Variable.get, Variable.get_variable_from_secrets, and Variable.setdefault now take an optional keyword-only session, forwarded only to MetastoreBackend. Variable.update forwards its own.

Affected today:

• Deadline Alerts using VariableInterval return "Variable not found" despite the Variable existing
• Custom timetables reading a Variable in next_dagrun_info leave next_dagrun NULL, so the Dag is never eligible and never runs, with nothing logged

Both of the above are reached from _create_dagruns_for_dags, inside the guard. Dag sync via update_dags, Variable.update (fixed here), and Variable.setdefault (fixed here) hold a session without the guard, so they fail the silent way instead.

There are likely others; these are the ones I found.

Scope: This fixes the core Variable.update and Variable.setdefault outright and lets the remaining callers be fixed by passing session. airflow.sdk.Variable.get and Connection.get_connection_from_secrets share the same issue and are left as follow-ups, so callers going through those are still affected.

Related (that I know of)

• Resolve VariableInterval deadlines safely at DagRun creation #68917 can drop its duplicated backend walk once this lands
• Add SerializedVariableInterval for deadline alerts #71802 needs to pass session through its Variable.get() call

closes: #71801

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

Generated-by: Kiro (Claude Opus 5) following the guidelines

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[ferruzzi]

Left a blocking comment to prevent accidental merge. PR #71802 will present a merge conflict and it will be easier to resolve if that one merges before this one. Also moving this to draft just in case, but please leave that comment unresolved as an enforced "do not merge" blocker.

[ferruzzi]

On hold until #71802 merges. That PR adds SerializedVariableInterval.resolve(), which reads a Variable while the scheduler is already holding a session, so it becomes the first real caller for
this parameter. Once it lands I'll add session to resolve() and pass it through to Variable.get() here, rather than merging a parameter with no consumers.