Allow Variable.get_variable_from_secrets to reuse session

[saitejabandaru-in]

Closes#71801.

Motivation

When resolving VariableInterval deadlines (such as in _process_dagrun_deadline_alerts), the scheduler operates under prohibit_commit. However, querying MetastoreBackend.get_variable through Variable.get_variable_from_secrets does not pass down the session, which triggers @provide_session to create a scoped session that automatically commits upon exiting, thus tripping UNEXPECTED COMMIT guards.

Because of this, callers had to duplicate the secrets backend iteration loop directly in dag.py (see #68917) instead of utilizing get_variable_from_secrets.

Changes

• Add session=None keyword-only argument to Variable.get_variable_from_secrets.
• Explicitly check if the secrets_backend is MetastoreBackend, and if session is provided, forward it in the call_secrets_backend_method arguments.
• This allows the caller's session to be reused across the metadata DB lookup without triggering unexpected commits, clearing the way for deduplicating the dag.py loop.

[vincbeck]

This is already being worked on here: #72121

[ferruzzi]

I believe Vincent meant to link to #71968, but yes, the fix for this is already in review.

For what it's worth, if I were to review this as it sits now:

• we would need unit tests for these changes,
• this wouldn't have passed the formatting static checks or mypy from what I can tell,
• you are also not actually passing session in to get_variable_from_secrets anywhere, so this wouldn't actually fix the bug.

[saitejabandaru-in]

Thanks for the feedback! Since a fix is already in review in #71968, I will close this PR to avoid duplication. Appreciate your time!