Skip to content

Enable ruff B023 (function-uses-loop-variable) and fix violations - #70640

Merged
potiuk merged 2 commits into
apache:mainfrom
dkranchii:enable-ruff-b023-fix-closure-loop-var
Aug 29, 2026
Merged

Enable ruff B023 (function-uses-loop-variable) and fix violations#70640
potiuk merged 2 commits into
apache:mainfrom
dkranchii:enable-ruff-b023-fix-closure-loop-var

Conversation

@dkranchii

Copy link
Copy Markdown
Contributor

B023 catches the late-binding closure-over-loop-variable footgun where a function defined inside a loop captures the loop variable by reference, so every function in the resulting list sees the same (final) value — a classic silent-bug source in Python. See the Python FAQ entry.

The one user-visible fix is in providers/standard/.../triggers/file.py, where the FileTrigger's os.walk lambda was dispatched to a worker thread via anyio.to_thread.run_sync while the outer glob iteration could advance, potentially walking the wrong path.

The rest are pre-existing latent-bug or false-positive sites fixed by binding the loop-derived variable as a default argument on the inner function or lambda:

  • airflow-core/.../0101_3_2_0_ui_improvements_for_deadlines.py (migration; 7 sites in one nested function)
  • dev/breeze/.../ui_commands.py (two i18n helpers)
  • devel-common/src/sphinx_exts/providers_extensions.py (two class-extras lambdas)
  • providers/apache/hive/tests/unit/.../test_s3_to_hive.py
  • providers/exasol/tests/unit/exasol/hooks/test_sql.py
  • providers/google/tests/system/.../example_cloud_sql_query.py
  • providers/smtp/src/airflow/providers/smtp/hooks/smtp.py (OAuth2 auth callback)
  • shared/secrets_masker/.../secrets_masker.py (subclass compat shim)

Follows the pattern of #66977 (B015), #66978 (PLE1205), #66979 (B008), and #66960 (trigger init/serialize static check).


Was generative AI tooling used to co-author this PR?
  • Yes — Cursor

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — B023 is a worthwhile rule to turn on; late-bound closures over loop variables are a genuinely nasty bug class because they fail silently and only under specific timing. Adding it to pyproject.toml alongside the fixes is the right way to land it.

I traced the two most interesting cases rather than assuming:

  • triggers/file.py is the nicest fix in the set — rewriting run_sync(lambda: list(os.walk(path))) as run_sync(lambda p: list(os.walk(p)), path) passes the value explicitly instead of relying on a default-argument trick. Worth using that form elsewhere where the callee accepts arguments; it reads as intent rather than as lint appeasement.
  • The migration (0101_3_2_0_ui_improvements_for_deadlines.py) is a false positive in practice: _migrate_dag_deadlines is defined at line 597 and called at line 688 within the same loop iteration, so late binding was never observable there.

Two changes requested inline before this lands — one about rewriting a shipped migration for no behavioural gain, one about a binding that leaks into a public keyword signature in the secrets-masking path. Neither is hard to address.

These are judgement calls rather than mechanical fixes, so I'd like your own reasoning in reply rather than just an autofix — my review here was AI-assisted and shouldn't be treated as settled.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Comment threadshared/secrets_masker/src/airflow_shared/secrets_masker/secrets_masker.py Outdated
@dkranchii
dkranchiiforce-pushed the enable-ruff-b023-fix-closure-loop-var branch from e2509d8 to adfe70dCompareAugust 1, 2026 19:22
B023 catches the late-binding closure-over-loop-variable footgun where
a function defined inside a loop captures the loop variable by
reference, so every function in the resulting list sees the same
(final) value — a classic silent-bug source in Python
(https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result).
The one user-visible fix is in providers/standard/.../triggers/file.py,
where the FileTrigger's os.walk lambda was dispatched to a worker
thread via anyio.to_thread.run_sync while the outer glob iteration
could advance, potentially walking the wrong path.
The rest are pre-existing latent-bug or false-positive sites in a
migration script, breeze translation helpers, sphinx extensions, a
system-test example DAG, the SMTP OAuth2 auth callback, secrets_masker
subclass compat shim, and two provider unit tests — fixed by binding
the loop-derived variable as a default argument on the inner function
or lambda.
@dkranchii
dkranchiiforce-pushed the enable-ruff-b023-fix-closure-loop-var branch from adfe70d to 743fc02CompareAugust 4, 2026 06:45
@eladkal
eladkal requested a review from potiukAugust 29, 2026 14:10

@potiukpotiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points addressed, and on the migration you were right to push back on my suggestion.

# noqa: B023 on the def line would indeed have suppressed nothing — ruff flags the reads inside the closure body, not the definition, so it would have taken seven per-line noqas to do what I asked for, in a shipped migration where body noise is exactly what we were trying to avoid. The module-level # ruff: noqa: B023 with the invariant written out, plus the explicit "do not rewrite the body for lint hygiene" note, is a better answer than the one I proposed. Thanks for checking rather than just applying it.

The secrets-masker factory is right too, and your reasoning is the reason it matters: _redact recurses through arbitrary user data, so a **kwargs path that let a caller slip in _f would substitute a different function inside the redaction walk. The comment at the site should stop someone collapsing it back to a default-arg binding.

I have updated the branch from main. The one red check — Postgres 18 / Py3.14 · API…CLI — was from 4 August, ran for six minutes, and produced no pytest summary at all, so it never reached the test stage; that is an infrastructure or expired-image failure rather than anything in this diff. A lint rule plus behaviour-preserving rewrites would not break a single matrix cell of core API/CLI DB tests and nothing else.

Approving on that basis. If the fresh run comes back red on something real, ping me and I will look again rather than leaving you to guess at it.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit af0a377 into apache:mainAug 29, 2026
581 of 583 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-3-testPR Link

potiuk added a commit that referenced this pull request Aug 29, 2026
…olations (#70640) (#72273)
B023 catches the late-binding closure-over-loop-variable footgun where
a function defined inside a loop captures the loop variable by
reference, so every function in the resulting list sees the same
(final) value — a classic silent-bug source in Python
(https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result).
The one user-visible fix is in providers/standard/.../triggers/file.py,
where the FileTrigger's os.walk lambda was dispatched to a worker
thread via anyio.to_thread.run_sync while the outer glob iteration
could advance, potentially walking the wrong path.
The rest are pre-existing latent-bug or false-positive sites in a
migration script, breeze translation helpers, sphinx extensions, a
system-test example DAG, the SMTP OAuth2 auth callback, secrets_masker
subclass compat shim, and two provider unit tests — fixed by binding
the loop-derived variable as a default argument on the inner function
or lambda.
(cherry picked from commit af0a377)
Co-authored-by: Deepak kumar <deepakkumar@meta.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@dkranchii@potiuk