Skip to content

Bump structlog>=26.1.0 and croniter>=6.2.2 to fix memory leaks - #70694

Merged
potiuk merged 2 commits into
apache:mainfrom
jason810496:fix/bump-structlog-croniter-memory-leak-pins
Jul 30, 2026
Merged

Bump structlog>=26.1.0 and croniter>=6.2.2 to fix memory leaks#70694
potiuk merged 2 commits into
apache:mainfrom
jason810496:fix/bump-structlog-croniter-memory-leak-pins

Conversation

@jason810496

@jason810496jason810496 commented Jul 29, 2026

Copy link
Copy Markdown
Member

Why

The dag-processor memory leak in #50708 has two library-level root causes that are already fixed upstream: structlog leaks one lock and file-handle entry in WRITE_LOCKS per Dag-file log reopen on every parse cycle (fixed in 26.1.0), and croniter leaks on every cron schedule evaluation. The current minimums (structlog>=25.4.0, croniter>=2.0.2) still admit the leaky versions, so constraint-less installs and release-branch constraints can resolve them. Every released Airflow to date ships a leaky structlog (3.2.2 constraints pin 25.5.0).

The croniter floor is 6.2.2 rather than 6.1.0: the 6.1.0 wheel still ships the EXPRESSIONS global cache (removed in 6.2.0 despite the 6.1.0 changelog claim), and 6.2.1 fixed get_prev skipping Feb 29 on leap years, which feeds CronMixin._get_prev via infer_manual_data_interval and _align_to_prev. constraints-3.2.2 and constraints-3.3.0 already pin croniter==6.2.2.

Verified empirically: on 3.2.2-era versions with example Dags only, the standalone stack leaks ~34 MB/h of anonymous memory concentrated in the dag-processor, while a main-based image already running structlog 26.1.0 and croniter 6.2.x is near-flat (~9 MB/h residual across 8 processes). constraints-main already resolves the fixed versions, so this bump changes nothing for main CI and only raises the floor.

What

  • Bump structlog>=25.4.0 to structlog>=26.1.0 in all 8 declaring distributions: airflow-core, task-sdk, airflow-ctl, and shared dagnode, listeners, logging, observability, providers_discovery.
  • Bump croniter>=2.0.2 to croniter>=6.2.2 in airflow-core.
  • Remove workarounds in shared/logging that are dead code at the new structlog floor: the WRITE_LOCKS WeakKeyDictionary monkeypatch in configure_logging (structlog 26.1.0 ships it natively) and the _BYTES_LOGGER_HAS_NAME fallback in NamedBytesLogger (BytesLogger always has the name slot now). The _WRITE_LOGGER_HAS_NAME detection stays, since WriteLogger has not received the upstream name slot yet.
  • Regenerate uv.lock (requirement metadata only, locked versions unchanged: structlog 26.1.0, croniter 6.2.4).

Was generative AI tooling used to co-author this PR?

Comment threadairflow-core/pyproject.toml Outdated
Comment threadshared/logging/pyproject.toml
@jason810496jason810496 changed the title Bump structlog>=26.1.0 and croniter>=6.1.0 to fix memory leaksBump structlog>=26.1.0 and croniter>=6.2.2 to fix memory leaksJul 30, 2026
@potiuk
potiuk merged commit 724aaad into apache:mainJul 30, 2026
124 checks passed
@potiukpotiuk added this to the Airflow 3.3.1 milestone Jul 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

StatusBranchResult
v3-3-testPR Link

vatsrahul1001 pushed a commit that referenced this pull request Aug 3, 2026
…leaks (#70694) (#70749)
* Bump structlog>=26.1.0 and croniter>=6.1.0 to fix memory leaks
* Raise croniter floor to 6.2.2 and drop obsolete structlog workarounds
(cherry picked from commit 724aaad)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
vatsrahul1001 pushed a commit that referenced this pull request Aug 5, 2026
…leaks (#70694) (#70749)
* Bump structlog>=26.1.0 and croniter>=6.1.0 to fix memory leaks
* Raise croniter floor to 6.2.2 and drop obsolete structlog workarounds
(cherry picked from commit 724aaad)
Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
…e#70694)
* Bump structlog>=26.1.0 and croniter>=6.1.0 to fix memory leaks
* Raise croniter floor to 6.2.2 and drop obsolete structlog workarounds
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jason810496@potiuk@kaxil