Skip to content

Reduce Iceberg snapshot watcher thread use - #72173

Merged
vincbeck merged 2 commits into
apache:mainfrom
1fanwang:1fanwang-iceberg-async-state-store
Aug 28, 2026
Merged

Reduce Iceberg snapshot watcher thread use#72173
vincbeck merged 2 commits into
apache:mainfrom
1fanwang:1fanwang-iceberg-async-state-store

Conversation

@1fanwang

Copy link
Copy Markdown
Contributor

Each stateful Iceberg snapshot watcher submits worker-thread jobs to load and save its watermark. A triggerer running many watchers spends executor capacity on storage that already has an async API.

After this change, stateful watchers await watermark storage directly. Worker threads remain limited to synchronous PyIceberg catalog calls. Watchers without an injected state store keep their existing behavior.

Depends on #72127 and must merge after it.

Test output
$ uv run --project providers/apache/iceberg pytest providers/apache/iceberg/tests/unit/apache/iceberg/triggers/test_iceberg.py -xvs
# BeforeFAILED providers/apache/iceberg/tests/unit/apache/iceberg/triggers/test_iceberg.py::test_resumes_from_the_stored_watermarkE AssertionError: assert [...] == []1 failed, 10 passed
# After19 passed, 1 warning in 17.57s

Was generative AI tooling used to co-author this PR?
  • Yes: GitHub Copilot CLI 1.0.80 (GPT-5.6 Sol)

Generated-by: GitHub Copilot CLI 1.0.80 (GPT-5.6 Sol) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.

The trigger already runs on the event loop, so synchronous state access needlessly consumes worker threads and blocks async-only state stores.
Signed-off-by: 1fanwang <1fannnw@gmail.com>
The previous wording still counted state-store calls as thread-pool work after the async migration.
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@vincbeck
vincbeck merged commit 62ecde0 into apache:mainAug 28, 2026
79 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@1fanwang@vincbeck