Skip to content

fix(receiver): harden prefetch and listener lifecycle - #645

Open
GefMar wants to merge 4 commits into
masterfrom
fix/receiver-prefetch-listener-lifecycle
Open

fix(receiver): harden prefetch and listener lifecycle#645
GefMar wants to merge 4 commits into
masterfrom
fix/receiver-prefetch-listener-lifecycle

Conversation

@GefMar

@GefMarGefMar commented Jul 15, 2026

Copy link
Copy Markdown
Member

Track delivery and prefetch capacity ownership across shutdown paths. Close pending listener reads, preserve late deliveries, and restore fail-fast transport error propagation.

Keep middleware accounting balanced, report discarded deliveries, and retain deferred lifecycle diagnostics when task group failures overlap.

Refs #528

im evaluated PR#630 and retained its valid requirement: the Receiver must not pull a message it cannot execute.

However, that implementation also introduces a broad retry loop that treats transport, middleware, and programming errors alike, while pending-read, iterator, late-delivery, and semaphore ownership remain implicit.

This PR instead fixes the underlying ownership invariant. Each admitted delivery explicitly owns capacity from broker read through callback completion, pending reads are cancelled and awaited, iterators are closed, and failures remain visible to the worker lifecycle owner.

Retry and backoff remain a separate policy rather than being introduced implicitly as part of the prefetch fix.

Track delivery and prefetch capacity ownership across shutdown paths.
Close pending listener reads, preserve late deliveries, and restore
fail-fast transport error propagation.
Keep middleware accounting balanced, report discarded deliveries, and
retain deferred lifecycle diagnostics when task group failures overlap.
@GefMar
GefMarforce-pushed the fix/receiver-prefetch-listener-lifecycle branch from e15d278 to 4321e09CompareJuly 15, 2026 17:14
@GefMarGefMar linked an issue Jul 15, 2026 that may be closed by this pull request
@codecov

codecovBot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.80240% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.49%. Comparing base (ae2b788) to head (d16cee2).

Files with missing linesPatch %Lines
taskiq/receiver/receiver.py98.78%2 Missing ⚠️
Additional details and impacted files
@@ Coverage Diff @@## master #645 +/- ##
==========================================
+ Coverage 81.29% 82.49% +1.20% 
==========================================
Files 69 69 Lines 2577 2697 +120 ==========================================
+ Hits 2095 2225 +130 + Misses 482 472 -10 

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GefMar
GefMar marked this pull request as ready for review July 15, 2026 17:19
@GefMar
GefMar requested a review from s3riusJuly 15, 2026 17:19
Reject invalid prefetch values before receiver retries and retain delivery accounting until execution capacity is acquired.
Add deterministic coverage for CLI defaults, cancellation races, and listener lifecycle cleanup.
Refs #528
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
Comment threadtaskiq/receiver/receiver.py Outdated
- remove undocumented prefetch middleware hooks and metrics
- preserve listener errors during task group failures
- restore capacity when callback handoff fails
- strengthen shutdown and cleanup coverage
Refs: #528
@GefMar
GefMar requested a review from s3riusAugust 1, 2026 16:49
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

max-prefetch argument doesn't work as expected

2 participants

@GefMar@s3rius