Skip to content

fix: harden outbox claim leases - #28

Merged
patrickleet merged 2 commits into
mainfrom
fix/outbox-claim-leases
May 21, 2026
Merged

patrickleet merged 2 commits into
mainfrom
fix/outbox-claim-leases

Conversation

@patrickleet

@patrickleet patrickleet commented May 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • allow expired in-flight outbox messages to be claimed again while skipping unexpired claims
  • add explicit NotFound/InvalidState errors for missing IDs, stale leases, mismatched workers, and already-published messages
  • add worker-aware completion and publish-failure recording with retry-ceiling failure behavior
  • update threaded outbox worker to enforce max attempts and document outbox/domain-event boundaries

Verification

  • cargo fmt --check
  • git diff --check
  • cargo test --doc --all-features
  • cargo test --all-features outbox -- --nocapture
  • cargo test --all-features

Note: all-features still reports the existing Reservation dead-code warning in tests/sagas/order/inventory.rs.

Summary by CodeRabbit

  • Documentation

    • Clarified Outbox Pattern semantics and expanded Outbox Worker docs with lease-based claiming, worker-scoped examples, retry ceiling guidance, and example worker flows.
  • New Features

    • Worker-scoped APIs for claiming, completing, releasing, and recording publish failures with retry-ceiling behavior.
    • Public helpers to check message claim/lease status and ownership.
    • Clearer repository error reporting for missing or invalid-state messages.
  • Tests

    • Added tests for lease expiry reclaiming and retry-vs-fail behavior at the retry ceiling.

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 21, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 78bf9a40-74e8-405f-b671-99fc96e35e82

📥 Commits

Reviewing files that changed from the base of the PR and between c8a076e and 0fa4373.

📒 Files selected for processing (8)
  • README.md
  • src/lib.rs
  • src/outbox/message.rs
  • src/outbox/mod.rs
  • src/outbox_worker/mod.rs
  • src/outbox_worker/repository_ext.rs
  • src/outbox_worker/thread.rs
  • src/repository/error.rs
✅ Files skipped from review due to trivial changes (2)
  • src/outbox/mod.rs
  • README.md

📝 Walkthrough

Walkthrough

This PR adds time-explicit lease/claim checks to OutboxMessage, worker-scoped repository operations (complete/release/record failure) with a retry ceiling, new RepositoryError variants for validation, updates worker threads to use the new APIs, and updates README/module docs and re-exports.

Changes

Outbox lease semantics and worker-scoped failure handling

Layer / File(s) Summary
Documentation and crate re-exports
README.md, src/lib.rs, src/outbox/mod.rs, src/outbox_worker/mod.rs
Clarifies outbox vs aggregate event records, updates worker example to use worker_id, claim_outbox_messages, complete_outbox_message_for_worker, and record_outbox_publish_failure; adds lease-based claiming prose and re-exports OutboxPublishFailureAction.
OutboxMessage lease expiry and claimability helpers
src/outbox/message.rs
Adds has_expired_lease_at, is_claimable_at, is_claimed_by; gates MessageClaimed on claimability; claim_at now takes explicit now; test verifies re-claim after lease expiry.
Repository error types for validation
src/repository/error.rs
Adds RepositoryError::NotFound { id } and RepositoryError::InvalidState { id, expected, actual } with Display formatting to support repository validation.
Worker-scoped repository operations and publish failure handling
src/outbox_worker/repository_ext.rs
Adds OutboxPublishFailureAction; extends OutboxRepositoryExt with complete_outbox_message_for_worker, release_outbox_message_for_worker, and record_outbox_publish_failure; centralizes hydration/update via update_outbox_message and ensure_active_claim; refactors claiming and status queries; expands tests for claimability, worker isolation, retry ceiling, NotFound, and invalid-state errors.
Worker thread publish flow and failure recording
src/outbox_worker/thread.rs
Worker threads now call worker-scoped completion on success and record_outbox_publish_failure(..., DEFAULT_MAX_ATTEMPTS) on publish failure; adds tests with a failing publisher asserting transition to Failed and stats reflect failures.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Poem

🐰 I nibble leases, time in paw and heart,

I mark when claims from sleeping workers part.
No endless hops — a ceiling keeps the race,
Failed blooms rest softly in their place.
Hooray for tidy queues and bounded grace.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 46.67% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main objective: hardening outbox claim leases through expired lease reclamation, lease expiration checks, and worker-aware claim/completion semantics.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/outbox-claim-leases

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
README.md (1)

613-618: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Align the Outbox Worker example with the new worker-scoped contract.

This section now documents lease ownership and retry-ceiling failure handling, but the example still directly commits message.entity. That bypasses the worker-scoped APIs the prose recommends and can mislead users into skipping lease/worker validation.

Suggested README adjustment
 let mut claimed = repo.claim_outbox_messages("worker-1", 100, Duration::from_secs(30))?;
-let _ = worker.process_batch(&mut claimed);
+let _ = worker.process_batch(&mut claimed);

 for message in &mut claimed {
-    repo.commit(&mut message.entity)?;
+    match message.status() {
+        OutboxMessageStatus::Published => {
+            repo.complete_outbox_message_for_worker(message.id(), "worker-1")?;
+        }
+        OutboxMessageStatus::InFlight => {
+            let _action = repo.record_outbox_publish_failure(message.id(), "worker-1", 5)?;
+        }
+        _ => {}
+    }
 }

Also applies to: 621-625

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@README.md` around lines 613 - 618, The example bypasses the new worker-scoped
contract by calling repo.commit(&mut message.entity) directly; update the
snippet to use the repository's worker-scoped commit API (the one that accepts
the claimed outbox message and worker/lease info) instead of committing
message.entity so lease ownership and retry-ceiling checks are enforced; after
worker.process_batch(&mut claimed) call the worker-scoped commit method for each
message (pass the claimed message and the worker id/lease token) and handle any
lease/ retry-ceiling errors per the README prose rather than mutating
message.entity directly.
🧹 Nitpick comments (2)
src/outbox_worker/repository_ext.rs (1)

36-47: ⚡ Quick win

Constrain legacy non-worker mutation APIs to avoid ownership bypass.

Line 224 and Line 244 validate lease freshness but skip claimant ownership (ensure_active_claim(..., None, ...)), so callers can still complete/release another worker’s active claim through the non-worker methods. Consider deprecating these methods (or making them internal/admin-only) and steering callers to *_for_worker.

Also applies to: 46-55, 222-248

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/outbox_worker/repository_ext.rs` around lines 36 - 47, The non-worker
mutation APIs (complete_outbox_message, release_outbox_message) allow bypassing
claimant ownership because ensure_active_claim is called with None; restrict or
deprecate these to prevent one worker completing/releasing another’s claim. Mark
complete_outbox_message and release_outbox_message as deprecated/internal (or
change their visibility), update implementations to either call through to the
*_for_worker variants only after verifying the caller is an admin, or remove
their public surface; change ensure_active_claim calls so they require a
claimant (pass worker_id) where appropriate and update all call sites to use
complete_outbox_message_for_worker and release_outbox_message_for_worker (and
ensure implementations of those methods enforce claimant checks).
src/outbox_worker/thread.rs (1)

117-122: ⚡ Quick win

Persist the actual publish error.

Recording a fixed "publish failed" message throws away the broker error that explains why the message exhausted DEFAULT_MAX_ATTEMPTS. Capture the Err(err) value here and persist its text instead.

Also applies to: 208-213

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/outbox_worker/thread.rs` around lines 117 - 122, The call to
repo.record_outbox_publish_failure currently records the fixed string "publish
failed" which discards the broker error; change the code around the publish
failure handling (the branches that call repo.record_outbox_publish_failure with
msg.id(), &worker_id, "publish failed", DEFAULT_MAX_ATTEMPTS) to capture the
Err(err) returned by the publish attempt and pass err.to_string() (or
format!("{}", err)) as the error message argument instead of the literal
"publish failed", and apply the same change for the other occurrence around the
208-213 block so the real broker error text is persisted.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/outbox_worker/mod.rs`:
- Around line 26-28: The example loop binds msg immutably but calls
worker.process_message(&mut msg); change the loop to bind msg as mutable (e.g.,
use a mutable iterator so the loop variable is mutable) so that
worker.process_message(&mut msg) compiles; update the for ... in messages line
that precedes calls to worker.process_message(&mut msg) and
repo.complete_outbox_message_for_worker(msg.id(), "worker-1") accordingly (use
either "for mut msg in messages" or "for msg in &mut messages" depending on
whether messages is owned or borrowed).

In `@src/outbox_worker/thread.rs`:
- Around line 109-123: The repo calls currently drop all errors from
complete_outbox_message_for_worker and record_outbox_publish_failure; change the
logic in the worker loop to explicitly match and handle the repo error variants
returned by complete_outbox_message_for_worker and record_outbox_publish_failure
(e.g., stale-lease / wrong-worker / invalid-state vs other failures) instead of
ignoring Err(_): for complete_outbox_message_for_worker, if you get a
stale-lease/wrong-worker/invalid-state error treat the message as already
reclaimed (log at debug/warn and do not increment messages_published), but for
transient/errors that indicate DB failure log an error and increment
messages_failed (or emit a metric) so it won’t silently be re-published;
similarly for record_outbox_publish_failure, detect and handle
lease/worker/state errors separately (log and advance attempts or surface the
error) and only swallow benign concurrency errors—ensure you call the concrete
error-match arms on the Result from repo.complete_outbox_message_for_worker and
repo.record_outbox_publish_failure (use their returned error enum/variants) so
retries/attempt counters and reclaim behavior remain correct.

---

Outside diff comments:
In `@README.md`:
- Around line 613-618: The example bypasses the new worker-scoped contract by
calling repo.commit(&mut message.entity) directly; update the snippet to use the
repository's worker-scoped commit API (the one that accepts the claimed outbox
message and worker/lease info) instead of committing message.entity so lease
ownership and retry-ceiling checks are enforced; after worker.process_batch(&mut
claimed) call the worker-scoped commit method for each message (pass the claimed
message and the worker id/lease token) and handle any lease/ retry-ceiling
errors per the README prose rather than mutating message.entity directly.

---

Nitpick comments:
In `@src/outbox_worker/repository_ext.rs`:
- Around line 36-47: The non-worker mutation APIs (complete_outbox_message,
release_outbox_message) allow bypassing claimant ownership because
ensure_active_claim is called with None; restrict or deprecate these to prevent
one worker completing/releasing another’s claim. Mark complete_outbox_message
and release_outbox_message as deprecated/internal (or change their visibility),
update implementations to either call through to the *_for_worker variants only
after verifying the caller is an admin, or remove their public surface; change
ensure_active_claim calls so they require a claimant (pass worker_id) where
appropriate and update all call sites to use complete_outbox_message_for_worker
and release_outbox_message_for_worker (and ensure implementations of those
methods enforce claimant checks).

In `@src/outbox_worker/thread.rs`:
- Around line 117-122: The call to repo.record_outbox_publish_failure currently
records the fixed string "publish failed" which discards the broker error;
change the code around the publish failure handling (the branches that call
repo.record_outbox_publish_failure with msg.id(), &worker_id, "publish failed",
DEFAULT_MAX_ATTEMPTS) to capture the Err(err) returned by the publish attempt
and pass err.to_string() (or format!("{}", err)) as the error message argument
instead of the literal "publish failed", and apply the same change for the other
occurrence around the 208-213 block so the real broker error text is persisted.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3fa88cc7-0e11-4eb2-9023-c61365c26a87

📥 Commits

Reviewing files that changed from the base of the PR and between 21b93b9 and c8a076e.

📒 Files selected for processing (8)
  • README.md
  • src/lib.rs
  • src/outbox/message.rs
  • src/outbox/mod.rs
  • src/outbox_worker/mod.rs
  • src/outbox_worker/repository_ext.rs
  • src/outbox_worker/thread.rs
  • src/repository/error.rs

Comment thread src/outbox_worker/mod.rs Outdated
Comment thread src/outbox_worker/thread.rs Outdated
@patrickleet
patrickleet force-pushed the fix/outbox-claim-leases branch from c8a076e to 0fa4373 Compare May 21, 2026 04:11
@patrickleet

Copy link
Copy Markdown
Collaborator Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented May 21, 2026

Copy link
Copy Markdown
✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant