Skip to content

test(partitions): reproduce purge offset directory sync failure - #4130

Draft
diegomrsantos wants to merge 2 commits into
apache:masterfrom
diegomrsantos:test/purge-offset-directory-sync
Draft

diegomrsantos wants to merge 2 commits into
apache:masterfrom
diegomrsantos:test/purge-offset-directory-sync

Conversation

@diegomrsantos

@diegomrsantos diegomrsantos commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Purge can return success and record its applied generation after syncing a consumer offset directory fails. A later power loss can restore the deleted offset files while the completion marker survives, leaving old consumer progress with no purge retry pending.

Refs #4128.

This draft adds regression tests against master at e0027506f2a0379ee3bb5508427ff570e3a9a435. It contains no production fix.

The tests call the real purge() method and inject a sync failure for each offset directory separately. The fixture persists progress for an individual consumer and a consumer group, records generation 4, then requests generation 5. It checks that the fault was reached and the offset files were deleted, then reloads the applied generation into a fresh partition.

Both failure cases expect purge to return an error and leave generation 4 in memory and on disk. Instead, purge returns success and both generations advance to 5. A third test allows both directory syncs to succeed and confirms generation 5 is recorded.

The two failure tests intentionally remain red until the bug is fixed. This PR reloads only the generation marker. It does not simulate power loss, restore deleted offset files, reload consumer progress, check Next, or test retries with fresh writes. The fixture writes durable offset files but does not run both consumer offset durability policies. The simulator coverage proposed in the issue discussion would cover those remaining cases.

Run the focused reproduction with

cargo test -p partitions purge_offset_directory_sync_tests -- --nocapture

The original macOS run produced one passing control and two expected failures. CI also reproduces the group directory failure. Formatting, dependency ordering, Clippy, and TOML checks pass in CI.

Purge records its applied generation even when synchronizing consumer
offset deletions fails, which can suppress cleanup during recovery.

Add deterministic failures for both offset directories and a successful
control. Verify the real unlinks and the generation recovered from disk.
The failure regressions remain red until the purge completion path is fixed.

Refs apache#4128.
@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.47%. Comparing base (ad8951f) to head (33c07b3).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4130      +/-   ##
============================================
- Coverage     87.47%   87.47%   -0.01%     
  Complexity     1575     1575              
============================================
  Files          1280     1280              
  Lines        223241   223230      -11     
  Branches     186604   186593      -11     
============================================
- Hits         195290   195273      -17     
- Misses        23241    23250       +9     
+ Partials       4710     4707       -3     
Components Coverage Δ
Rust Core 88.54% <ø> (-0.01%) ⬇️
Java SDK 68.68% <ø> (ø)
C# SDK 77.41% <ø> (-0.05%) ⬇️
Python SDK 90.97% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 96.43% <ø> (-0.02%) ⬇️
Go SDK 70.11% <ø> (-0.02%) ⬇️
Files with missing lines Coverage Δ
core/partitions/src/iggy_partition.rs 92.83% <ø> (+<0.01%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant