Skip to content

[Backend] Indexer replay triggerPoll() bypasses the activeBatch mutex - concurrent poll/replay race and shutdown drain gap #843

Description

@grantfox-oss

Telegram (ask questions / claim the issue here first): https://t.me/+DOylgFv1jyJlNzM0

Labels: bug, backend, Stellar Wave

An admin replay can run the indexer concurrently with a scheduled poll, and the slower one can rewind the cursor.

At soroban-event-worker.ts:132-140, triggerPoll() calls fetchAndProcessEvents() directly, while the scheduled poll() wraps it in this.activeBatch. There's no mutex between them, so a replay and a scheduled poll can be in flight at once. Both write a cursor when they finish, and whichever writes last wins, so a slower run can regress lastCursor. On top of that, waitForDrain() only awaits activeBatch, so a replay batch is invisible to graceful shutdown and can get cut off mid-batch.

What the fix has to hold to

  • only one fetchAndProcessEvents runs at a time, whether it came from the scheduler or triggerPoll
  • shutdown waits for a replay batch to finish, not just scheduled ones
  • the cursor can't be moved backwards by an overlapping run

Done when

  • all fetchAndProcessEvents runs serialized behind one in-flight guard/promise
  • triggerPoll registers its batch with activeBatch so waitForDrain awaits it
  • test asserting overlapping triggerPoll/poll calls don't run fetchAndProcessEvents concurrently
  • tests + CI green

Where to start
backend/src/workers/soroban-event-worker.ts, with a look at backend/src/services/indexerService.ts. Removing the legacy SorobanIndexerService is tracked separately, leave it. The guard is small; the shutdown-drain part is where it's easy to miss a case.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26Stellar WaveIssues in the Stellar wave programbackendBackend related tasksbugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions