fix(core-backend): reconnection logic by Kriys94 · Pull Request #6861 · MetaMask/core

Kriys94 · 2025-10-16T14:45:45Z

Explanation

Current State

The WebSocket service experienced a critical issue when the backend server accepted connections but immediately disconnected afterward. I was only able to reproduce this behavior locally, not in prod endpoint.

The Problem

When the server accepts the WebSocket connection but then disconnects rapidly:

The client would immediately attempt to reconnect (fast reconnection)
This created a connection thrashing loop: connect → disconnect → reconnect → disconnect
Multiple code paths could trigger reconnection attempts independently
The scheduleConnect method could be called multiple times during rapid disconnect cycles
Multiple overlapping reconnection attempts could be scheduled simultaneously
All disconnected clients would reconnect at exactly the same time, creating a "thundering herd" problem
This led to unpredictable behavior, duplicate connection attempts, and worsened the reconnection storm

The Solution

This PR fixes the issue with three key improvements:

1. Centralize Reconnection via `scheduleConnect`

All reconnection logic now flows through a single entry point:

All code paths that need to trigger reconnection now use scheduleConnect
Eliminates scattered reconnection logic across the codebase
Provides a single point of control for reconnection behavior
Makes the reconnection strategy consistent and predictable
Easier to maintain, test, and debug reconnection logic

2. Idempotent `scheduleConnect`

Makes scheduleConnect idempotent to ensure multiple calls don't create duplicate or overlapping connection attempts:

Checks if a reconnection is already scheduled before creating a new one
Ensures only one reconnection timer is active at any given time
Eliminates race conditions from concurrent connection attempts
Maintains clear, single-threaded reconnection state

3. Jitter on Reconnection

Adds randomized jitter to reconnection timing to prevent thundering herd problems:

Randomizes the reconnection delay to spread out reconnection attempts across clients
When the server disconnects many clients simultaneously, they won't all reconnect at the exact same moment
Reduces load spikes on the server during recovery from incidents
Improves overall system stability and reliability

Combined Effect:
With all three fixes, when the server accepts connections but disconnects immediately:

All reconnection attempts flow through a single, centralized method (consistency)
Only one reconnection attempt will be scheduled per client (idempotency)
Reconnection attempts are staggered across the client population (jitter)
The reconnection logic follows proper backoff strategy with randomization
No duplicate timers or overlapping connection attempts
Predictable, controlled reconnection behavior that respects server health
Server experiences gradual reconnection load rather than synchronized spikes

Technical Details

The fix centralizes all reconnection logic through scheduleConnect, makes it idempotent by guarding against duplicate timer creation, and adds jitter to the reconnection delay calculation to randomize timing across clients. This prevents both the individual reconnection storm (from rapid connect/disconnect cycles) and the collective thundering herd problem (from synchronized reconnections), while ensuring consistent behavior across the entire service.

References

Related to BackendWebSocketService in @metamask/core-backend
Fixes # [ADD ISSUE NUMBER HERE]

Checklist

I've updated the test suite for new or updated code as appropriate
I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate
I've communicated my changes to consumers by updating changelogs for packages I've changed, highlighting breaking changes as necessary
I've prepared draft pull requests for clients and consumer packages to resolve any breaking changes

Note

Introduces a controlled forceReconnection() flow, idempotent/jittered backoff scheduling, and stability timer to prevent reconnect thrashing; updates AccountActivityService to use it and expands tests.

BackendWebSocketService
- New API: forceReconnection() messenger action and method for controlled disconnect-then-reconnect.
- Reconnection: Idempotent scheduler using ExponentialBackoff with jitter; connect() no-ops if a reconnect is scheduled; stable-connection timer (10s) resets attempts/backoff.
- Behavior changes: disconnect() now synchronous and resets attempts; improved error handling and timer cleanup; avoids duplicate timers and races.
- Actions/Types: Expose forceReconnection; update method action types union.
AccountActivityService
- Switch to BackendWebSocketService:forceReconnection for recovery; remove direct disconnect/connect sequence.
- Update allowed messenger actions accordingly.
Tests
- Add/adjust tests for new reconnection semantics, stable timer, idempotent scheduling, and force reconnection flows; minor test cleanup tweaks.
Changelog
- Document new API, reconnection improvements, fixes for race conditions/memory leaks, and breaking messenger action change for AccountActivityService.

^{Written by Cursor Bugbot for commit 373a4f6. This will update automatically on new commits. Configure here.}

Kriys94 · 2025-10-21T13:10:05Z

- }
+ // Use the dedicated forceReconnection method which performs a controlled
+ // disconnect-then-connect sequence to clean up subscription state
+ await this.#messenger.call('BackendWebSocketService:forceReconnection');


I did this because I want to fully control the reconnection via scheduleConnect

Kriys94 · 2025-10-21T14:24:19Z

+export type BackendWebSocketServiceForceReconnectionAction = {
+ type: `BackendWebSocketService:forceReconnection`;
+ handler: BackendWebSocketService['forceReconnection'];
+};


I did this because I want to fully control the reconnection via scheduleConnect

Kriys94 · 2025-10-21T14:29:15Z

@@ -299,6 +300,8 @@ export class BackendWebSocketService {

 #connectionTimeout: NodeJS.Timeout | null = null;

+ #stableConnectionTimer: NodeJS.Timeout | null = null;


Even if the backend accepts a connection, we make sure that we dont re-initialise the the reconnectionAttempt (used for the exponential backoff). This makes sure that the exponential backoff is not bypassed.

Kriys94 · 2025-10-21T14:29:49Z

+ // This prevents rapid loops when server accepts then immediately closes connections
+ if (this.#reconnectTimer) {
+ return;
+ }


// If a reconnect is already scheduled, defer to it to avoid bypassing exponential backoff

Kriys94 · 2025-10-21T14:30:23Z

@@ -452,15 +459,11 @@ export class BackendWebSocketService {
 'AuthenticationController:getBearerToken',
 );
 if (!token) {
- this.#scheduleReconnect();


move this into the catch block L488

Kriys94 · 2025-10-21T14:44:49Z

- #scheduleReconnect(): void {
+ #scheduleConnect(): void {
+ // If a reconnect is already scheduled, don't schedule another one
+ if (this.#reconnectTimer) {


Important, before I was just replacing this.#reconnectTimer. Now I just bypass if there is already an reconnectTimer in flight

mcmire

It looks like there was a breaking change to AccountActivityService not mentioned in the changelog, but other than that, seems like a good set of changes.

mcmire · 2025-10-21T18:49:13Z

@@ -1561,7 +1765,8 @@ describe('BackendWebSocketService', () => {
 mockEnabledCallback.mockReturnValue(false);

 // Advance time to trigger reconnection check
- await completeAsyncOperations(50);
+ // With jitter (±25%), delay with reconnectDelay=50ms is 37.5-62.5ms
+ await completeAsyncOperations(70);


Tip: Similar as above, you might try mocking Math.random if this test starts failing intermittently.

Now I mock Math.random, but I prefer to keep 70 because 50 is causing some test flakiness, given that in this test, reconnectDelay: 50, it's better to have some room to make the reconnection really happen.

mcmire · 2025-10-21T18:52:48Z

 this.#reconnectAttempts += 1;

+ // Calculate delay with exponential backoff


Nit: Probably not important now, but if we like we might want to reuse the same exponential backoff formula we are using for network requests. The Cockatiel library provides a way to generate a series of backoffs here: https://github.com/connor4312/cockatiel/blob/f87137b9bb55e0cdfb02e7d5c401e47c6f5fcc9f/src/backoff/ExponentialBackoff.ts (this is the formula itself).

THanks for this. I replaced with the Cockatiel library and it works well

## Explanation This PR releases **@metamask/core-backend@3.0.0** with significant improvements to WebSocket reconnection reliability and robustness for real-time balance updates. ### Current State & Problems Addressed The previous WebSocket implementation had several reliability issues: 1. **Duplicate reconnection timers** could be created when multiple events triggered reconnects simultaneously, leading to memory leaks and inflated reconnect attempt counters 2. **Rapid reconnection loops** could occur when the server accepted connections but immediately closed them 3. **Race conditions** in reconnection scheduling could bypass exponential backoff, causing aggressive retry behavior 4. **Manual subscription cleanup** was fragile when recovering from subscription issues or orphaned subscriptions ### Solution This release introduces robust reconnection logic with the following improvements: #### 1. **Idempotent Reconnection Scheduling** - Prevents duplicate reconnection timers from being created - Checks if a reconnect is already scheduled before creating a new timer - Eliminates memory leaks from orphaned timers - Prevents unnecessary growth of reconnect attempt counters #### 2. **Stable Connection Timer** - Connection must remain stable for 10 seconds before resetting reconnect attempts - Prevents rapid reconnection loops when server accepts then immediately closes connections - Provides grace period for connection stability before declaring success #### 3. **Force Reconnection Method** - New `forceReconnection()` public API for controlled subscription state cleanup - Performs a controlled disconnect-then-reconnect sequence with exponential backoff - Useful for recovering from subscription/unsubscription issues and cleaning up orphaned subscriptions - Exposed via `BackendWebSocketService:forceReconnection` messenger action #### 4. **Improved Error Handling** - Always schedules reconnect on connection failure (exponential backoff prevents aggressive retries) - Removes redundant schedule calls from error paths - Better logging throughout for easier debugging #### 5. **Better State Management** - `disconnect()` now properly resets reconnect attempts counter - Fixed race condition in `connect()` that could bypass exponential backoff - More predictable state transitions ### Technical Details **Reconnection Behavior:** - Uses Cockatiel's exponential backoff with decorrelated jitter to prevent thundering herd - Initial delay: 1000ms, max delay: 30000ms - Backoff sequence resets only after 10 seconds of stable connection - Automatic reconnect on unexpected disconnects, stays disconnected on manual disconnects **Breaking Changes:** - `AccountActivityService` messenger allowed actions updated: removed `BackendWebSocketService:disconnect`, added `BackendWebSocketService:forceReconnection` ### Impact This improves the reliability of real-time balance updates across MetaMask Extension and Mobile by: - ✅ Preventing connection thrashing and resource exhaustion - ✅ Providing graceful recovery from network instability - ✅ Eliminating memory leaks from reconnection logic - ✅ Offering controlled subscription cleanup via `forceReconnection()` - ✅ Enabling better monitoring with improved logging ## References - Related to the broader WebSocket balance update initiative for MetaMask Mobile - Fixes#6861 - WebSocket reconnection improvements ## Checklist - [x] I've updated the test suite for new or updated code as appropriate - [x] I've updated documentation (JSDoc, Markdown, etc.) for new or updated code as appropriate - [x] I've communicated my changes to consumers by updating changelogs for packages I've changed, highlighting breaking changes as necessary - [ ] I've prepared draft pull requests for clients and consumer packages to resolve any breaking changes (MetaMask Extension and Mobile updated)  --- > [!NOTE] > Releases `@metamask/core-backend@3.0.0` with robust WebSocket reconnection and updates dependent packages/peer deps (`assets-controllers@83`, `bridge-controller@55`, `bridge-status-controller@55`); bumps monorepo version to 638.0.0. > > - **Core Backend** > - Release `@metamask/core-backend@3.0.0` with improved, idempotent WS reconnection, stable-connection timer, `forceReconnection()` API, logging and error‑handling tweaks. > - BREAKING: `AccountActivityService` messenger actions updated (remove `disconnect`, add `BackendWebSocketService:forceReconnection`). > - **Assets** > - `@metamask/assets-controllers@83.0.0`: BREAKING peer dep bump to `@metamask/core-backend@^3.0.0`. > - **Bridge** > - `@metamask/bridge-controller@55.0.0`: BREAKING peer/dev dep bump to `@metamask/assets-controllers@^83.0.0`. > - `@metamask/bridge-status-controller@55.0.0`: BREAKING peer/dev dep bump to `@metamask/bridge-controller@^55.0.0`. > - **Monorepo** > - Bump root version to `638.0.0`; update changelogs and package versions accordingly. > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 54996e2. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>

…r/multichain-transactions-controller * origin/main: (35 commits) feat: `JsonRpcEngineV2` (#6176) Release 641.0.0 (#6940) feat: Add transaction emulation actions (#6935) Release/640.0.0 (#6934) fix(core-backend): control randomness to fix flaky test (#6936) chore: Add `@metamask-previews/*` to NPM age gate exceptions (#6937) Release/639.0.0 (#6931) feat: make getCryptoApproveTransactionParams synchronous (#6930) feat: add new actions to `KeyringController` (#6928) feat: add `getAccounts` to `AccountsController` (#6927) chore: remove `Monad Mainnet` single call balance contract and add into account v4 (#6929) Release/638.0.0 (#6923) fix: Downgrade `multiformats` to `^9.9.0` to avoid ESM-only dependency (#6920) Release/637.0.0 (#6919) feat(account-tree-controller): add callbacks for hidden and pinned data (#6910) Release 636.0.0 (#6918) fix(core-backend): reconnection logic (#6861) fix: Tx state listener and signature coverage (#6906) Release/635.0.0 (#6917) fix(base-controller): add TypeScript declaration file for legacy module resolution (#6915) ...

## Summary - Fix `OHLCVService` unsubscribe cleanup so failed WebSocket unsubs no longer leak gateway subscription slots and block future OHLCV streams. - Flush grace-period channels immediately when subscribing to a different asset/interval, while keeping same-channel grace reuse for rapid back navigation. - Retry failed unsubs with backoff (`1s → 2s → 4s`) and call `forceReconnection` only after retries are exhausted. ## Explanation **Current state:** When `#performUnsubscribe` failed, `OHLCVService` removed the channel from local tracking before the gateway unsub succeeded. That left orphaned server-side subscriptions, which could exhaust the gateway's 2 `market-data` subscription cap and cause later subscribes to fail (clients fall back to REST `/latest` polling). **Solution:** 1. **Delete-on-success** — keep channel entries until WS unsub succeeds. 2. **Grace-period flush** — when subscribing to a different channel, immediately unsub other channels in grace/failed-cleanup instead of holding two live subs for up to 3s. Same-channel grace reuse also flushes other channels first (Bugbot fix). 3. **Retry + last-resort reconnect** — on unsub failure, retry with backoff; after 3 retries, call `BackendWebSocketService:forceReconnection` to reset server subscription state. Same-channel grace (3s) is unchanged so users returning to the same token/interval within 3s can reuse the existing subscription. ## Important: `forceReconnection` affects the shared WebSocket `BackendWebSocketService` is a **single shared connection** on mobile, used by multiple services (not OHLCV-only): | Consumer | Purpose | |----------|---------| | `AccountActivityService` | Real-time account balance / activity push | | `OHLCVService` | Chart OHLCV streaming | | `AssetsController` (indirect) | Forwards `AccountActivityService:balanceUpdated` | Calling `forceReconnection()` closes and reconnects **that entire WebSocket**. On disconnect it clears all local subscription state; on reconnect each service resubscribes what it needs: - **`AccountActivityService`** — marks chains down on `DISCONNECTED`, then resubscribes the selected account on `CONNECTED` (this service already uses `forceReconnection` for its own cleanup today). - **`OHLCVService`** — only resubscribes channels with `refCount > 0` (active chart consumers). Failed-cleanup / grace entries (`refCount === 0`) are dropped on reconnect and are **not** resubscribed. ### When does OHLCV trigger it? Only as a **last resort**, after unsub has failed **4 times total** (initial grace-period attempt + 3 retries at 1s / 2s / 4s). Typical navigation (leave chart, switch token, return within grace) uses delete-on-success, flush, or grace reuse — **not** force reconnect. Rough timeline after user leaves a chart with a stuck unsub: ~**10s** of background retries before reconnect fires. ### User-visible impact | Scenario | Impact | |----------|--------| | User **left chart**, unsub stuck → force reconnect (Test 5) | Brief (~1–2s) WS gap for **account activity** push; chart unaffected (hook unmounted). OHLCV does not resubscribe. | | User **has chart open** when reconnect fires (unlikely — retries run after `refCount === 0`) | Brief WS gap; OHLCV resubscribes active channel; mobile hook may poll REST briefly during staleness window. | | Normal chart navigation | **No** force reconnect | ### Why we still use it - A permanently leaked `market-data` sub hits the gateway **2-sub cap** and can break **all** OHLCV streaming on that connection — worse than a one-time reconnect blip. - `forceReconnection` is an existing, shared primitive (introduced in MetaMask#6861; `AccountActivityService` already relies on it). - Retries are attempted first; reconnect is the escalation path only when repeated unsub attempts fail. **Open question for reviewers:** If the cross-service reconnect blast radius is unacceptable, alternatives are (a) gateway-side sub cleanup API, or (b) OHLCV-only reconnect/isolation — neither exists today on the shared `BackendWebSocketService`. ## Fixes - [ASSETS-3763](https://consensyssoftware.atlassian.net/browse/ASSETS-3763) ## References - Related mobile investigation: `UNSUBSCRIBE_LEAK_FIX_PLAN.md` in metamask-mobile - Gateway behavior confirmed with backend: steady state is 1 live OHLCV sub per connection; cap of 2 is a race buffer for unsub→sub overlap - `BackendWebSocketService.forceReconnection()` — MetaMask#6861 ## Test plan - [x] `yarn workspace @metamask/core-backend run jest --no-coverage packages/core-backend/src/ws/ohlcv/OHLCVService.test.ts` (43 tests) - [x] Manual (mobile, debug build): unsub retry — fail 4×, succeed on 5th - [x] Manual: same-token return within grace — reuse WS sub, no new subscribe - [x] Manual: token switch within grace — flush old channel immediately - [x] Manual: grace reuse flushes other channels first (Bugbot scenario) - [x] Manual: exhausted retries → `forceReconnection` → WS disconnected / connected cycle - [ ] Manual: rapid token hopping on TDP — verify third subscribe succeeds (no `limit would be exceeded`) ## Checklist - [x] I've updated the test suite for new or updated code as appropriate - [ ] I've documented the shared-WebSocket impact of `forceReconnection` in code (JSDoc) — **PR description above; happy to add JSDoc in follow-up if reviewers want it in-tree** - [x] I've communicated my changes to consumers by updating changelogs for packages I've changed - [ ] I've introduced breaking changes in this PR and have prepared draft pull requests for clients and consumer packages to resolve them [ASSETS-3763]: https://consensyssoftware.atlassian.net/browse/ASSETS-3763?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ  --- > [!NOTE] > **High Risk** > Changes real-time OHLCV subscription lifecycle and can trigger shared WebSocket `forceReconnection`, affecting other consumers (e.g. account activity) after repeated unsub failures. > > **Overview** > Fixes **OHLCV WebSocket unsubscribe leaks** that could leave orphaned gateway `market-data` subscriptions and hit the **2-sub cap**, breaking later chart streams. > > **`OHLCVService`** now keeps channel entries until the server unsub succeeds (**delete-on-success**). When subscribing to a **different** asset/interval, it **flushes** other channels still in grace or failed cleanup instead of holding two live subs for up to 3s; same-channel grace reuse is unchanged but also flushes other pending channels first. Failed unsubs are retried via **cockatiel** with **1s → 2s → 4s** backoff; only after retries are exhausted does it call **`BackendWebSocketService:forceReconnection`** on the shared connection. Retries are cancellable via **`AbortController`** (resubscribe, `destroy`, reconnect drops `refCount === 0` entries). **`destroy`** clears grace and retry timers; reconnect no longer resurrects stuck cleanup channels. > > Adds **`cockatiel`** to `@metamask/core-backend` and expands **`OHLCVService.test.ts`** for flush, retry, destroy races, and connection-state edge cases. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f44eea8. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>

Kriys94 force-pushed the fix/WebsocketReconnection branch 5 times, most recently from d26e7f6 to f128e8bCompare October 20, 2025 09:09

Kriys94 marked this pull request as ready for review October 20, 2025 09:14

Kriys94 requested review from a team as code owners October 20, 2025 09:14

salimtb previously approved these changes Oct 21, 2025
View reviewed changes

Kriys94 commented Oct 21, 2025
View reviewed changes

mcmire reviewed Oct 21, 2025
View reviewed changes

Kriys94 dismissed salimtb’s stale review via ee521a1October 22, 2025 12:08

Kriys94 force-pushed the fix/WebsocketReconnection branch from ee521a1 to 4c1ce0bCompare October 22, 2025 12:10

fix(core-backend): reconnection logic
c165f71

Kriys94 force-pushed the fix/WebsocketReconnection branch 2 times, most recently from 0c2f17b to d1bfe99Compare October 22, 2025 12:40

fix(core-backend): clean code
d602e67

Kriys94 force-pushed the fix/WebsocketReconnection branch from d1bfe99 to d602e67Compare October 22, 2025 12:58

salimtb approved these changes Oct 22, 2025
View reviewed changes

Merge branch 'main' into fix/WebsocketReconnection
373a4f6

Kriys94 merged commit 8974527 into mainOct 22, 2025
255 checks passed

Kriys94 deleted the fix/WebsocketReconnection branch October 22, 2025 15:23

Kriys94 mentioned this pull request Oct 23, 2025
Release/638.0.0 #6923
Merged
4 tasks

sahar-fehri mentioned this pull request Jul 28, 2026
chore: improve unsub failure and retry logic #9678
Merged
11 tasks

		@@ -299,6 +300,8 @@ export class BackendWebSocketService {

		#connectionTimeout: NodeJS.Timeout \| null = null;

		#stableConnectionTimer: NodeJS.Timeout \| null = null;

		this.#reconnectAttempts += 1;

		// Calculate delay with exponential backoff

Uh oh!

Uh oh!

Conversation

Kriys94 commented Oct 16, 2025• edited by cursor BotLoading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Explanation

Current State

The Problem

The Solution

1. Centralize Reconnection via scheduleConnect

2. Idempotent scheduleConnect

3. Jitter on Reconnection

Technical Details

References

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kriys94Oct 21, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mcmire left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kriys94Oct 22, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Kriys94 commented Oct 16, 2025•
edited by cursor Bot
Loading

1. Centralize Reconnection via `scheduleConnect`

2. Idempotent `scheduleConnect`

Kriys94Oct 21, 2025•
edited
Loading

Kriys94Oct 22, 2025•
edited
Loading