perf: make m_peer_mutex a SharedMutex

[PastaPastaPasta]

Issue being fixed or feature implemented

m_peer_mutex is a great use case for a shared mutex; it's private, well contained scope, and 99% of usages are for the read case. The only time we hold an exclusive lock is when adding or removing a peer, which happens rarely compared to all other usages.

On latest nightly, this mutex only represents about 2% of contention (all in PeerRef PeerManagerImpl::GetPeerRef), however, I intend to move over mnauth / verified protx related stuff over into PeerManager, and I expect that will result in significantly more contention over m_peer_mutex, making this change more valuable.

I expect this will reduce contention over this mutex to ~0

What was done?

How Has This Been Tested?

On testnet

Breaking Changes

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have added or updated relevant unit/integration/functional/e2e tests
• I have made corresponding changes to the documentation
• I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

[DashCoreAutoGuix]

Guix Automation has began to build this PR tagged as v22.1.0-devpr6468.fba3a04a. A new comment will be made when the image is pushed.

[DashCoreAutoGuix]

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v22.1.0-devpr6468.fba3a04a. The image should be on dockerhub soon.

[github-actions]

✅ No Merge Conflicts Detected

This PR currently has no conflicts with other open PRs.

[DashCoreAutoGuix]

Guix Automation has began to build this PR tagged as v23.0.0-devpr6468.f0445074. A new comment will be made when the image is pushed.

[PastaPastaPasta]

Below is the aggregated contention data I collected on masternode-1 on testnet during 2 minutes of ~ 8tps, so lots of tx locking happening. As you can see, m_peer_mutex does have significant contention over it.

Lock Name	Location	Count	Total (μs)	Avg (μs)	Max (μs)	Nodes
connman.m_nodes_mutex	net.cpp:4807	4342	948287	218.4	38087	1
m_peer_mutex	net_processing.cpp:1731	920	258870	281.4	13418	1
m_nodes_mutex	net.cpp:2087	571	186587	326.8	38671	1
::cs_main	llmq/quorums.cpp:1211	513	280406	546.6	13249	1
m_nodes_mutex	./net.h:1365	397	230037	579.4	46523	1
cs_main	net_processing.cpp:5468	234	39956	170.8	6082	1
pnode->cs_vSend	net.cpp:2482	171	15629	91.4	5853	1
::cs_main	validation.cpp:5942	153	9540	62.4	882	1
::cs_main	llmq/quorums.cpp:523	85	23442	275.8	7872	1
mutexMsgProc	net.cpp:2651	75	3245	43.3	2215	1
cs_main	index/base.cpp:339	65	44940	691.4	8926	1
pnode->cs_vSend	net.cpp:4742	61	3933	64.5	1371	1
m_peer_mutex	net_processing.cpp:2476	61	7407	121.4	1272	1
m_nodes_mutex	net.cpp:3006	36	78091	2169.2	5097	1
m_send_mutex	net.cpp:1622	32	6337	198.0	5515	1
cs_main	net_processing.cpp:5917	31	6537	210.9	2726	1
m_nodes_mutex	net.cpp:2206	27	82	3.0	17	1
::cs_main	instantsend/instantsend.cpp:272	22	5858	266.3	1023	1
cs_db	instantsend/db.cpp:240	22	2790	126.8	493	1
cs_cache	spork.cpp:244	19	725	38.2	464	1

Also; when running a local functional test for testing is lock latency, I saw this contention (contention detected 563 times):

m_peer_mutex net_processing.cpp:1731 563

After this change, there is no contention over the lock.

[coderabbitai]

Walkthrough

This change replaces the exclusive Mutex protecting PeerManagerImpl::m_peer_map with a SharedMutex (mutable SharedMutex m_peer_mutex). Read-only accesses (including GetPeerRef and other iteration/read sites such as SendPings, AskPeersForTransaction, RelayInv, RelayDSQ, RelayRecoveredSig, RelayAddress, and relay-related paths) now acquire READ_LOCK(m_peer_mutex) instead of LOCK. Mutating operations that modify m_peer_map or related state remain guarded by exclusive LOCK. Overall control flow is preserved; the intent is to allow concurrent readers while keeping writers exclusive.

Sequence Diagram(s)

mermaid
sequenceDiagram
autonumber
actor Reader1 as Reader Thread A
actor Reader2 as Reader Thread B
actor Writer as Writer Thread
participant PeerMgr as PeerManagerImpl
participant Mutex as m_peer_mutex

Note over Reader1,Reader2: Old flow (exclusive lock)
Reader1->>Mutex: LOCK()
Mutex-->>Reader1: acquired (exclusive)
Reader1->>PeerMgr: read m_peer_map
Reader1->>Mutex: UNLOCK()
Reader2->>Mutex: LOCK()
Mutex-->>Reader2: acquired (exclusive)
Reader2->>PeerMgr: read m_peer_map
Reader2->>Mutex: UNLOCK()

Note over Reader1,Reader2,Writer: New flow (shared locks for readers)
Reader1->>Mutex: READ_LOCK()
Mutex-->>Reader1: acquired (shared)
Reader2->>Mutex: READ_LOCK()
Mutex-->>Reader2: acquired (shared)
Reader1->>PeerMgr: read m_peer_map
Reader2->>PeerMgr: read m_peer_map
Writer->>Mutex: LOCK()
Mutex-->>Writer: waits until readers release
Reader1->>Mutex: READ_UNLOCK()
Reader2->>Mutex: READ_UNLOCK()
Writer->>Mutex: acquired (exclusive)
Writer->>PeerMgr: modify m_peer_map
Writer->>Mutex: UNLOCK()

Note right of Mutex: Colored blocks — shared reads (green), exclusive writes (orange)

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

• Areas to double-check:
  • Sites converted to READ_LOCK (ensure none actually perform hidden mutations or call methods that mutate peer state).
  • GetPeerRef changes and any code paths that depend on upgrading a read lock to a write lock (no lock upgrades introduced).
  • Any iterator usage lifetimes while holding shared lock (ensure objects referenced remain valid for lock duration).

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title Check	✅ Passed	The PR title "perf: make m_peer_mutex a SharedMutex" directly and specifically describes the main change in the changeset, which is replacing the synchronization primitive guarding the peer map from a plain Mutex to a SharedMutex in PeerManagerImpl. The title is concise, clear, and unambiguous, making it immediately understandable to someone reviewing the repository history. It appropriately uses the "perf:" prefix to indicate this is a performance optimization.
Description Check	✅ Passed	The PR description is clearly related to the changeset, explaining that m_peer_mutex is well-suited to be a shared mutex because 99% of usages are read-only and exclusive locks occur only rarely when adding or removing peers. It provides concrete motivation for the change, noting that the mutex currently represents about 2% of contention on the latest nightly and that future planned changes to move mnauth/verified protx functionality into PeerManager will increase contention further. The description directly addresses the optimization being implemented in the changeset.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

src/net_processing.cpp (1)

2515-2532: Fix lifetime hazard: raw Peer used after releasing m_peer_mutex.*

You store Peer* in best[...] under a shared lock, release the lock, then call PushAddress(*best[i].second, …). If a peer is removed concurrently, this can become a dangling pointer (UAF). Capture PeerRef instead to keep the object alive post-unlock.

Apply this diff:

- std::array<std::pair<uint64_t, Peer*>, 2> best{{{0, nullptr}, {0, nullptr}}};+ std::array<std::pair<uint64_t, PeerRef>, 2> best{{{0, {}}, {0, {}}}};
@@
- best[i] = std::make_pair(hashKey, peer.get());+ best[i] = std::make_pair(hashKey, peer);
@@
- for (unsigned int i = 0; i < nRelayNodes && best[i].first != 0; i++) {- PushAddress(*best[i].second, addr, insecure_rand);- }+ for (unsigned int i = 0; i < nRelayNodes && best[i].first != 0; i++) {+ PushAddress(*best[i].second, addr, insecure_rand);+ }

This mirrors the safe pattern used elsewhere (e.g., collecting PeerRefs in AskPeersForTransaction) and removes the UAF risk.

🧹 Nitpick comments (1)

src/net_processing.cpp (1)

2370-2375: Shrink critical section to reduce contention (optional).

Current loops hold m_peer_mutex (shared) while acquiring per‑peer locks and doing work. Consider:

• Collect std::vector<PeerRef> under READ_LOCK, then release m_peer_mutex and process each peer (call GetInvRelay(), PushInv(), etc.).
• This reduces time readers hold m_peer_mutex and lets writers (insert/remove) proceed sooner.

Also applies to: 2463-2471, 2476-2482

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between e5ce1f0 and f0445074a18e4961f3be8880dbc8a7268bcd1f4e.

📒 Files selected for processing (1)

• src/net_processing.cpp (9 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

src/**/*.{cpp,h,cc,cxx,hpp}

📄 CodeRabbit inference engine (CLAUDE.md)

src/**/*.{cpp,h,cc,cxx,hpp}: Dash Core C++ codebase must be written in C++20 and require at least Clang 16 or GCC 11.1
Dash uses unordered_lru_cache for efficient caching with LRU eviction

Files:

• src/net_processing.cpp

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (4)

• GitHub Check: win64-build / Build source
• GitHub Check: Lint / Run linters
• GitHub Check: arm-linux-gnueabihf / Build depends
• GitHub Check: x86_64-pc-linux-gnu / Build depends

🔇 Additional comments (4)

src/net_processing.cpp (4)

1731-1734: GetPeerRef under shared lock — LGTM.

Read-only map lookup with READ_LOCK(m_peer_mutex) is correct; returns PeerRef to extend lifetime post-unlock.

---

2304-2306: SendPings: shared lock + atomic write — LGTM.

Iterating under READ_LOCK and setting std::atomic<bool> m_ping_queued is safe.

---

2314-2324: AskPeersForTransaction: shared lock to collect PeerRefs — LGTM.

Collecting PeerRefs under READ_LOCK then acting under cs_main avoids holding m_peer_mutex across downstream locks.

---

2382-2395: RelayDSQ: shared lock usage — LGTM.

Read-only iteration and deferring sends via nodes_send_all after unlocking is sound.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

SharedMutex swap looks good; confirm macro support/fairness.

Change to SharedMutex is appropriate for read-heavy access. Please confirm:

• READ_LOCK(...) is defined by our sync wrappers and mapped to a shared-lock type.
• No writer starvation policy surprises on our platforms (clang/gcc toolchains).

---

🏁 Script executed:

#!/bin/bash# Ensure shared-lock macros/types exist and are used consistently
rg -nP '(^|[^A-Z_])READ_LOCK\s*\(' --type=cpp
rg -nP 'class|using\s+SharedMutex|struct\s+SharedMutex' --type=cpp
rg -nP 'LOCK_SHARED\s*\(' --type=cpp

Length of output: 205545

---

I can confirm the macro support is in place. Now I need to verify the actual code at the flagged lines to assess the lifetime hazard mentioned in the scratchpad.

I need to verify the actual code at the flagged lines to assess the lifetime hazard mentioned in the scratchpad.

---

🏁 Script executed:

#!/bin/bash# Inspect RelayAddress and _RelayTransaction code context for potential UAF issuesecho"=== RelayAddress context (line 2515-2528) ===" sed -n '2510,2535p' src/net_processing.cpp
echo""echo"=== _RelayTransaction context (line 2463-2471) ===" sed -n '2458,2480p' src/net_processing.cpp
echo""echo"=== Check if PeerRef is defined/used ===" rg -n "struct PeerRef|class PeerRef|using PeerRef" src/net_processing.cpp

Length of output: 1797

---

🏁 Script executed:

#!/bin/bash# Inspect RelayAddress and _RelayTransaction code context for potential UAF issuesecho"=== RelayAddress context (line 2515-2528) ===" sed -n '2510,2535p' src/net_processing.cpp
echo""echo"=== _RelayTransaction context (line 2463-2471) ===" sed -n '2458,2480p' src/net_processing.cpp
echo""echo"=== Check if PeerRef is used and defined ===" rg -n "PeerRef|using.*Ref" src/net_processing.cpp | head -20

Length of output: 2935

---

Fix the lifetime hazard in RelayAddress where raw peer pointers are used after releasing the lock.

The READ_LOCK(...) macro is properly supported (defined at src/sync.h:343 and mapped to ReadLock<decltype(cs)> on std::shared_mutex). However, RelayAddress stores raw Peer* pointers in the best array at line 2522, then uses them after the lock scope ends at line 2532. If a peer is removed from m_peer_map between lock release and the PushAddress call, this becomes a use-after-free.

Lines 2515–2528 (RelayAddress):
Capture PeerRef (shared_ptr, line 439) instead of raw pointers to extend peer lifetime beyond the lock. Replace peer.get() with the shared_ptr directly, or collect into std::vector<PeerRef> like AskPeersForTransaction does (line 2310).

The other READ_LOCK sites (_RelayTransaction, RelayRecoveredSig) are safe because all peer access occurs within the lock scope.

🤖 Prompt for AI Agents

src/net_processing.cpp lines 2515-2528: RelayAddress currently collects raw
Peer* pointers into `best` while holding the READ_LOCK, then uses them after the
lock is released causing a potential use-after-free; change the collection to
capture PeerRef (the shared_ptr alias used for peers) instead of raw pointers
(e.g., gather a std::vector<PeerRef> while inside the READ_LOCK), replace later
uses of `peer.get()` with the stored PeerRef, and then call PushAddress/other
post-lock APIs using the preserved PeerRef so peers remain alive after the lock
is released.

[PastaPastaPasta]

Please look at this again;

For the first case, it appears to me the READ_LOCK is held through the destruction of the best array. I don't see a UAF.

[coderabbitai]

Oops, something went wrong! Please try again later. 🐰 💔

[PastaPastaPasta]

Please look at this again;

For the first case, it appears to me the READ_LOCK is held through the destruction of the best array. I don't see a UAF.

[coderabbitai]

@PastaPastaPasta You’re right — in RelayAddress the READ_LOCK(m_peer_mutex) is acquired in the function scope and remains held through both loops, including the PushAddress calls. With the shared lock held, m_peer_map cannot be modified by writers, so the Peer* captured into best stays valid during its use. No UAF there. My earlier concern was unfounded — thanks for the nudge.

Optional: if you ever want to reduce the time m_peer_mutex is held, you could collect PeerRef entries under the read lock, release it, then call PushAddress on those refs. Not necessary for correctness here, just a potential micro-optimization.

[DashCoreAutoGuix]

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v23.0.0-devpr6468.f0445074. The image should be on dockerhub soon.

[kwvg]

utACK f0445074a18e4961f3be8880dbc8a7268bcd1f4e

[github-actions]

This pull request has conflicts, please rebase.

[knst]

utACK f0445074a18e4961f3be8880dbc8a7268bcd1f4e (need to rebase)

[knst]

re-utACK 43fd60f

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (6)

src/net_processing.cpp (6)

2293-2298: SendPings: read-lock + atomic write is safe; consider shortening lock scope.

Optional: collect PeerRef under the read lock, then set m_ping_queued outside to minimize time holding m_peer_mutex.

- READ_LOCK(m_peer_mutex);- for(auto& it : m_peer_map) it.second->m_ping_queued = true;+ std::vector<PeerRef> peers;+ {+ READ_LOCK(m_peer_mutex);+ peers.reserve(m_peer_map.size());+ for (const auto& [_, peer] : m_peer_map) peers.emplace_back(peer);+ }+ for (const auto& peer : peers) {+ peer->m_ping_queued = true;+ }

---

2361-2366: RelayInv: avoid holding m_peer_mutex while pushing to peers.

Optional: mirror the AskPeersForTransaction pattern to shorten lock time.

- READ_LOCK(m_peer_mutex);- for (const auto& [_, peer] : m_peer_map) {- if (!peer->GetInvRelay()) continue;- PushInv(*peer, inv);- }+ std::vector<PeerRef> peers;+ {+ READ_LOCK(m_peer_mutex);+ peers.reserve(m_peer_map.size());+ for (const auto& [_, peer] : m_peer_map) peers.emplace_back(peer);+ }+ for (const auto& peer : peers) {+ if (!peer->GetInvRelay()) continue;+ PushInv(*peer, inv);+ }

---

2370-2393: RelayDSQ: consistent with shared lock; optionally move INV path out of lock.

You already defer DSQ::ALL sends via NodeId vector. For DSQ::INV, consider collecting PeerRef under READ_LOCK, then PushInv after releasing it to further reduce contention.

---

2453-2464: _RelayTransaction: shorten lock scope while broadcasting.

Optional: collect PeerRef, then PushInv outside the READ_LOCK.

- READ_LOCK(m_peer_mutex);- for(auto& it : m_peer_map) {- Peer& peer = *it.second;- auto tx_relay = peer.GetTxRelay();- if (!tx_relay) continue;- PushInv(peer, inv);- };+ std::vector<PeerRef> peers;+ {+ READ_LOCK(m_peer_mutex);+ peers.reserve(m_peer_map.size());+ for (const auto& [_, peer] : m_peer_map) peers.emplace_back(peer);+ }+ for (const auto& peer : peers) {+ if (auto tx_relay = peer->GetTxRelay()) {+ PushInv(*peer, inv);+ }+ }

---

2468-2475: RelayRecoveredSig: same optional lock-narrowing.

Collect PeerRef under READ_LOCK, then PushInv after releasing it.

- READ_LOCK(m_peer_mutex);- for (const auto& [_, peer] : m_peer_map) {- if (peer->m_wants_recsigs) {- PushInv(*peer, inv);- }- }+ std::vector<PeerRef> peers;+ {+ READ_LOCK(m_peer_mutex);+ peers.reserve(m_peer_map.size());+ for (const auto& [_, peer] : m_peer_map) peers.emplace_back(peer);+ }+ for (const auto& peer : peers) {+ if (peer->m_wants_recsigs) {+ PushInv(*peer, inv);+ }+ }

---

2504-2525: RelayAddress: current approach is safe; optional lifetime-friendly tweak.

With READ_LOCK held across both loops, raw Peer* remains valid — no UAF. If you want to release m_peer_mutex before PushAddress, store PeerRef instead of raw pointers.

- std::array<std::pair<uint64_t, Peer*>, 2> best{{{0, nullptr}, {0, nullptr}}};+ std::array<std::pair<uint64_t, PeerRef>, 2> best{{{0, {}}, {0, {}}}};
@@
- READ_LOCK(m_peer_mutex);- for (auto& [id, peer] : m_peer_map) {+ {+ READ_LOCK(m_peer_mutex);+ for (const auto& [id, peer] : m_peer_map) {
@@
- best[i] = std::make_pair(hashKey, peer.get());+ best[i] = std::make_pair(hashKey, peer);
@@
- };+ } // release m_peer_mutex
@@
- PushAddress(*best[i].second, addr, insecure_rand);+ PushAddress(*best[i].second, addr, insecure_rand);

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between f0445074a18e4961f3be8880dbc8a7268bcd1f4e and 43fd60f.

📒 Files selected for processing (1)

• src/net_processing.cpp (9 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

src/**/*.{cpp,h,cc,cxx,hpp}

📄 CodeRabbit inference engine (CLAUDE.md)

src/**/*.{cpp,h,cc,cxx,hpp}: Dash Core C++ codebase must be written in C++20 and require at least Clang 16 or GCC 11.1
Dash uses unordered_lru_cache for efficient caching with LRU eviction

Files:

• src/net_processing.cpp

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (9)

• GitHub Check: linux64-build / Build source
• GitHub Check: arm-linux-build / Build source
• GitHub Check: linux64_nowallet-build / Build source
• GitHub Check: x86_64-apple-darwin / Build depends
• GitHub Check: x86_64-pc-linux-gnu_multiprocess / Build depends
• GitHub Check: Lint / Run linters
• GitHub Check: x86_64-w64-mingw32 / Build depends
• GitHub Check: x86_64-pc-linux-gnu / Build depends
• GitHub Check: Lint / Run linters

🔇 Additional comments (3)

src/net_processing.cpp (3)

815-824: SharedMutex swap is correct for read-dominant access.

m_peer_map still uses exclusive LOCKs on mutation (InitializeNode/RemovePeer). An appropriate upgrade.

---

1724-1729: GetPeerRef now uses READ_LOCK — safe lifetime.

Returning PeerRef under a shared lock, then releasing, is correct per the comment that a shared_ptr keeps the peer alive.

---

2304-2315: AskPeersForTransaction: good pattern.

Collecting PeerRef under READ_LOCK and using them after releasing the lock is exactly the right approach here.

[kwvg]

utACK 43fd60f

[UdjinM6]

utACK 43fd60f