Avoid using ordered maps in LLMQ signing code by codablock · Pull Request #2708 · dashpay/dash

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

codablock · 2019-02-15T15:19:23Z

As profiling has shown, these get too expensive when they get larger. Better to use unordered maps whenever possible.

~~This PR currently also includes #2706 as it depends on the hashing for std::pair<Consensus::LLMQType, uint256>. I'll remove it when it got merged.~~

UdjinM6

LGTM, will re-review after rebase

codablock · 2019-02-15T15:54:08Z

@UdjinM6 not sure I understand...why LGTM? We first need to merge #2706.
With re-review, do you mean that you already did a review or do you plan to review after merge?

UdjinM6 · 2019-02-15T15:55:54Z

I mean I reviewed parts that are specific to this PR and they look good to me (LGTM) but I will formally review/ack this after rebase :)

codablock · 2019-02-15T16:04:50Z

Ahhh, so "LGTM" doesn't mean "Let's get this merged!" :D 🙈 Learned something new (it's "Looks good to me!") :)

codablock · 2019-02-16T14:54:32Z

Rebased and force-pushed

UdjinM6

utACK

I wonder if you have some benchmarks for "old code vs new code" 👀

UdjinM6 · 2019-02-16T17:00:14Z

needs rebase

The old implementation was relying on the maps being ordered, which allowed us to grab all sig shares for the same signHash by doing range queries on the map. This has the disadvantage of being unnecessarily slow when the maps get larger. Using an unordered map would be the naive solution, but then it's not possible to query by range anymore. The solution now is to have a specialized map "SigShareMap" which is indexed by "SigShareKey". It's internally just an unordered map, indexed by the sign hash and another unordered map for the value, indexed by the quorum member index.

These are faster when maps/sets get larger.

codablock · 2019-02-17T11:18:30Z

Rebased and force-pushed.

@UdjinM6 regarding benchmarks: I didn't do benchmarks in the usual way, as this code is too complex to isolate individual parts to benchmark. I did the following flow multiple times:

Start up a fresh devnet with 100 MNs (every node on it's own VM, by using a Kubernetes cluster with 100 nodes)
Let it run through at least 2 DKGs of LLMQ size 50 (so that there is some load balancing between nodes)
SSH connect to one of the VMs that has a member MN running. Install linux-perf on it.
Run a local script that connects to 50 MNs in parallel and let each one of them send out 20 TXs, so that we have a total 1000 TXs. As I'm using my local branch of LLMQ based InstantSend, this causes thousands of threshold signing sessions.
After a few seconds, when the LLMQs start to get overloaded, I run perf record --call-graph dwarf -e cpu-clock --pid XXX for a few seconds on the SSH connected VM. Important to let it run only a few seconds, as otherwise you end up with GBs of profiling data 🙈
Download the perf.data to my local machine and then run https://github.com/KDAB/hotspot on the data

The HotSpot tool shows so called "flame graphs", which give some very good hints on where most time is spent. It can filter per thread, you can zoom in, and so on. The dashd binary must be built with -g but it's ok to build with -O2 or -O3.

For these tests, the data usually shows (as expected) 2 threads which do the most work. The first is the message handler thread and the second one is the CSigSharesManager::WorkThreadMain thread.

My target was to reduce the amount of time spent for LLMQ messages in the message handler thread and move all the load to the other thread. The reason is, that processing one message from one peer might end up blocking receival/processing of many other messages from all other peers. Many optimizations in the LLMQ code assume that many signatures come in at once and can be batch verified in one go. If the message handler thread needs too much time per message, the work thread will end up verifying batches of size one too often. This is ok when the load is low, but gets worse and worse as higher the load becomes.

At the same time, I tried to reduce the overhead seen in the work thread. The assumption is that most time should be spent on signature (batch) verification and a little bit on signing. Everything else is overhead and should be reduced as much as possible.

This is for example all the time spent in book keeping of the sig shares, which IMHO should be minimal. The first rounds of the profiling have shown that actually a lot of time (~10%) was spent in std::map::find in various places. The first rounds have also shown that even more time (don't remember the percentage) was spent in LevelDB due to many calls to CRecoveredSigsDb::HasRecoveredSigForId and CRecoveredSigsDb::HasRecoveredSigForSession. These calls are done very often to ensure that sig shares are not processed unnecessarily.

The first rounds of profiling have also shown that PushMessage is loosing a lot (again ~10%) of time in send() calls, which kind of defeated the purpose of the worker thread (it should NOT waste time on waiting/blocking stuff!).

After all the optimizations, I'm pretty happy with how the flame graphs look now. Most of the time is spent in signature verification and signing, which is what I would expect. When everything is merged (including @UdjinM6's changes to intra quorum comms), I'll do another round of profiling and post some screenshots of the flame graphs.

UdjinM6

re-utACK

UPDATE: and thanks for a very informative explanation!

codablock · 2019-02-17T13:06:16Z

I locally reverted all optimizations and re-run one profiling session. This is how the flame graphs looked before and after the optimizations:

The message handler thread (before optimizations):

The message handler thread (with optimizations):

The relative reduction of time spent in CSigSharesManager::ProcessMessages is the most interesting part here.

The worker thread (before optimizations):

The worker thread (with optimizations):

Multiple interesting things here:

If you look at the time line (at the bottom), a lot of CPU time is wasted due to sleeping/idling
LevelDB was the largest overhead here. Unfortunetely this does not show the overhead of the ordered maps, which only gets visible when the cache optimization is included.
After the optimizations, most time is spent BLS code, like verification and signing. This is what one would expect/assume, as it is clearly the most expensive part of LLMQs.

UdjinM6 · 2019-02-17T13:17:10Z

Ohhh, nice :) 👍

PastaPastaPasta

post-utACK

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

UdjinM6 added this to the 14.0 milestone Feb 15, 2019

UdjinM6 reviewed Feb 15, 2019
View reviewed changes

codablock force-pushed the pr_llmq_optimizations5 branch from 3439377 to fc2367cCompare February 16, 2019 14:54

UdjinM6 previously approved these changes Feb 16, 2019
View reviewed changes

codablock added 3 commits February 17, 2019 11:50

Only use unordered maps/sets in CSigSharesManager
5823375
These are faster when maps/sets get larger.

Use unorderes sets/maps in CSigningManager
81bd958

codablock dismissed UdjinM6’s stale review via 81bd958February 17, 2019 10:51

codablock force-pushed the pr_llmq_optimizations5 branch from fc2367c to 81bd958Compare February 17, 2019 10:51

UdjinM6 approved these changes Feb 17, 2019
View reviewed changes

UdjinM6 merged commit 9f58690 into dashpay:developFeb 17, 2019

codablock deleted the pr_llmq_optimizations5 branch February 17, 2019 12:57

PastaPastaPasta reviewed Feb 20, 2019
View reviewed changes

Uh oh!

Conversation

codablock commented Feb 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

UdjinM6 left a comment

Choose a reason for hiding this comment

Uh oh!

codablock commented Feb 15, 2019

Uh oh!

UdjinM6 commented Feb 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codablock commented Feb 15, 2019

Uh oh!

codablock commented Feb 16, 2019

Uh oh!

UdjinM6 left a comment

Choose a reason for hiding this comment

Uh oh!

UdjinM6 commented Feb 16, 2019

Uh oh!

codablock commented Feb 17, 2019

Uh oh!

UdjinM6 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codablock commented Feb 17, 2019

Uh oh!

UdjinM6 commented Feb 17, 2019

Uh oh!

PastaPastaPasta left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

codablock commented Feb 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

UdjinM6 left a comment

Choose a reason for hiding this comment

Uh oh!

codablock commented Feb 15, 2019

Uh oh!

UdjinM6 commented Feb 15, 2019• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codablock commented Feb 15, 2019

Uh oh!

codablock commented Feb 16, 2019

Uh oh!

UdjinM6 left a comment

Choose a reason for hiding this comment

Uh oh!

UdjinM6 commented Feb 16, 2019

Uh oh!

codablock commented Feb 17, 2019

Uh oh!

UdjinM6 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codablock commented Feb 17, 2019

Uh oh!

UdjinM6 commented Feb 17, 2019

Uh oh!

PastaPastaPasta left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

codablock commented Feb 15, 2019•
edited
Loading

UdjinM6 commented Feb 15, 2019•
edited
Loading

UdjinM6 left a comment •
edited
Loading

codablock commented Feb 15, 2019•
edited
Loading

UdjinM6 commented Feb 15, 2019•
edited
Loading

UdjinM6 left a comment •
edited
Loading

codablock commented Feb 15, 2019•
edited
Loading

UdjinM6 commented Feb 15, 2019•
edited
Loading

UdjinM6 left a comment •
edited
Loading

codablock commented Feb 15, 2019•
edited
Loading