Implement "concentrated recovery" of LLMQ signatures - #3389

Merged
codablock merged 10 commits into
dashpay:developfrom
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged

Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom
codablock:pr_concentrated_recovery

Conversation

@codablock

@codablockcodablock commented Mar 30, 2020

Copy link
Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).

In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.

The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.

The new system is activated with the previously added spork21.

My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.

@codablockcodablock added RPC Some notable changes to RPC params/behaviour/descriptions P2P Some notable changes on p2p level backport-candidate-15.x labels Mar 30, 2020
@codablockcodablock added this to the 16 milestone Mar 30, 2020
@PastaPastaPasta

PastaPastaPasta commented Mar 30, 2020

Copy link
Copy Markdown
Member

Will review tonight or tomorrow

thephez
thephez reviewed Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp Outdated

@thephezthephezMar 31, 2020

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.

@codablockcodablockMar 31, 2020

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.

UdjinM6
UdjinM6 reviewed Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp Outdated
Comment threadsrc/llmq/quorums_signing_shares.cpp Outdated
PastaPastaPasta
PastaPastaPasta previously approved these changes Mar 31, 2020
View reviewed changes

@PastaPastaPastaPastaPastaPasta left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK

@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed the pr_concentrated_recovery branch from 8a6082d to 76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6 reviewed Apr 1, 2020
View reviewed changes

@UdjinM6UdjinM6 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.

(test failures seems to be random/unrelated, fails on develop too...)

Comment threadsrc/llmq/quorums_signing_shares.cpp Outdated
Comment threadtest/functional/llmq-signing.py Outdated
codablockand others added 2 commits April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6 approved these changes Apr 2, 2020
View reviewed changes

@UdjinM6UdjinM6 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@codablock
codablock merged commit c7b6eb8 into dashpay:developApr 2, 2020
@codablock
codablock deleted the pr_concentrated_recovery branch April 2, 2020 11:59
@UdjinM6UdjinM6 mentioned this pull request Feb 28, 2021
Merged
panleone pushed a commit to panleone/PIVX that referenced this pull request Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit to PIVX-Project/PIVX that referenced this pull request Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees

No one assigned

Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects

None yet

Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Implement "concentrated recovery" of LLMQ signatures - #3389
Merged
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Apr 2, 2020
Merged
Implement "concentrated recovery" of LLMQ signatures#3389
codablock merged 10 commits into
dashpay:developfrom 
codablock:pr_concentrated_recovery
Conversation
@codablock
@codablockcodablock
commented
Mar 30, 2020

Copy link


Copy Markdown

This implements "concentrated recovery" (I'm happy for suggestions of better names...).
In the current system, signature shares are propagated to all LLMQ members, until one of them has enough shares collected to recover the signature. Until this recovered signature is propagated in the LLMQ, all members will keep propagating shares and verifying each one. This causes quite some load on the LLMQ, which will be avoided with the new system.
The new system sends all shares to only a single deterministically selected node, so that this node can recover the signature and propagate the recovered signature. This way only the recovered signature needs to be propagated and verified by all members. After sending the share to this single node, each member waits for one second and repeats sending it to another deterministically selected member. This process is repeated until a recovered signature is finally created and propagated.
The new system is activated with the previously added spork21.
My assumption with this change is that it can reduce the load on a LLMQ by a factor of up to 40x-50x.
@codablockcodablock
added

RPC
Some notable changes to RPC params/behaviour/descriptions
P2P
Some notable changes on p2p level
backport-candidate-15.x

labels
Mar 30, 2020
@codablockcodablock
added this to the 16 milestone
Mar 30, 2020
@PastaPastaPasta
PastaPastaPasta
commented
Mar 30, 2020

Copy link


Copy Markdown

Member
Will review tonight or tomorrow
thephez
thephez
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/chainparams.cpp
Outdated

@thephezthephezMar 31, 2020

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was the number of .recoveryMembers chosen arbitrarily? I noticed the smaller quorums use a higher percent of the overall LLMQ size.
@codablockcodablockMar 31, 2020

Copy link


Copy Markdown

Author
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah...mostly arbitrary. I chose the numbers in an absolute way because there wouldn't be a good percentage that would work well on all LLMQ types.
UdjinM6
UdjinM6
reviewed
Mar 31, 2020
View reviewed changes
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

PastaPastaPasta
PastaPastaPasta
previously approved these changes
Mar 31, 2020
View reviewed changes
@PastaPastaPastaPastaPastaPasta
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, don't see any problems. utACK
@codablock
Rename sigSharesToSend to sigShareBatchesToSend
45064d8
@codablock
Implement new way of concentrated signature recovery
b212f21
Instead of propagating all sig shares to all LLMQ members, this will now
make all members send their individual sig share to a single member, which
is then responsible for the recovery and propagation of the recovered
signature. This process is repeated by all members every second for another
target/recovering member, until a recovered signature appears.
@codablock
Also run llmq-signing.py with spork21 enabled
5edbeaf
@codablock
Implement "quorum selectquorum" RPC
80533f6
@codablock
Implement DashTestFramework.get_mninfo
beaf3f4
@codablock
Test failure of recovery on first node
91f811e
@codablock
Use range based for loop in SelectMemberForRecovery
900ee0f
@codablock
Use auto keyword in CollectSigSharesToSend
76b6614
@codablock
codablockforce-pushed
the
pr_concentrated_recovery
branch
from
8a6082d to
76b6614CompareMarch 31, 2020 17:58
UdjinM6
UdjinM6
reviewed
Apr 1, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple more nits, otherwise looks good.
(test failures seems to be random/unrelated, fails on develop too...)
Comment threadsrc/llmq/quorums_signing_shares.cpp
Outdated

Comment threadtest/functional/llmq-signing.py
Outdated

codablockand others
added 2 commits
April 1, 2020 16:12
@codablock@UdjinM6
Only update id in spork21 case
47bd5d7
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
@codablock@UdjinM6
Use GetTime instead of GetAdjustedTime
148bbdd
Co-Authored-By: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6
UdjinM6
approved these changes
Apr 2, 2020
View reviewed changes
@UdjinM6UdjinM6
left a comment


Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK
@codablock
codablock
merged commit c7b6eb8
into
dashpay:developApr 2, 2020
@codablock
codablock
deleted the

pr_concentrated_recovery

branch
April 2, 2020 11:59
@UdjinM6UdjinM6
mentioned this pull request
Feb 28, 2021
 Merged

panleone pushed a commit
to panleone/PIVX
that referenced
this pull request
Nov 15, 2024
@codablock@panleone
Merge pull requestdashpay#3389from codablock/pr_concentrated_recovery
d942439
Implement "concentrated recovery" of LLMQ signatures
Fuzzbawls added a commit
to PIVX-Project/PIVX
that referenced
this pull request
Jan 11, 2025
@Fuzzbawls
Merge#2951: Dash LLMQ backports part 5
08eb5bb
a85b450 Merge pull request dashpay#3399 from codablock/pr_speedups2 (Alexander Block)
136f900 cherry-pick dashpay#2833 (Alexander Block)
d942439 Merge pull request dashpay#3389 from codablock/pr_concentrated_recovery (Alexander Block)
36790d2 cherry pick dashpay#3368 (Author Alexander Block)
dac01a9 cherry-pick dashpay#2780 (Alexander Block)
39d0ed9 cherry-pick dashpay#3367 (Alexander Block)
5084bbf Allow re-signing of IS locks when performing retroactive signing (dashpay#3219) (Alexander Block)
802c006 Only track last seen time instead of first and last seen time (dashpay#3165) (Alexander Block)
479b64b Avoid propagating InstantSend related old recovered sigs (dashpay#3145) (Alexander Block)
27fa2af cherry-pick dashpay#3117 (Pasta)
cf138e0 cherry-pick dashpay#3097 (Pasta)
23b140e Introduce getbestchainlock rpc and fix llmq-is-cl-conflicts.py (dashpay#3094) (UdjinM6)
Pull request description:
as usual each commit backports a different PR
ACKs for top commit: a85b450
Duddino:
utACK a85b450
Liquid369:
uTACK a85b450
Fuzzbawls:
ACK a85b450
Tree-SHA512: e9024d180888d8a6cc300ba9df74fc15929e3ade1773e5d312bd8cc93f6c9fd3898c5bf2d14672abf4faba576575c33936708e6e1dfd01a393479d264d3f2c57
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@thephezthephezthephez left review comments
@UdjinM6UdjinM6UdjinM6 approved these changes
@PastaPastaPastaPastaPastaPastaPastaPastaPasta left review comments

Assignees


No one assigned


Labels

P2PSome notable changes on p2p levelRPCSome notable changes to RPC params/behaviour/descriptions

Projects


None yet


Milestone

16

Development

Successfully merging this pull request may close these issues.

4 participants

@codablock@PastaPastaPasta@UdjinM6@thephez