This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
This repository was archived by the owner on Nov 15, 2023. It is now read-only.

Add StorageList support in decl_storage - #2261

Closed
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list
Closed

Add StorageList support in decl_storage#2261
sorpaas wants to merge 3 commits into
masterfrom
sp-storage-list

Conversation

@sorpaas

Copy link
Copy Markdown
Contributor

Add back storage list support in decl_storage macro. Can fix issues like #2228.

A QoL function StorageList::push is also added.

@sorpaassorpaas added the A0-please_review Pull request needs code review. label Apr 12, 2019
@sorpaas
sorpaas requested a review from bkchrApril 12, 2019 01:06
Comment threadsrml/metadata/src/lib.rs Outdated
#[cfg_attr(feature = "std", derive(Decode, Debug, Serialize))]
pub enum StorageFunctionType {
Plain(DecodeDifferentStr),
List(DecodeDifferentStr),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please put this to the end to improve backward compatibility


/// Push a new item to the list.
fn push(item: &T);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no pop method? how do I delete the last item?

@bkchr

Copy link
Copy Markdown
Member

I thought we won't bring back storage list because of the performance issues. See here #772

@tomusdrw

Copy link
Copy Markdown
Contributor

We do have linked_map now that should be more efficient

@tomusdrw

tomusdrw commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

To clarify on my "more efficient" comment, after discussing that with @bkchr .
linked_map and StorageList have many similarities and in most cases the performance would be similar, so iterating over unbounded collection inside runtime is still not recommended.

However:

  1. linked_map offers arbitrary (sparse) keys (not only a consistent range of 0..len)
  2. That makes removals more efficient (no need to re-order) and also makes the key persistent, hence reliable

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

I see some use cases of storage list. Every data structure have its trade offs.
Storage list can be used as a stack, an append only storage (like events). It use less storage than linked map (which needs to store next and prev for every item).

I think Substrate should offer (secure and safe) options and guidances to help developers to make right decisions.

@bkchr

Copy link
Copy Markdown
Member

You will need to iterate events at the end of each block and that will create a lot of database accesses.

@xlc

xlc commented Apr 12, 2019

Copy link
Copy Markdown
Contributor

Why do I need to iterate events? To conver to Vec for compatibility reason? I believe polkadotjs is able to somehow detects the format and handle both cases just like metadata versions

@bkchr

Copy link
Copy Markdown
Member

Yeah, good point^^

@sorpaas

Copy link
Copy Markdown
ContributorAuthor

@bkchr

You will need to iterate events at the end of each block and that will create a lot of database accesses.

It does not necessarily need any database access. For something like Events, things are directly stored in the overlay. We only need to generate trie structure when storage_root is called, which usually happens after Events are killed.

@bkchr

Copy link
Copy Markdown
Member

Yeah I know that we don't always access the real database. Nevertheless, we still need to have 1 lookups + 2 writes for storing one event.

(events are also part of the storage root, as they stay for the client to inspect them)

@gavofyork

Copy link
Copy Markdown
Member

Indeed. Events are not free - the stick around until the next block and therefore bloat the trie and increase hashing required and then you'll need to iterate over all of the last block's events to delete them, increasing I/O.

@bkchr

Copy link
Copy Markdown
Member

Events bottleneck is also already fixed in master.

So how do we continue with this one? Close?

@sorpaassorpaas closed this Apr 19, 2019
@arkpar
arkpar deleted the sp-storage-list branch April 23, 2020 18:12
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@sorpaas@bkchr@tomusdrw@xlc@gavofyork@devops-parity