This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002

Closed
cheme wants to merge 85 commits into
paritytech:masterfrom
cheme:kv-statedb
Closed

state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom
cheme:kv-statedb

Conversation

@cheme

@chemecheme commented Nov 2, 2019

Copy link
Copy Markdown
Contributor

This PR extract the state-db logic from #3774 to try to make it more reviewable.
For general goal of key value storage, please refer to #3774.
This PR cover state-db additions.

As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.

  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.

Note that some pruning enhancement described in #3774 could be added.

@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others added 26 commits October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme added the A0-please_review Pull request needs code review. label Nov 2, 2019
@cheme

cheme commented Nov 11, 2019

Copy link
Copy Markdown
ContributorAuthor

Closing this PR because Child trie associated issue will be work around in a way that does not require this.
It make this a non prioritary thing.
Generally it adds complexity, and would require its own crate separation.
Some other part could maybe get merge.
Rethinking the design could also be interesting.

@chemecheme closed this Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

No reviews

Assignees

No one assigned

Labels

A0-please_reviewPull request needs code review.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@cheme
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content

This repository was archived by the owner on Nov 15, 2023. It is now read-only.

state-db key value storage - #4002
Closed
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Closed
state-db key value storage#4002
cheme wants to merge 85 commits into
paritytech:masterfrom 
cheme:kv-statedb
Conversation
@cheme
@chemecheme
commented
Nov 2, 2019

Copy link


Copy Markdown

Contributor
This PR extract the state-db logic from #3774 to try to make it more reviewable.

For general goal of key value storage, please refer to #3774.

This PR cover state-db additions.
As an extract from #3774, some TODO are present on state machine and client interfaces and tests are lacking.
  • dedicated journal (KvJournalRecord) are added (this could be merge in existing journal but focus here on non breaking change) to store block change, and pruning.
  • fork are handled with a sequential 'branch id' used as a per value index for memory historic of data (see historical-data tree).
    
The branch id are not persisted since this is state db.
  • pruning which is more a client problematic is handled in this pr in the state machine test over a test backend.
  • apply commit operation got new variants, because some key value content do not make sense.
Note that some pruning enhancement described in #3774 could be added.
@cheme
init from other pr, have some code up to client (not client db),
af6a8fd
complete some test method by providing child access to values.
@cheme
Dumb state_db implementation, mostly boiler plate, but cannot work as
4db0a7f
is.
Need to use historied values, maybe remove pinned, and put some context
as parameter for get_offstate
@cheme
TODO2 variant for in memory
19df6e9
@cheme
traits in place, still need to parameterized with blocknb &| branchix
3989fea
@cheme
passing block hash as state in a convoluted way
6a6606d
@cheme
passing block hash as state in a convoluted way
934c800
@cheme
Recover previous branch index code without persistence.
5d3d47e
@cheme
let's remove branch range set of changed and removed in next commit
aeab064
@cheme
removed delta in branch (not needed if no persistence)
57e862e
@cheme
Before using range instead of hash (we move range to upper client on
4edbf27
pin).
@cheme
use also block number for query (when not in statedb we need number).
91b87d7
@cheme
Fixing finalize state algo.
966a09f
@cheme
remove keys from commit set, this likely indicates broken code, will
a37a32f
probably need to store state, apply gc on state and depending on result
restore state or actually gc historied values.
@cheme
put a client test, for rest of test reset_storage and genesis build
4714b70
should get updated.
@cheme
Get needed component from branch history-data
5e90b20
@cheme
use history from utils
f6956b1
@cheme
Test issue is that query on remove hash do not work
9ba8057
@cheme
test commit set
31c1ee3
@cheme
fix unpin gc
6209f4d
@cheme
missing offstate prunnig handling
70f69b6
@cheme
previous pruning did not make sense, switch to only storing touched key
6803c86
for gc. Need to change commit set to contain an history of values and
adjust test db.
@cheme
Figure out pruning (see prune_one/prune_two tests).
2497575
@cheme
end fixing pruning test, remainings root lib tests to adjust
e86fdd2
@cheme
fix gc call block index.
e48cf43
@cheme
state_db tests passing.
0e3c65e
@cheme
fix underflow
13ea088
@cheme
before switching to absolute indexing (option<u64> to u64).
92c699c
@cheme
complete first test in client db, new todo.
bf79879
@cheme
Merge branch 'master' into chain-aux-merge
3913ba7
@cheme
serialized with default
4dc2c8d
chemeand others
added 26 commits
October 18, 2019 09:25
@cheme
Merge branch 'master' into chain-aux2
5ce414c
@cheme
Merge branch 'master' into chain-aux2
31400ae
@cheme
Merge branch 'master' into chain-aux2
2fcc675
@cheme
rename pairs to in_memory when it is not pairs but inmemory storage for
704178b
a state.
@cheme
clean client crates code, add a FullStorageContent struct, not
5b0f4c2
integrated into storage cache (no handling of kv there yet).
@cheme
cleaning in state-db
6dc9ef0
@cheme
end clean state_db
9ee40a7
@cheme
clean state-machine
b3e7d00
@cheme
renaming as in state-machine, and use only range
b642c69
@cheme
Move tree implementation state to test.
dd686c1
@cheme
Fix get on tree history (and iterator).
7713d8b
@cheme
Rename historied to historical
f0909fa
@cheme
A few renaming and documentation precision.
19f183c
@cheme
Switch test from Vec<u8> to Ser and removing Serialized buf mut variant.
4a834dd
@cheme
Merge branch 'master' into chain-aux2
b4f7805
@cheme
use copy_within
4268daf
@cheme
Remove truncate specific implementation.
397a77f
@cheme@gui1117
Apply documentation suggestions from code review
6caadf2
Co-Authored-By: thiolliere <gui.thiolliere@gmail.com>
@cheme
Remove into_owned function.
7410029
Break big line.
@cheme
Merge branch 'master' into chain-aux2
5e93dec
@cheme
strip down client an state machine
06790bf
@cheme
switch linear in state test to remove linear code from this pr.
914e1c1
@cheme
Merge branch 'master' into chain-aux2
979db3f
@cheme
Merge branch 'chain-aux2' into kv-statedb
90b3837
@cheme
Merge branch 'master' into chain-aux2
4ba47b5
@cheme
Merge branch 'chain-aux2' into kv-statedb
37ad244
@chemecheme
added
the 
A0-please_review
Pull request needs code review.
label
Nov 2, 2019
@cheme
cheme
commented
Nov 11, 2019

Copy link


Copy Markdown

ContributorAuthor
Closing this PR because Child trie associated issue will be work around in a way that does not require this.

It make this a non prioritary thing.

Generally it adds complexity, and would require its own crate separation.

Some other part could maybe get merge.

Rethinking the design could also be interesting.
@chemecheme
closed this
Nov 11, 2019
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.


Reviewers


No reviews


Assignees


No one assigned


Labels

A0-please_reviewPull request needs code review.

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

1 participant

@cheme