Changes tries build cache

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM

[svyatonik]

Problem: when digest tries are built, we need to read some previous changes tries.
Example: in 4^2 configuration, when block#16 is built, we need to read tries of blocks: 4, 8, 12, 13, 14, 15.
Proposed optimization: maintain in-memory HashMap<Block => Set<ChangedKeys>> && use it instead of reading tries directly from the database.

Some details:

• if entry for block is missing from the cache, we still read trie from database;
• the cache is cleared on node shutdown => there could be performance regression on restart;
• if there's a fork during digest CT creation, then the first imported block purges all used cache entries from the cache => the second block import will read everything from DB. It could be fixed if we would teach cache about finalization/etc, but it'll make it complex + memory demanding.

The cache itself isn't very useful when there's a plenty of memory dedicated to database cache + when top-level digests aren't covering large blocks ranges. E.g. block#512 in 512^1 configuration, that is normally imported in 0.3511s, with this optimization is imported in 0.2932s. Block#1024 in 1024^1 configuration, that is normally imported in 0.6961s, with this optimization is imported in 0.6047s. The difference isn't that big, but if we'll consider situation when there's lack of memory dedicated to database cache (i.e. when tries nodes are actually read from DB, not the rocks-db cache), then the situation seems much better with this optimization: 0.9514s -> 0.6167s.

[cheme]

Cache logic looks good.

I could not really assert what happen on fork (doc says it get reinit on first next CT insert, but I could not really find it).

[cheme]

LGTM