Replace KVStorePersister with KVStore

[tnull]

So far, the main persistence interface implemented by the user was the KVStorePersister which provided a number of blanket implementations for convenience. However, as we'll need additional functionalities soon (namely, the capability of removing previously persisted data items), this interface was rather limited.

Here, we upstream the KVStore interface with slight modifications from LDK Node and add the FilesystemStore implementation as part of a new lightning-storage crate that supersedeslightning-persister.

[codecov-commenter]

Codecov Report

Patch coverage: 76.19% and project coverage change: +1.12% 🎉

Comparison is base (073f078) 90.59% compared to head (af2bd74) 91.72%.
Report is 24 commits behind head on main.

❗ Current head af2bd74 differs from pull request most recent head 7a65671. Consider uploading reports for the commit 7a65671 to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@ Coverage Diff @@## main #2472 +/- ##
==========================================
+ Coverage 90.59% 91.72% +1.12% 
==========================================
Files 110 112 +2 Lines 57509 66839 +9330 Branches 57509 66839 +9330 ==========================================
+ Hits 52101 61307 +9206 - Misses 5408 5532 +124 

Files Changed	Coverage Δ
lightning/src/util/test_utils.rs	68.70% <0.00%> (-4.91%)	⬇️
lightning-persister/src/utils.rs	71.42% <71.42%> (ø)
lightning/src/util/persist.rs	84.78% <80.55%> (-15.22%)	⬇️
lightning-background-processor/src/lib.rs	81.41% <82.05%> (-0.91%)	⬇️
lightning-persister/src/fs_store.rs	87.71% <87.71%> (ø)
lightning-persister/src/lib.rs	100.00% <100.00%> (+10.83%)	⬆️
lightning-persister/src/test_utils.rs	100.00% <100.00%> (ø)

... and 13 files with indirect coverage changes

☔ View full report in Codecov by Sentry.

📢 Have feedback on the report? Share it here.

[tnull]

Ran a few bench iterations with n = 1000 sample size each to check there is no regression in write performance. Unsurprisingly (as FilesystemStore is in good part the same code as FilesystemPersister), I saw no significant change or at times even a slight improvement.

For example some results (Linux on Intel(R) Xeon(R) CPU E3-1226 v3 @ 3.30GHz):

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 235.35 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
time: [210.07 ms 217.72 ms 225.39 ms]
change: [-16.944% -12.610% -8.2165%] (p = 0.00 < 0.05)
Performance has improved.

Benchmarking bench_filesystem_persisted_sends: Collecting 1000 samples in estimated 114.87 s (1000 iterations)
Benchmarking bench_filesystem_persisted_sends: Analyzing
bench_filesystem_persisted_sends
time: [234.67 ms 243.50 ms 252.42 ms]
change: [-2.5607% +2.5055% +7.8026%] (p = 0.34 > 0.05)
No change in performance detected.

[tnull]

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

[tnull]

Rebased on main.

[tnull]

(Not exactly sure why the BP test is stuck on windows but no other platform, will need to setup a windows box to debug this further)

Windows issues are fixed, CI failure is #2438.

[tnull]

Alright, I now squashed the previous round of fixups and made the following changes:

• All KVStore operations now take a sub_namespace argument.
• KVStore::remove now takes a lazy flag that signals of backends lazy/eventual persistence of deletes is allowed.

[TheBlueMatt]

Yay, LGTM 🎉. A few minor comments left on the filesystem storage, specifically.

[TheBlueMatt]

I think basically LGTM, feel free to squash and let's land this!

[tnull]

I think basically LGTM, feel free to squash and let's land this!

Squashed fixups without further changes.

[G8XSU]

Overall LGTM!

[G8XSU]

I am not sure if we want to take responsibility for replaying removes.

Therefore, this flag should only be set for remove operations that are not necessary for functional correctness of application and will not have any side-effects if remove failed or is eventually executed.

[TheBlueMatt]

I don't see how the flag could be interpreted any other way? If its async and we crash, the thing that's storing stuff has to handle that.

[tnull]

I am not sure if we want to take responsibility for replaying removes.

But that's not what the comment is saying? It says that lazy should only be used if we're fine risking the remove getting lost, we may see removed keys in list, and we're okay re-removing them once more if that's the case, i.e., our logic won't get tripped by the described behavior.

[G8XSU]

we're fine risking the remove getting lost

I see that it is mentioned before, but this wasn't very clear in the "therefore" statement.

Also wanted to make sure that functional correctness of application doesn't depend on "eventually".

[tnull]

Uh, I struggle to come up with a case where a stronger guarantee would make something fail. In particular as 'eventually' means 'at any point in time from now', it includes 'right now, before the method returns'?

But more concretely I thought I had that covered by stating that this is optional for the backend implementation ("If the lazy flag is set to true, the backend implementation might choose to lazily remove the given key at some point in time after the method returns"). Let me know if something in particular is missing.

[G8XSU]

I think it is fine, I was more concerned about deletes getting lost altogether by backend implementation. (for lazy delete and not even executing eventually)

[tnull]

Yeah, they might get lost under certain circumstances if we crash before they are actually executed. I don't think we can get around that because if we require the backend to remember and persist the eventual deletes atomically we might as well just do them...

[G8XSU]

If its async and we crash, the thing that's storing stuff has to handle that.

Yes, ideally. Moreover, if it is async and backend forgets about it or lost them, it shouldn't matter to us(ldk) at all.

As long as we are on same page 👍

[G8XSU]

feel free to squash

[tnull]

feel free to squash

Squashed fixups and included the following changes (as I decided to move the key length to a pub const to be in line with the alphabet):

> git diff-tree -U2 09b9fa8c 9e89a53bdiff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index bf042ff9..ae83aa47 100644
--- a/lightning-persister/src/utils.rs+++ b/lightning-persister/src/utils.rs@@ -1,6 +1,5 @@-use lightning::util::persist::KVSTORE_NAMESPACE_KEY_ALPHABET;+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};
use lightning::util::string::PrintableString;
-const KVSTORE_MAX_KEY_LEN: usize = 120;
pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index d1f70e24..4f00d3de 100644
--- a/lightning/src/util/persist.rs+++ b/lightning/src/util/persist.rs@@ -32,4 +32,7 @@ use crate::util::ser::{ReadableArgs, Writeable};
pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";
+/// The maximum number of characters keys may have.+pub const KVSTORE_MAX_KEY_LEN: usize = 120;+
/// The namespace under which the [`ChannelManager`] will be persisted.
pub const CHANNEL_MANAGER_PERSISTENCE_NAMESPACE: &str = "";
@@ -66,6 +69,6 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
///
/// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than 120 characters. Empty namespaces and-/// sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces+/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,
/// `sub_namespace` is required to be empty, too. This means that concerns should always be
/// separated by namespace first, before sub-namespaces are used. While the number of namespaces

[tnull]

So, I grew paranoid about hitting the max path length limitations on windows and added a corresponding test case. Low and behold there of course was an issue, so I now included a fix (canonicalizating paths on Windows):

g> git diff-tree -U2 9e89a53b af2bd741
diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 7e5c2568..77c03a64 100644
--- a/lightning-persister/src/fs_store.rs+++ b/lightning-persister/src/fs_store.rs@@ -67,4 +67,26 @@ impl FilesystemStore {
}
}
++ fn get_dest_dir_path(&self, namespace: &str, sub_namespace: &str) -> std::io::Result<PathBuf> {+ let mut dest_dir_path = {+ #[cfg(target_os = "windows")]+ {+ let data_dir = self.data_dir.clone();+ fs::create_dir_all(data_dir.clone())?;+ fs::canonicalize(data_dir)?+ }+ #[cfg(not(target_os = "windows"))]+ {+ self.data_dir.clone()+ }+ };++ dest_dir_path.push(namespace);+ if !sub_namespace.is_empty() {+ dest_dir_path.push(sub_namespace);+ }++ Ok(dest_dir_path)+ }
}
@@ -73,9 +95,5 @@ impl KVStore for FilesystemStore {
check_namespace_key_validity(namespace, sub_namespace, Some(key), "read")?;
- let mut dest_file_path = self.data_dir.clone();- dest_file_path.push(namespace);- if !sub_namespace.is_empty() {- dest_file_path.push(sub_namespace);- }+ let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
dest_file_path.push(key);
@@ -100,9 +118,5 @@ impl KVStore for FilesystemStore {
check_namespace_key_validity(namespace, sub_namespace, Some(key), "write")?;
- let mut dest_file_path = self.data_dir.clone();- dest_file_path.push(namespace);- if !sub_namespace.is_empty() {- dest_file_path.push(sub_namespace);- }+ let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
dest_file_path.push(key);
@@ -191,9 +205,5 @@ impl KVStore for FilesystemStore {
check_namespace_key_validity(namespace, sub_namespace, Some(key), "remove")?;
- let mut dest_file_path = self.data_dir.clone();- dest_file_path.push(namespace);- if !sub_namespace.is_empty() {- dest_file_path.push(sub_namespace);- }+ let mut dest_file_path = self.get_dest_dir_path(namespace, sub_namespace)?;
dest_file_path.push(key);
@@ -284,10 +294,5 @@ impl KVStore for FilesystemStore {
check_namespace_key_validity(namespace, sub_namespace, None, "list")?;
- let mut prefixed_dest = self.data_dir.clone();- prefixed_dest.push(namespace);- if !sub_namespace.is_empty() {- prefixed_dest.push(sub_namespace);- }-+ let prefixed_dest = self.get_dest_dir_path(namespace, sub_namespace)?;
let mut keys = Vec::new();
diff --git a/lightning-persister/src/test_utils.rs b/lightning-persister/src/test_utils.rs
index 718b8ef1..91557500 100644
--- a/lightning-persister/src/test_utils.rs+++ b/lightning-persister/src/test_utils.rs@@ -1,3 +1,3 @@-use lightning::util::persist::{KVStore, read_channel_monitors};+use lightning::util::persist::{KVStore, KVSTORE_NAMESPACE_KEY_MAX_LEN, read_channel_monitors};
use lightning::ln::functional_test_utils::{connect_block, create_announced_chan_between_nodes,
create_chanmon_cfgs, create_dummy_block, create_network, create_node_cfgs, create_node_chanmgrs,
@@ -39,4 +39,20 @@ pub(crate) fn do_read_write_remove_list_persist<K: KVStore + RefUnwindSafe>(kv_s
let listed_keys = kv_store.list(namespace, sub_namespace).unwrap();
assert_eq!(listed_keys.len(), 0);
++ // Ensure we have no issue operating with namespace/sub_namespace/key being KVSTORE_NAMESPACE_KEY_MAX_LEN+ let max_chars: String = std::iter::repeat('A').take(KVSTORE_NAMESPACE_KEY_MAX_LEN).collect();+ kv_store.write(&max_chars, &max_chars, &max_chars, &data).unwrap();++ let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();+ assert_eq!(listed_keys.len(), 1);+ assert_eq!(listed_keys[0], max_chars);++ let read_data = kv_store.read(&max_chars, &max_chars, &max_chars).unwrap();+ assert_eq!(data, &*read_data);++ kv_store.remove(&max_chars, &max_chars, &max_chars, false).unwrap();++ let listed_keys = kv_store.list(&max_chars, &max_chars).unwrap();+ assert_eq!(listed_keys.len(), 0);
}
diff --git a/lightning-persister/src/utils.rs b/lightning-persister/src/utils.rs
index ae83aa47..54ec230d 100644
--- a/lightning-persister/src/utils.rs+++ b/lightning-persister/src/utils.rs@@ -1,8 +1,8 @@-use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_MAX_KEY_LEN};+use lightning::util::persist::{KVSTORE_NAMESPACE_KEY_ALPHABET, KVSTORE_NAMESPACE_KEY_MAX_LEN};
use lightning::util::string::PrintableString;
pub(crate) fn is_valid_kvstore_str(key: &str) -> bool {
- key.len() <= KVSTORE_MAX_KEY_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))+ key.len() <= KVSTORE_NAMESPACE_KEY_MAX_LEN && key.chars().all(|c| KVSTORE_NAMESPACE_KEY_ALPHABET.contains(c))
}
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index 4f00d3de..c476edd7 100644
--- a/lightning/src/util/persist.rs+++ b/lightning/src/util/persist.rs@@ -32,6 +32,6 @@ use crate::util::ser::{ReadableArgs, Writeable};
pub const KVSTORE_NAMESPACE_KEY_ALPHABET: &str = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_-";
-/// The maximum number of characters keys may have.-pub const KVSTORE_MAX_KEY_LEN: usize = 120;+/// The maximum number of characters namespaces and keys may have.+pub const KVSTORE_NAMESPACE_KEY_MAX_LEN: usize = 120;
/// The namespace under which the [`ChannelManager`] will be persisted.
@@ -69,7 +69,7 @@ pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
///
/// Keys and namespaces are required to be valid ASCII strings in the range of
-/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_MAX_KEY_LEN`]. Empty namespaces-/// and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is empty,-/// `sub_namespace` is required to be empty, too. This means that concerns should always be+/// [`KVSTORE_NAMESPACE_KEY_ALPHABET`] and no longer than [`KVSTORE_NAMESPACE_KEY_MAX_LEN`]. Empty+/// namespaces and sub-namespaces (`""`) are assumed to be a valid, however, if `namespace` is+/// empty, `sub_namespace` is required to be empty, too. This means that concerns should always be
/// separated by namespace first, before sub-namespaces are used. While the number of namespaces
/// will be relatively small and is determined at compile time, there may be many sub-namespaces

[tnull]

Addressed the feedback and force-pushed with the following changes:

> git diff-tree -U2 af2bd741 7a656719diff --git a/lightning-persister/src/fs_store.rs b/lightning-persister/src/fs_store.rs
index 77c03a64..56d071da 100644
--- a/lightning-persister/src/fs_store.rs+++ b/lightning-persister/src/fs_store.rs@@ -240,5 +240,5 @@ impl KVStore for FilesystemStore {
//
// In order to assert we permanently removed the file in question we therefore
- // call `fsync` on the parent directory on platforms that support it,+ // call `fsync` on the parent directory on platforms that support it.
dir_file.sync_all()?;
}
@@ -328,5 +328,5 @@ impl KVStore for FilesystemStore {
// If we otherwise don't find a file at the given path something went wrong.
if !metadata.is_file() {
- debug_assert!(false, "Failed to list keys of {}/{}: file coulnd't be accessed.",+ debug_assert!(false, "Failed to list keys of {}/{}: file couldn't be accessed.",
PrintableString(namespace), PrintableString(sub_namespace));
let msg = format!("Failed to list keys of {}/{}: file couldn't be accessed.",
diff --git a/lightning/src/util/persist.rs b/lightning/src/util/persist.rs
index c476edd7..ca0605c9 100644
--- a/lightning/src/util/persist.rs+++ b/lightning/src/util/persist.rs@@ -61,5 +61,5 @@ pub const SCORER_PERSISTENCE_SUB_NAMESPACE: &str = "";
pub const SCORER_PERSISTENCE_KEY: &str = "scorer";
-/// Provides an interface that allows to store and retrieve persisted values that are associated+/// Provides an interface that allows storage and retrieval of persisted values that are associated
/// with given keys.
///
@@ -147,5 +147,5 @@ impl<'a, A: KVStore, M: Deref, T: Deref, ES: Deref, NS: Deref, SP: Deref, F: Der
L::Target: 'static + Logger,
{
- /// Persist the given ['ChannelManager'] to disk, returning an error if persistence failed.+ /// Persist the given [`ChannelManager`] to disk, returning an error if persistence failed.
fn persist_manager(&self, channel_manager: &ChannelManager<M, T, ES, NS, SP, F, R, L>) -> Result<(), io::Error> {
self.write(CHANNEL_MANAGER_PERSISTENCE_NAMESPACE,

[valentinewallace]

This seems like it's had a lot of detailed review already so I'll ack in case we want to merge today and take another glance tomorrow. Can't see there being any blocking feedback.

[G8XSU]

Ship it !

[TheBlueMatt]

One thing that needs addressing in a followup, but given the stack of PRs waiting, gonna go ahead and land this.