kv: iterators

[barraguda]

Problem

"Get me all my keys in X db"
"Get me all keys that have the prefix "nodes:"

Solution

IterTypes Requests and Responses.

// get all users (returns Vec<(String, User)>)let all_users = kv.iter_all(None,100)?;for(key, user)in all_users {println!("Key: {}, User: {:?}", key, user);}// get users with prefix "user_" (batches of 50)let user_prefix = "user_".to_string();let filtered_users = kv.iter_all(Some(&user_prefix),50)?;// get all keys starting with "user_"let keys = kv.collect_keys(Some(&"user_".to_string()))?;for key in keys {println!("Found user key: {}", key);}// get all values (users)let users = kv.collect_values(None)?;

Docs Update

TODO
Corresponding docs PR

Notes

Could do a full iterator impl, but might be too complex with lifetimes and an extra struct, something like

pubstructKvIterator<'a,K,V>{kv:&'aKv<K,V>,// Reference to KV storeiterator_id:u64,// Server-side iterator IDbatch_size:u64,// How many items to fetch per batchcurrent_batch:Vec<(K,V)>,// Current batch of itemsdone:bool,// Whether we've reached the endprefix:Option<Vec<u8>>,// Optional prefix filter_marker:PhantomData<(K,V)>,// Required for generic types}impl<'a,K,V>IteratorforKvIterator<'a,K,V>whereK:Serialize + DeserializeOwned,V:Serialize + DeserializeOwned,{typeItem = anyhow::Result<(K,V)>;fnnext(&mutself) -> Option<Self::Item>{ifself.current_batch.is_empty() && !self.done{ifletErr(e) = self.fetch_next_batch(){returnSome(Err(e));}}self.current_batch.pop().map(Ok)}}// would allow for this:let iter = kv.iter(Some(&"user_"),100)?;for result in iter {let(key, value) = result?;println!("Key: {}, Value: {:?}", key, value);}

[nick1udwig]

It feels somewhat fake without an easy-to-call .next(), but given the infra added in the Kinode core PR, shouldn't be too crazy to add that method later. LGTM

[nick1udwig]

A fun thing we can do in future if we ever write a Wasm process_lib is package up this iterator in a resource