Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Paginate by creation time instead of key order by benthecarman · Pull Request #96 · lightningdevkit/vss-server · GitHub
Skip to content

Paginate by creation time instead of key order - #96

Merged
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination
May 5, 2026
Merged

Paginate by creation time instead of key order#96
tnull merged 1 commit into
lightningdevkit:mainfrom
benthecarman:creation-pagination

Conversation

@benthecarman

Copy link
Copy Markdown
Contributor

Clients often need to fetch recent payments or entries without iterating over the entire keyspace. Ordering by created_at lets callers retrieve the newest records first and stop early, which is not possible with lexicographic key ordering.

The page token now encodes (created_at, key) so the cursor remains unique even when multiple rows share the same timestamp. A composite index on (user_token, store_id, created_at, key) keeps the query efficient, and a migration back-fills any NULL created_at values and adds the NOT NULL constraint.

@ldk-reviews-bot

ldk-reviews-bot commented Apr 2, 2026

Copy link
Copy Markdown

👋 Thanks for assigning @tankyleo as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks !

Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/migrations.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/impls/src/postgres_store.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll want to update types.rs to require this ordering too

Comment threadrust/impls/src/postgres_store.rs Outdated
@benthecarman
benthecarmanforce-pushed the creation-pagination branch 2 times, most recently from dc205c9 to cd32024CompareApril 4, 2026 18:18
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

responded to review

Comment threadproto/vss.proto Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadproto/vss.proto Outdated
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (oldest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm here, the server is free to choose any ordering they would like in case two keys have the same timestamp.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have the server sort by key after timestamp. I'll add that here

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I don't this is a requirement on the VSS API right ? Ie clients don't care what ordering is picked beyond newest first.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better to me more explicit than less. We have to define a tiebreaker anyways so may as well put it in the docs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let me see this defines an API constraint right, in addition to just documentation ? For sure I see the use for documentation here, but I don't want clients to start relying on this behavior.

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

tankyleo commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

please request review when you are ready so i don't miss this PR

Comment threadproto/vss.proto
// Use this value to query for next-page of paginated `ListKeyVersions` operation, by specifying
// this value as the `page_token` in the next request.
//
// If `next_page_token` is empty (""), then the "last page" of results has been processed and

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I think we were intentionally following protobuf's/Google's best practices here. https://google.aip.dev/158 states:

  • Response messages for collections should define a string next_page_token field, providing the user with a page token that may be used to retrieve the next page.
    • The field containing pagination results should be the first field in the message and have a field number of 1. It should be a repeated field containing a list of resources constituting a single page of results.
    • If the end of the collection has been reached, the next_page_token field must be empty. This is the only way to communicate "end-of-collection" to users.
    • If the end of the collection has not been reached (or if the API can not determine in time), the API must provide a next_page_token.

IMO would be good to revert and include this context in the docs.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay updated

@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rushed some comments out, will be back after lunch.

Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
Comment threadrust/api/src/kv_store_tests.rs Outdated
}

let first_page = ctx.list(None, Some(page_size), None).await?;
assert_eq!(first_page.key_versions.len(), page_size as usize);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the VSS API, the page could have length 0 here. As long as the page token is not empty, the client would be expected to make another request with the new page token.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm confused what you're saying here. This is the first page we are requesting, the only way we'd a length of 0 would be if we had no items.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just that I think a VSS-server is within bounds if it returns a response with an empty list, but a non-empty token. The client would be expected to continue asking for pages.

I'm mostly going from this line in the docs of ListKeyVersionRequest:

 /// `page_size` is used by clients to specify the maximum number of results that can be returned by
/// the server.
/// The server may further constrain the maximum number of results returned in a single page.
/// If the `page_size` is 0 or not set, the server will decide the number of results to be returned.
#[prost(int32, optional, tag = "3")]
pub page_size: ::core::option::Option<i32>,

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TLDR can't assume the page you get back is the same length as the page size in your ListKeyVersionsRequest

This is true.

if it returns a response with an empty list, but a non-empty token.

But there is no reason to respond with empty list and non-empty pagination-token. (I don't think this should happen, can add a kvstore test/assert for it if doesn't exist already.)

Comment threadrust/impls/src/postgres_store.rs
@tankyleo

Copy link
Copy Markdown
Contributor

Done with this pass here, just needed to add the comment about the VSS server API constraint

@G8XSU

G8XSU commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

Adding some historical context, VSS was purposefully built to be storage engine agnostic. The VSS protocol/API/data-model doesn't enforce a specific underlying storage database.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

I understand the need to efficiently paginate through large result sets. A couple of alternatives that preserve storage engine agnosticism:

  • Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
    via lexicographic sort on the key itself.
  • Clients could scope their list_key_versions calls to known key prefixes to keep result sets
    manageable.
  • Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
    requiring timestamps.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Fixed @tankyleo comments about tests

@tankyleo

tankyleo commented Apr 9, 2026

Copy link
Copy Markdown
Contributor

@G8XSU Thank you for the feedback, I'll be considering the tradeoffs over the next few days.

Was wondering do you have an email where I can reach you ? Feel free to ping me at "hello at leonash dot net" I was wondering if we could get the vss-client crate at https://crates.io/crates/vss-client from you.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@G8XSU my perspective

Clients could use ordered key IDs like UUIDv7, which gives you time-based ordering for free
via lexicographic sort on the key itself.

in theory, yes, but ldk-node isn't using this and migrating would be a huge lift

Clients could scope their list_key_versions calls to known key prefixes to keep result sets
manageable.

ldk-node uses txid/payment_hash as key so we can't really use prefixes

Cursor-based pagination on the key (lexicographic ordering) works on every KV store without
requiring timestamps.

Yes, but then the pagination isn't really useful anymore. You just end up getting a random set of payments rather an actual ordered list.

We can totally revisit this requirement and say VSS only works with postgres-like relational databases. But extending pagination to order by creation_time limits the capability to use any modern pure KV store as an underlying database engine like DynamoDB or CosmosDB.

imo postgres scales really well and you're scaling past something like postgres then it's likely you aren't using the off the shelf solution anymore and will have a whole team to manage things like stuff. And being able to sort by something useful rather than just lexicographical is worth it

Comment threadrust/api/src/types.rs Outdated

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Right, but if it's not part of the API guarantees, isn't it odd to lean on/assume that the client only wants global version. At the very least the comment is misleading, IMO.

Let's just address this comment from tnull thank you, perhaps something like "page 0 means we get to decide"

@tankyleo
tankyleo requested a review from tnullApril 15, 2026 17:46
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Updated the comment about 0 page size

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally fine by me I think, one comment.

Aside from that, I do however wonder if we should finally add a protocol version to VSS? While this is not an API breaking change, clients will have no good idea whether they can expect pagination to work when they connect to a particular backend?

@tankyleo Any thoughts?

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remind me, is there any particular reason we need to use time for this? Wouldn't we get around the need for a tie breaker if we'd simply use a monotonically increasing atomic counter instead (e.g., a Postgres BIGSERIAL column) ? Then we'd be certain than each entry has a unique value?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude:

 Monotonic counter vs. creation_time
Your intuition is right — a monotonically increasing counter (e.g., a Postgres BIGSERIAL column) would be strictly simpler here. The current
approach has to deal with:
- Tie-breaking: The compound condition (created_at < $3 OR (created_at = $3 AND key > $4)) in the SQL query
- A composite page token: Encoding both the timestamp and the key (0:<micros>:<key>)
- A composite index: (user_token, store_id, created_at DESC, key ASC) INCLUDE (version)
- A dedicated test just for the tie-breaking behavior
With an auto-incrementing counter (say row_id BIGSERIAL), all of that collapses to:
- WHERE row_id < $cursor ORDER BY row_id DESC LIMIT $N
- Page token = just the counter value
- Simple single-column addition to the index
- No tie-breaking needed since values are unique by definition
The only argument for creation_time would be if it carried semantic meaning the client cares about (e.g., "show me keys created after X"). But
looking at the ListKeyVersionsResponse, created_at isn't exposed to the client — it's purely an internal ordering mechanism. So you're paying the
complexity tax of timestamps without getting the semantic benefit.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tnull I ping you on the response above in case it helps bubble this up in your inbox :)

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I didn't know we could use BIGSERIAL for non-primary keys (in sqlite you can't).

Yeah this might be better. On backwards compat, we still should be able to backfill the column by sorting by creation time, however, the migration that claude generated for this is pretty big/ugly.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you tnull for raising this point. I did some back-and-forth with claude, and yea this is interesting !

My one question at this point is backwards compat. Do you have thoughts on this point ? I'm thinking if keys created before this commit don't have strict creation order, this is OK. Perhaps we can use the VSS version you described above to encourage people to upgrade once we start relying on PaginatedKVStore in LDK Node.

@benthecarman let me know what you think.

I think the backfill should still handle it for the most part? But yeah, apart from that I think it might be good to lean on the versioning byte for this going forward, and make sure we publish vss-server v0.1 (with protocol versioning support) prior to LDK Node v0.8?

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

I do however wonder if we should finally add a protocol version to VSS?

Yeah i think this makes sense probably for a follow up tho

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Pushed to use the monotonic counter. Just did the migration by adding the column and letting postgres backfill it in scan order. Doing it ourselves by sorting by creation time got really ugly and after talking with @tankyleo, didn't seem worth it.

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM will take another pass tomorrow

@tankyleo
tankyleo requested a review from tnullApril 16, 2026 21:22
@ldk-reviews-bot

Copy link
Copy Markdown

🔔 1st Reminder

Hey @tnull! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good I think, but there are some follow-up changes to we should do, now that we don't use creation time anymore.

Comment threadproto/vss.proto
message ListKeyVersionsResponse {

// Fetched keys and versions.
// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These docs are now inaccurate, no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our implementation of this contract changed, but the contract remains the same, I think this is still accurate. Same for the other comments below.

Ok(())
}

async fn list_should_return_results_ordered_by_creation_time() -> Result<(), VssError> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here and below, the test names seems inaccurate now that we don't sort by time?

Comment threadrust/api/src/types.rs
#[derive(Clone, PartialEq, ::prost::Message)]
pub struct ListKeyVersionsResponse {
/// Fetched keys and versions.
/// Fetched keys and versions, ordered by creation time (newest first).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also inaccurate now.

Comment threadrust/impls/src/postgres_store.rs Outdated
const VERSION_COLUMN: &str = "version";
const SORT_ORDER_COLUMN: &str = "sort_order";

const CURRENT_PAGE_TOKEN_VERSION: char = '0';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think now that we don't have extra semantics, we should be good to drop the page token versioning byte and all associated logic again and simply use the sort_order as page token? (Especially given that we're discussing adding a protocol-level version, which we could also use if we'd ever find that we need to switch page token semantics again?)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to me yes given the upcoming protocol-level version

Comment threadREADME.md
VSS ships with a PostgreSQL implementation by default and can be hosted in your favorite infrastructure/cloud provider
(AWS/GCP) and its backend storage can be switched with some other implementation for KeyValueStore if needed.
(AWS/GCP). The backend storage can be switched with another implementation, but it must support ordering by creation
time, a simple key-value store is not sufficient.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't sort by creation time anymore. Might be good to be more accurate here, too.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

@tnull

tnull commented May 4, 2026

Copy link
Copy Markdown
Contributor

@tnull all your comments here are wrong it seems, we still do sort by creation time, we just use a counter in the db.

I guess it depends on how literal you want to take the term 'creation time'? Fine to leave the docs if you think the slight semantic difference doesn't matter, but IMO we should still drop the now-unnecessary semantics in the page token itself (#96 (comment))?

Clients often need to fetch recent entries without iterating over
the entire keyspace. Ordering by a monotonic insertion counter lets
callers retrieve the newest records first and stop early, which is
not possible with lexicographic key ordering.
A BIGSERIAL sort_order column is added to vss_db. New rows get a
monotonically increasing value from its sequence, and list queries
order by sort_order DESC. Because sort_order is UNIQUE, the page
token collapses to a single integer with no tiebreaker needed.
A composite index on (user_token, store_id, sort_order DESC)
INCLUDE (key, version) keeps list queries as index-only scans.
Pre-existing rows receive sequence values in heap-scan order
during the column rewrite, so their list ordering will not reflect
creation time; new rows onward do.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Okay removed version number from page token

@tankyleo
tankyleo self-requested a review May 4, 2026 15:24

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran this through gpt-5.5 xhigh, and it found these two things, sorry I didn't do this earlier.

  • Medium: next_page_token now exposes the raw global sort_order. Since sort_order is a global BIGSERIAL UNIQUE (rust/impls/src/migrations.rs:38) and the token is just sort_order.to_string() (rust/impls/src/postgres_store.rs:40), clients can infer service-wide write volume and gaps caused by other
    tenants. For a multi-user storage service, make the token opaque or use a per-user/store ordering value.
  • Low: negative page_size is still unvalidated, and this commit makes page_size = -1 silently return an empty successful page because fetch_limit becomes 0 (rust/impls/src/postgres_store.rs:688). Other negative values still become PostgreSQL limit errors. Reject page_size < 0 with
    InvalidRequestError before computing limit.

Curious what you think of the first one here, it's a good point I find.

@benthecarman

Copy link
Copy Markdown
ContributorAuthor

Curious what you think of the first one here, it's a good point I find.

I had similar thought but I don't really seem the harm. Maybe you can infer something about user activity but not really the end of the world. Doing a per user token doesn't totally work because the big serial is across the whole table, we could encrypt it or something but it is nice that the user can easily compare between 2 tokens

@tankyleotankyleo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good thanks again for the PR

@tnulltnull left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@tankyleo Will you pick up the protocol version follow-up?

@tnull
tnull merged commit 025b96a into lightningdevkit:mainMay 5, 2026
6 checks passed
@benthecarman
benthecarman deleted the creation-pagination branch May 5, 2026 10:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants

@benthecarman@ldk-reviews-bot@tankyleo@G8XSU@tnull