Add deduplication or stable IDs for user-facing events

[joostjager]

Problem

Without stable event ids in LDK, ldk-node has no reliable upstream identity to persist with app-facing events. A replayed LDK event can therefore look like a fresh event to ldk-node, which can lead to duplicate app-facing events after restart.

Payment events usually carry enough payment-level identifiers for careful applications to dedupe the highest-risk side effects, but that is an event-specific workaround rather than a general event identity contract. For non-payment events, applications may create duplicate notifications, accounting entries, operational actions, or fulfillment work if they treat ldk-node events as exactly-once.

Upstream and Remaining ldk-node Work

The upstream rust-lightning side is tracked in https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/issues/4794. That issue covers the missing public logical identity for replayed lightning::events::Events.

If LDK exposes a stable event identity or idempotency token, ldk-node should persist it, or an app-event identity derived from it, with the queued app-facing event. On replay, ldk-node should then recognize that the corresponding app event was already queued or handled instead of appending a duplicate.

ldk-node should also decide whether to expose this identity to applications. If app-facing events remain at-least-once, exposing a stable event id or idempotency key would make that contract explicit and give applications a reliable dedupe key.

[joostjager]

Thinking this issue should probably also cover the persistence ordering inside EventQueue itself.

Currently EventQueue::add_event and EventQueue::event_handled mutate the in-memory queue before writing the serialized queue to the store. If the store write fails, the durable queue and in-memory queue can diverge: add_event can leave an event visible in the current process even though it was not durably queued, while event_handled can remove an event from memory even though it remains in the durable queue and may reappear after restart.

So besides stable IDs / dedupe for replayed LDK events, there is a simpler queue consistency question here too: should EventQueue write the prospective queue first and only mutate the live in-memory queue after persistence succeeds?

[tnull]

We discussed this at some point upstream, and I think if we want to do this we should add idempotency tokens in the LDK events first.

@joostjager Are you fine with moving the issue there?

[joostjager]

Make sense. Will move the issue there. The question about queue consistency in ldk-node (write before in-memory mutate) is one thing that remains?

[tnull]

Make sense. Will move the issue there. The question about queue consistency in ldk-node (write before in-memory mutate) is one thing that remains?

I guess we could switch to that model (recently did the same for DataStore). To do this efficiently we might need to switch the event queue mutex over to be a tokio::sync::Mutex, which in turn requires any methods accessing the queue to be async, would need to explore if that works out.

[joostjager]

I think today’s mechanism already has a similar ordering issue? It mutates the in-memory queue, takes a snapshot, but then releases the lock before writing. So there can be out-of-order writes?

So perhaps tokio mutex is needed either way?

[joostjager]

Created upstream issue https://git.rust-bitcoin.org/lightningdevkit/rust-lightning/issues/4794 and updated body of this issue to describe the remaining work.

[comwanga]

I’ve been examining EventQueue::add_event and event_handled. I’d like to work on the ldk-node-local queue consistency portion discussed here, separately from upstream stable event IDs.

My proposed scope is to serialize queue mutations, persist prospective state before mutating the live queue, and add tests for failed and out-of-order writes. Would you prefer this under #967 or as a separate issue?

[tnull]

I’ve been examining EventQueue::add_event and event_handled. I’d like to work on the ldk-node-local queue consistency portion discussed here, separately from upstream stable event IDs.

Hmm, feel free to explore it in a draft, but I'm not quite sure if it's worth attempting before #926 lands.

[comwanga]

Sounds good, thanks. I'll explore it in a draft for now and wait for #926 to land before deciding whether the queue consistency changes are still worthwhile.