Payment Persistence Gap Matrix

[joostjager]

This issue tracks crash and replay boundaries between LDK's persisted state and ldk-node's app-level payment records. It focuses on Lightning payments from both the sending and receiving node perspectives.

LDK persists ChannelManager, ChannelMonitors, and replayable LDK events. ldk-node separately persists PaymentDetails in PaymentStore and user-facing events in EventQueue. Because these are independent persistence streams, a crash or store error can leave the two views temporarily or permanently out of sync unless the event handling path or startup logic reconciles them.

Current main basis: 73be359ec5e8cddfc2463b8ed04bf2de038cdcd1.

PR is populated only when a currently open PR was identified as closing the scenario. Empty means no PR was identified yet.

Outbound

Scenario	Consequence	PR	Issue
LDK accepts or starts an outbound BOLT11 payment before ldk-node durably writes a Pending PaymentStore record.	The payment may be in flight or may later complete while absent from ldk-node payment APIs.		#969
ldk-node has a Pending outbound payment record after restart, but LDK no longer has corresponding recent payment state.	The payment may appear permanently pending, and local duplicate protection may block retry incorrectly.		#969, #171
PaymentSent or PaymentFailed update returns NotFound.	A success or failure event may be queued while the payment record stays missing. For success, this can also weaken duplicate-payment protection if no Succeeded record is reconstructed.		#968

Inbound

Scenario	Consequence	PR	Issue
BOLT11 invoice is created, but ldk-node crashes before the payment record is durably written and the invoice is never paid.	The invoice may remain payable but invisible to ldk-node.		#425, #51
BOLT11 auto-claim payment record is missing when PaymentClaimable or PaymentClaimed is handled.	Current main can claim or report the payment without backfilling durable payment history; #948 reconstructs the missing record from replayed LDK payment events.	#948	#298
Manual-claim BOLT11 pending registration is lost before its durable write.	The HTLC may not be surfaced for manual claim.		#425, #51, #280
BOLT12 or spontaneous inbound PaymentClaimable insert fails.	Debug builds panic at debug_assert!(false); non-debug builds log but do not request LDK replay, so funds may be claimed without a durable payment record. #948 fixes this path.	#948	#298, #381
PaymentClaimed update returns NotFound.	Current main logs and still queues PaymentReceived, so the app can see a received-payment event while payment history is missing. #948 reconstructs the missing record.	#948	#298
Repeated inbound payments share a payment hash, but ldk-node does not track the LDK event payment_id.	Distinct receives can overwrite or confuse each other. #948 tracks the LDK payment ID.	#948	#298, #472
Missing inbound payment records are not eagerly reconciled from LDK state on startup.	Missing records surface only if the relevant LDK events are replayed; there is no startup backfill pass.		#298

Cross-Cutting

Scenario	Consequence	PR	Issue
Startup restores PaymentStore by listing persisted keys, so stores that return only one page of keys require pagination to load every payment record.	Records beyond the first page can remain persisted but absent from the in-memory PaymentStore after startup.	#960
User-facing events derived from replayed LDK events have no stable app-event ID, event fingerprint, or queue-level dedupe.	ldk-node may deliver duplicate app events. Payment events carry useful identifiers for app-side dedupe, but duplicate delivery is still implicit API behavior rather than an enforced queue contract.		#967

[joostjager]

I thought it might be useful to keep an overview of where we are at with making the persistence layering safe. Analysis is obviously AI-executed. Looking for concept ack on the idea of a tracking issue for this class of problems.

[joostjager]

Somewhat separate from the payment-specific rows above, there are a few broader persistence/replay patterns elsewhere in ldk-node that may be worth tracking too:

• Some aggregate stores still mutate in-memory state before the durable write succeeds. EventQueue, PeerStore, node metrics, and the BDK wallet persister cache all have variants of this shape. That can leave the running process observing state that is ahead of the durable store after a write failure.
• Some important work is held in volatile in-memory queues. The transaction broadcaster queue and async onion-message mailbox are the two obvious examples. Recovery may be possible through later wallet sync, LDK rebroadcast, or LDK event replay, but the current ldk-node-level recovery contract is not very explicit.
• The on-chain wallet/payment boundary has similar split-stream questions as Lightning payments. For example, on-chain sends persist BDK wallet state and enqueue broadcast before ldk-node has a payment record, and RBF updates touch wallet state, broadcast state, and payment records in separate steps.
• Startup reconstruction via key listing is not payment-specific. Anywhere we load object namespaces with read_all_objects, paginated backing stores need the helper to consume every page, not just the first returned key set.
• Running-node persistence failures in LDK-managed state remain a broader process-recovery question, especially for remote persistence. That seems related to Address remaining panic-on-persistence failures #381 rather than something this payment matrix should try to fully cover.