Reconcile outbound payments between LDK state and `PaymentStore` on restart

[joostjager]

Problem

ldk-node stores outbound payment records independently from LDK's ChannelManager and ChannelMonitor persistence. That creates two restart-time split-brain cases: LDK can know about an outbound payment that is missing from ldk-node's PaymentStore, or PaymentStore can contain a Pending outbound payment that LDK no longer knows about.

The relevant BOLT11 send path is src/payment/bolt11.rs, Bolt11Payment::send_internal:

1. ldk-node derives the local PaymentId from the invoice payment hash with PaymentId(invoice.payment_hash().0).
2. It checks self.payment_store.get(&payment_id) and blocks only if the existing record is Pending or Succeeded.
3. It calls self.channel_manager.pay_for_bolt11_invoice(invoice, payment_id, amount_msat, optional_params).
4. Only after pay_for_bolt11_invoice returns Ok(()), it constructs a PaymentDetails with PaymentDirection::Outbound and PaymentStatus::Pending.
5. It persists that record with self.runtime.block_on(self.payment_store.insert(payment))?.

Because the LDK payment start and the ldk-node payment-record write are separate persistence streams, a crash or store error around this boundary can leave only one side durable.

On startup, src/builder.rs reads stored payment records with read_all_objects(... PAYMENT_INFO_PERSISTENCE_...) and builds PaymentStore::new(payments, ...). That reconstructs ldk-node's payment view from the ldk-node payment namespace. Node::payment, Node::list_payments, and Node::list_payments_with_filter in src/lib.rs all read from PaymentStore.

I did not find a startup pass that compares ChannelManager::list_recent_payments() against PaymentStore. Without that reconciliation, ldk-node appears to trust whichever side survived in the ldk-node payment namespace and does not repair disagreements with LDK's recent outbound payment state.

Consequence

If LDK knows about a payment but PaymentStore does not, the payment may be in flight or may later complete while remaining invisible to ldk-node payment APIs. This is primarily a payment-history/accounting gap rather than an LDK channel-fund-safety claim. It also feeds into duplicate-payment risk if duplicate protection later depends on a completed record that was never written.

If PaymentStore has a Pending outbound payment but LDK does not, the API can show a permanently stuck pending payment. The same local pending record can also block a retry because Bolt11Payment::send_internal treats Pending records as duplicates before calling LDK.

Related Open Issues

#171 is a partial duplicate. It asks to retry pending payments on restart and points at ChannelManager::list_recent_payments(), so it overlaps with the PaymentStore-pending-but-LDK-missing direction. It does not appear to cover the opposite direction, where LDK has a recent outbound payment but ldk-node has no PaymentStore record, and it does not spell out a general reconciliation contract between the two stores.

[Jolah1]

I'd like to work on this if it's open.

Before I start, two things from the surrounding threads that seem to bear on scope.
The two directions look to have very different exposure to the LDK/LDK Node boundary question. "PaymentStore has a Pending record, LDK doesn't know" overlaps #171, which is assigned and just slipped to v0.9, with the
responsibility split explicitly reopened. "LDK has a recent outbound payment, PaymentStore has no record" — the direction you noted #171 doesn't cover — looks like a self-contained LDK Node startup read: compare
ChannelManager::list_recent_payments() against PaymentStore and act on the difference. list_recent_payments() is currently called exactly once in the tree, in src/probing.rs:771, and nothing runs at startup.
The second question is what to do on divergence. On #968 you pushed back on reconstructing missing records as hacky, and that concern applies here too — repairing a missing PaymentStore entry from LDK's view is the same move.

So I'd propose splitting detection from repair: a startup reconciliation pass that detects and surfaces divergence (logging, and optionally an API to query it) without writing speculative records. That's useful on its own, it
doesn't prejudge the root-cause fix, and it gives the repair decision somewhere to land later once #1003 settles.

Does that split make sense, and is the LDK-knows/LDK-Node-doesn't direction something I can take independently while #171 and the #968 fix direction are pending? Happy to hold off entirely if you'd rather this wait on the persistence architecture discussion.