Add locking for safe node failover

[benthecarman]

We currently support postgres and VSS as remote persistence backends, which makes it possible to run LDK Node in environments where another process could restart the same node identity using the same persisted state.

The problem is that there is currently no ownership mechanism or locking for accessing the persisted state and we can't the 2 copies of the same node running at once. We need a way to ensure that exactly one process owns and mutates a node’s persisted state at a time, while allowing another instance to wait and take over quickly if the active instance crashes.

We should add a locking mechanism in our remote storage to allow for nodes to fail over to another instance if one crashes. Ideally this would also work for mobile use cases where if one mobile wallet is running, it stops another from spinning up until the other has closed.

[joostjager]

What is the ldk-node-specific part of this? Once remote persistence exists, doesn’t every serious LDK integration need an ownership or lease primitive to ensure only one process mutates a given node identity and persisted state at a time?

[joostjager]

I looked into locking in more detail. Nearly all of the lease logic belongs inside the store implementations. Since the PostgreSQL and VSS stores both live in ldk-node, this also seems like the sensible place to land these changes.

For rust-lightning specifically, we need to ensure that every critical external effect is preceded by the required persistence. It could also optionally preserve and propagate a specific lease-lost error, but that is not required for safety if the store freezes the node when it detects lease loss.

[joostjager]

I am thinking of the following scope for an initial minimal implementation:

A. Scope the first implementation to ldk-node’s built-in PostgreSQL store. VSS and generic user-provided KVStore implementations remain out of scope.

B. Use an ordinary PostgreSQL lease table rather than advisory locks. Each named KV table gets a companion lease table containing one singleton lease row. Multiple node identities can share a database when they use distinct KV table names.

C. Acquire the lease before deserializing any node state. A fallback that cannot acquire it does not construct a standby Node. Its supervisor retries later by constructing a fresh node.

D. Treat PostgreSQL time and the lease row as authoritative. The lease has a random per-process owner ID, is renewed periodically, and ownership loss is terminal for that in-memory Node.

E. Fence every PostgreSQL write and removal in one transaction. The transaction locks the lease row, validates ownership and expiry, renews the lease, performs the KV mutation, and commits. Reads remain unfenced.

F. Do not treat API-entry lease checks as a safety boundary. Leadership can be lost immediately after such a check. Only the fenced PostgreSQL mutation establishes authority for a critical operation.

G. Detect lease loss inside PostgresStore. Missing or expired ownership immediately marks the lease lost. Ambiguous database failures do so once the conservative local renewal deadline has elapsed. The store marks the node lost before returning an identifiable persistence error.

H. Treat lease loss as a one-way emergency shutdown. ldk-node freezes the node, stops background processing, and disconnects peers before notifying the application. Safety does not depend on callers correctly propagating the persistence error.

I. Preserve the identifiable persistence error as Error::NodeLeaseLost in higher layers. This mapping is diagnostic rather than the primary safety mechanism.

J. Keep process termination outside ldk-node. ldk-server waits for the lease-loss notification, exits without normal final persistence, and relies on its supervisor to restart it with freshly deserialized state.

K. Keep the lease while Node::stop() is active because the same Node may be restarted. A planned handoff requires stopping and dropping the node. Dropping it conditionally releases the lease on a best-effort basis.

L. Require every process accessing the node identity to use the lease-aware implementation. An older process that ignores the lease can still corrupt state.

M. Recognize the external-effect limitation. A PostgreSQL transaction cannot be atomic with a later Lightning peer message or Bitcoin transaction broadcast. A process can pause after the fenced commit and resume after another process has taken over.

N. Assume every critical external effect has an immediately preceding fenced persistence mutation and completes within the renewed lease interval. Protecting against arbitrary pauses would require fencing support from the external system itself, which Lightning peers and Bitcoin transaction broadcast generally do not provide.

O. Under that assumption, no rust-lightning lease API is required. rust-lightning only needs to guarantee that each critical external effect is preceded by the necessary durable mutation.

[TheBlueMatt]

Why does leasing logic belong in the store? If we assume some amount of synced clocks, you can just write a UUID and a lease-expiry timestamp into a leases K-V entry. Maybe the synced clocks assumption is too strong, but it'd be nice to do it all at a higher-level.

[joostjager]

I'd think it cannot be safe without doing it atomically. There might be a race when two standby nodes both read an expired lease and want to take over. And more generally, when processes can be suspended at any time, they may no longer have the lease when they do their writing if it's not a single op.

[joostjager]

@TheBlueMatt do you agree that it isn't safe without atomicity?

[TheBlueMatt]

There might be a race when two standby nodes both read an expired lease and want to take over.

Mmm, yea, I was thinking of VSS semantics and doing a RWR - as long as you're sure the write will only overwrite the state you read its fine, but that relies on the VSS versioning scheme (or a similar SQL restriction).

And more generally, when processes can be suspended at any time, they may no longer have the lease when they do their writing if it's not a single op.

If we assume synced clocks, you can gate any writes on the lease not expiring. I suppose there's a race before checking if you still have the lease and doing a VSS write, though.

Given we'd have to rely on some kind of VSS semantics anyway, seems reasonable to just require the "server-side" of the KVStore enforce locking.

[joostjager]

I’ll go ahead then with the atomic approach. For a minimally viable start, I might simplify two points from the earlier outline:

H. No separate in-memory frozen state. On lease loss, ldk-node stops background processing, disconnects peers, and notifies ldk-server, which immediately exits without normal shutdown.

I. No public Error::NodeLeaseLost variant. The failed operation continues to surface as an ordinary persistence failure, while the store-level fence and process termination provide the safety boundary.