RATIS-1866. Maintain leader lease after AppendEntries

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!

[SzyWilliam]

What is a Leader Lease

In Raft, the leader is responsible for processing and coordinating client requests, replicating data among other followers, and maintaining the distributed state machine.
Vanilla Raft requires the leader to obtain majority acknowledgements before serving every read requests. During normal operations, this prerequisite leads to unnecessary rpcs exchanged among the cluster, diminished read throughput and increased latency.
The leader lease is a concept that allows the leader to maintain its leadership without obtaining majority acknowledgements for a certain period of time (lease duration), during which it can directly serve client read requests.

How to extend lease during normal operations

Prerequisite

• Suppose the CPU clocks are perfectly synchronized among cluster machines.
• Let each AppendEntries request AE(i) contain a send time T(i)

Initialize

Once a leader is elected and its authority being comfirmed by majorities through successfully replicating its first no-op log, the leader gains the lease. The lease validity starts from T(0).

Renewal

As long as the leader continues to send heartbeats and receives acknowledgments from a majority of other nodes, it can renew its lease. Theoretically, if the most recent acknowledged heartbeat was sent at time T(n), the validity of the new lease commences at T(n).
In practice, rather than updating the lease with every heartbeat, we opt for a more efficient approach by lazily updating the leader's lease upon each query. Here's how it works:
At time T(n), when the leader is questioned about its authority, it first collects the send times of the last replied AppendEntries from each of its followers, denoted as TR(1), TR(2), ..., TR(2n), sorted in descending order.
Next, it selects the maximum timestamp at when the majority of followers are known to be active, that is, TR(n).
If TR(n) falls within the time range [T(n), T(n) + LeaseTimeoutDuration], then the lease can be successfully renewed.

Revoke

If the lease is expired and the leader cannot renew it, it loses the lease and stops serving read-only requests directly.

How to handle lease during configuration changes

During the configuration changes, the lease can only be renewed if acknowledgments be received by both the old group and the new group. It is the same to leader election restrictions during reconfiguration.

What to do when forced step down

When a leader is forced down, its lease should be effectively revoked.

How to handle CPU drifts

We can lower the ratio allowed for lease timeouts. If the CPU drifts are unbound, better not to use lease read :)

See https://issues.apache.org/jira/browse/RATIS-1866.

[szetszwo]

@SzyWilliam , thanks a lot for working on this! Please see the comments inlined.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for this detailed review! I will address these issues a bit later ;)

[szetszwo]

@SzyWilliam , thanks a lot for working on this! I have some questions and comments inlined. The changing leader case is tricky.

[szetszwo]

We should deduplicate the peers.

[SzyWilliam]

@szetszwo Thanks very much for the detailed review! I'll elaborate the leader changing process. (may be in next PR).

[OneSizeFitsQuorum]

make it static?

[OneSizeFitsQuorum]

Why still update LastRespondedAppendEntriesSendTime at the time of sending?

[SzyWilliam]

getServerRpc().appendEntries(request) is a blocking operation, and once this call returns, the response for the current AppendEntries request has been received. Therefore, we can update its(LastRespondedAppendEntries)sendTime.

[OneSizeFitsQuorum]

got it~

[SzyWilliam]

Made changes on code. @szetszwo@OneSizeFitsQuorum PTAL, thanks!

[OneSizeFitsQuorum]

LGTM

[szetszwo]

@SzyWilliam , thanks for the update! The change looks good. Just a comment inlined.

[szetszwo]

Since the leader is not in the peer list, we should use (lastRespondedAppendEntriesSendTimes.size() - 1)/ 2:

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Instead of creating a list, we may use limit and skip as below.

privateTimestampgetMaxTimestampWithMajorityAck(List<FollowerInfo> followers) {
if (followers == null || followers.isEmpty()) {
returnTimestamp.currentTime();
}
finalintmid = (followers.size() - 1) / 2;
returnfollowers.stream()
.map(FollowerInfo::getLastRespondedAppendEntriesSendTime)
.sorted()
.limit(mid + 1)
.skip(mid)
.iterator()
.next();
}

[szetszwo]

• 1 or 2 followers: use index 0
• 3 or 4 followers: use index 1

Oops, the timestamps are sorted in ascending order but not descending order. Then it should be

• 1 follower: use index 0
• 2 or 3 followers: use index 1
• 4 or 5 followers: use index 2

You formula actually is correct!

finalintmid = followers.size() / 2;

[SzyWilliam]

Thanks a lot for the reviews! Didn't know we can use limit and skip. Now the code is more light-weighted!

[szetszwo]

+1 the change looks good.

[SzyWilliam]

@szetszwo@OneSizeFitsQuorum Thanks a lot for your careful and thorough reviews!