RATIS-993. Support pre vote

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.

[runzhiwang]

What changes were proposed in this pull request?

1. Implement Preventing disruptions when a server rejoins the cluster according to raft paper section 9.6.

2. In ozone, rejoin happen easily because the unstable network, for example sometimes one server's network is slow, it will try to trigger election, but which is unnecessary, this can be prevented by this PR.

3. In this PR, before follower trigger leader election, it send pre-vote request to the other servers, if other server allow the follower to trigger leader election, it reply PASSED to the pre-vote request. If majority of the other servers allow the follower trigger leader election, the follower will trigger leader election.

4. The other server allow the follower to trigger leader election only when 1. the voters have not received heartbeats from a valid leader for at least a baseline election timeout. and 2. the follower's log is newer than voter's, or the follower's log is equal to voter's and follower's priority is not less than voter's

5. If the above condition can not satisfied, even though the follower trigger leader election, it can not be elected as a leader but it can step down the leader by it's bigger term, so we do not allow it trigger leader election.

6. If the follower is not in the conf, we will shutdown the follower.

Details of the paper:

One downside of Raft’s leader election algorithm is that a server that has been partitioned from the
cluster is likely to cause a disruption when it regains connectivity. When a server is partitioned, it
will not receive heartbeats. It will soon increment its term to start an election, although it won’t
be able to collect enough votes to become leader. When the server regains connectivity sometime
later, its larger term number will propagate to the rest of the cluster (either through the server’s
RequestVote requests or through its AppendEntries response). This will force the cluster leader to
step down, and a new election will have to take place to select a new leader. Fortunately, such events
are likely to be rare, and each will only cause one leader to step down.
If desired, Raft’s basic leader election algorithm can be extended with an additional phase to
prevent such disruptions, forming the Pre-Vote algorithm. In the Pre-Vote algorithm, a candidate
only increments its term if it first learns from a majority of the cluster that they would be willing
to grant the candidate their votes (if the candidate’s log is sufficiently up-to-date, and the voters
have not received heartbeats from a valid leader for at least a baseline election timeout). This was
inspired by ZooKeeper’s algorithm [42], in which a server must receive a majority of votes before
it calculates a new epoch and sends NewEpoch messages (however, in ZooKeeper servers do not
solicit votes, other servers offer them).
The Pre-Vote algorithm solves the issue of a partitioned server disrupting the cluster when it
rejoins. While a server is partitioned, it won’t be able to increment its term, since it can’t receive
permission from a majority of the cluster. Then, when it rejoins the cluster, it still won’t be able
to increment its term, since the other servers will have been receiving regular heartbeats from the
leader. Once the server receives a heartbeat from the leader itself, it will return to the follower state
(in the same term).
We recommend the Pre-Vote extension in deployments that would benefit from additional robustness.
We also tested it in various leader election scenarios in AvailSim, and it does not appear
to significantly harm election performance.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/RATIS-993

How was this patch tested?

add ut: testPreVote

[runzhiwang]

@szetszwo@bshashikant@lokeshj1703 @GlenGeng Could you help review this patch ? Thank you very much.

[szetszwo]

@runzhiwang thanks for working on this.

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

[szetszwo]

2. the follower is not in the conf.

I cannot see this requirement in the paper. Could you give me a pointer?

[runzhiwang]

... the voters have not received heartbeats from a valid leader for at least a baseline election timeout ...

This requirement seems not yet implemented in the current change.

@szetszwo Thanks for review. You are right, I did not implement it in this PR because I want to make this PR simpler to review, and I will implement it after this PR. Even though without checkout timeout, it will not introduce new bug, because in this case pre-vote passed and trigger a new leader election. I can also implement it in this PR, it's ok to me. Which one do you prefer ?

[runzhiwang]

the follower is not in the conf.
I cannot see this requirement in the paper. Could you give me a pointer?

@szetszwo You are right, it does not exist in the paper. I implement it to adapt current ratis implementation. Because in current ratis, when leader election, leader will send shutdown to follower if follower, such as follower1, is not in leader's conf at shouldSendShutdown(candidateId, candidateLastEntry). So I have two choice to make follower1 shutdown: 1.pass the pre-vote in this case, so that follower1 can shutdown in real leader election; 2.pre-vote in charge of shutdown follower1, but this will make pre-vote complicated. So I choose the first option. What do you think?

[GlenGeng-awx]

Thanks for the work!

[szetszwo]

@runzhiwang Let's just implement exactly raft paper section 9.6.

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

[runzhiwang]

When the follower is not in the conf, it should not allowed to join the group. Otherwise, any followers could join any groups.

@szetszwo Thanks for explanation. Do you mean shutdown the follower not in the conf when pre-vote ?

[szetszwo]

Yes, we should shutdown the follower for that group.

[runzhiwang]

@szetszwo Thanks for review. I have updated the patch. Could you help review it again ?

[runzhiwang]

@szetszwo I have updated this pre-vote patch. Could you help review it ?

[runzhiwang]

will fix the failed ut.

[szetszwo]

Thanks a lot for working on this.

• Terminology: Let's use "Pre-Vote" or "pre-vote" as in https://github.com/ongardie/dissertation
• There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

Some other comments inlined.

[runzhiwang]

There are quite many code duplications between pre-vote and vote. We may refactor the code here or in a separated JIRA.

@szetszwo Thanks for review. I will refactor the code in next PR.

Besides, when s1 transferLeaderShip to s2, s2 should skip askForPreVote, and askForVote directly. Because when s2 askForPreVote, s1 and s3 will reject it because current leader is still valid. So I add a flag: LeaderElection#force which means we should skip askForPreVote and askForVote directly, this flag will be true when s2 receive StartLeaderElectionRequest.

[szetszwo]

+1 the change looks good.

[szetszwo]

... I will refactor the code in next PR.

@runzhiwang , thanks a lot. Since you are working on RATIS-1273, let me do the refactoring.

[runzhiwang]

@szetszwo Sure, thanks a lot.