Uh oh!
There was an error while loading. Please reload this page.
HBASE-28420 Update the procedure's field to store for ServerRemoteProcedure - #5816
Merged
Conversation
Apache-HBase
commented
Apr 10, 2024
💔 -1 overall
This message was automatically generated. |
Apache9
reviewed
Apr 10, 2024
| return; | ||
| } | ||
| //below persistence is added so that if report goes to last active master, it throws exception | ||
| state = MasterProcedureProtos.ServerRemoteProcedureState.SERVER_REMOTE_PROCEDURE_REPORT_SUCCEED; |
Contributor
There was a problem hiding this comment.
If there is an error, we should not marked it as succeed? And we also need to persist the error field?
ContributorAuthor
There was a problem hiding this comment.
I have changed the status. For persisting the error field I am thinking about it. I did see an example of persisting the exception. If need to persist I think I can use those.
| state = MasterProcedureProtos.ServerRemoteProcedureState.SERVER_REMOTE_PROCEDURE_REPORT_SUCCEED; | ||
| env.getMasterServices().getMasterProcedureExecutor().getStore().update(this); | ||
| complete(env, error); |
Contributor
There was a problem hiding this comment.
Here, we should not call the complete directly again. We should wake up the procedure and process the result in the execute method.
ContributorAuthor
There was a problem hiding this comment.
If I have to call the complete method in execute methods, I have to store the Error as complete methods are using that to take the decision.
Another solution can be I play with the state. According to the error in remoteOperationDone method, I can change the state and then take all the decisions in complete method based on the state. But I don't think this would be possible without introducing some state specific to some parent procedure like SnapshotVerifyProcedure.
Contributor
There was a problem hiding this comment.
Persisting the error is a must...
Think of this scenario, after you persist the state, master crashes before you call complete, and after restart, you enter the execute method, if you do not have the error object then how do you plan to call the comlete method...
ContributorAuthor
There was a problem hiding this comment.
First I want to understand what is your thought for not calling complete directly again - is it because of design? We want state transition in execute only?
Think of this scenario, after you persist the state, master crashes before you call complete, and after restart.
If we are fine with calling complete in remoteOperationDone if we call complete before persisting the state will that be okay ?
ContributorAuthor
There was a problem hiding this comment.
I tried to write another approach as well. You can take a look. https://github.com/apache/hbase/compare/master...Umeshkumar9414:hbase:HBASE_28420_2?expand=1
ContributorAuthor
Contributor
There was a problem hiding this comment.
I do not think you need to consider a master with old code become active, as we are trying to fix a bug in the old code...
And I still do not get your point on why you must call complete in remoteOperationDone method, as I explained above, if master crashes after we persist the state and then restarts, we need to call complete in the execute method, so why not just follow the same pattern since you must have the code in execute method?
And there is another advantage to prevent potential data race if you call complete method in execute method, as the execute method will be called by PEWorker, where we have a bunch of fencing ways to make sure that there is no race.
ContributorAuthor
There was a problem hiding this comment.
I wanted to call complete in remoteOperationDone so that we didn't encounter the scenario of a master with old code becoming active. Anyway as we don't need to consider this, I have updated the PR with new changes.
Apache-HBase
commented
Apr 10, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 10, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 10, 2024
🎊 +1 overall
This message was automatically generated. |
| @@ -80,6 +81,8 @@ public abstract class ServerRemoteProcedure extends Procedure<MasterProcedureEnv | |||
| protected ServerName targetServer; | |||
| protected boolean dispatched; | |||
| protected boolean succ; | |||
ContributorAuthor
There was a problem hiding this comment.
@Apache9 do you think I should replace this succ flag with state ?
Contributor
There was a problem hiding this comment.
Do it in your own convenience. We must have a state here, you can either add a state but still keep this flag, or you can implement all the logic with the state and drop this succ flag.
| protected ServerName targetServer; | ||
| protected boolean dispatched; | ||
| protected boolean succ; | ||
| protected MasterProcedureProtos.ServerRemoteProcedureState state = |
ContributorAuthor
There was a problem hiding this comment.
If I am using state for making decisions then I think we need to handle HMaster restart scenarios while HBase upgrade.
Apache-HBase
commented
Apr 11, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 11, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 12, 2024
🎊 +1 overall
This message was automatically generated. |
Apache9
reviewed
Apr 20, 2024
| @Override | ||
| public synchronized void remoteOperationFailed(MasterProcedureEnv env, | ||
| RemoteProcedureException error) { | ||
| state = MasterProcedureProtos.ServerRemoteProcedureState.SERVER_REMOTE_PROCEDURE_SERVER_CRASH; |
Contributor
There was a problem hiding this comment.
I do not think this should be server crash? The remote region server could also report the failure?
ContributorAuthor
There was a problem hiding this comment.
For this, I relied on the documentation comments.
If the targetServer crashed but this
* procedure has no response, than dispatcher will call remoteOperationFailed()
ContributorAuthor
There was a problem hiding this comment.
I went through the code and found that this is not the case(you are right it is not a server crash). Let me change it in that scenario.
| state = MasterProcedureProtos.ServerRemoteProcedureState.SERVER_REMOTE_PROCEDURE_REPORT_SUCCEED; | ||
| env.getMasterServices().getMasterProcedureExecutor().getStore().update(this); | ||
| complete(env, error); |
Apache-HBase
commented
Apr 24, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 24, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 24, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 24, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 29, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 29, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 29, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 29, 2024
🎊 +1 overall
This message was automatically generated. |
Umeshkumar9414
commented
Apr 30, 2024
ContributorAuthor
The reason for the last build failure is |
Apache-HBase
commented
Apr 30, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
May 30, 2024
Apache-HBase
commented
May 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 30, 2024
💔 -1 overall
This message was automatically generated. |
added 8 commits
May 31, 2024 00:35
…istence and complete change
virajjasani
commented
May 30, 2024
Contributor
@Umeshkumar9414 could you pull from latest master one more time? |
virajjasani
left a comment
Contributor
There was a problem hiding this comment.
The change overall looks good, I wonder if we should add WARN/ERROR log while setting ForeignExceptionMessage at some places though
ContributorAuthor
On worker side, at RS do have logs for example - At master side also we do have debug logs - |
Apache-HBase
commented
May 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 30, 2024
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
May 30, 2024
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
May 31, 2024
💔 -1 overall
This message was automatically generated. |
Umeshkumar9414 added a commit
to Umeshkumar9414/hbase
that referenced
this pull request
Jun 3, 2024
…cedure (apache#5816) Co-authored-by: ukumawat <ukumawat@salesforce.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>
Umeshkumar9414 added a commit
to Umeshkumar9414/hbase
that referenced
this pull request
Jun 4, 2024
…cedure (apache#5816) Co-authored-by: ukumawat <ukumawat@salesforce.com> Signed-off-by: Duo Zhang <zhangduo@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ServerRemoteProcedure does not have its own serializing implementation, so had to make changes in all the sub-procedures.