Uh oh!
There was an error while loading. Please reload this page.
HBASE-27698 Migrate meta locations from zookeeper to master data may … - #5167
HBASE-27698 Migrate meta locations from zookeeper to master data may …#5167chrajeshbabu wants to merge 1 commit into
Conversation
…not always possible if we migrate from 1.x HBase
Apache-HBase
commented
Apr 9, 2023
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 9, 2023
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Apr 9, 2023
💔 -1 overall
This message was automatically generated. |
chrajeshbabu
commented
Apr 11, 2023
The test case failures are not related to this change. @taklwu could you please review the change. Thanks. |
Apache9
commented
Apr 11, 2023
Why change from throwing an exception to returning false? |
@Apache9 Here I have mentioned how the upgrade went smooth with my patch. |
virajjasani
commented
Apr 12, 2023
Thank you @chrajeshbabu. It seems the changes make sense from the upgrade viewpoint. On the other hand, if this rare scenario were to happen on a healthy 2.x cluster, does this change make any difference?
If this happens on 2.x cluster, what would now be different? Only that meta would be assigned on any random server, correct? |
chrajeshbabu
commented
Apr 13, 2023
I have tried two scenarios in healthy 2.x cluster with the change
By throwing exception also we need to delete master data because of failed init meta procedure not bring the master up and need to follow one of these steps
|
virajjasani
commented
Apr 14, 2023
Got it, thanks. This makes sense, +1 from my side. |
virajjasani
commented
Apr 14, 2023
@Apache9 does this look good to you? |
chrajeshbabu
commented
Apr 19, 2023
@Apache9 will commit it if it's fine for you. Could you please confirm. |
Apache9
commented
Apr 19, 2023
I still need to check the code. This is a very critical part, if we run InitMetaProcedure when meta exists, it could cause serious data loss problem... We should try to prevent scheduling the InitMetaProcedure, instead of just letting it go and hoping it would work... |
Apache9
commented
Apr 20, 2023
After chekcing the code, I think the correct way for fixing this problem is to do the work in HMaster.tryMigrateMetaLocationsFromZooKeeper. We can some more code in this method, to check whether the meta directory is already there and if it is, if we can make sure that this is an upgrading from 1.x(by trying to read something on the filesystem? Is this possible?), then insert a record to the master region so we can skip scheduling the InitMetaProcedure. |
chrajeshbabu
commented
Apr 20, 2023
we can check the column families in meta table to detect whether it's from 1.x or current versions. |
Apache9
commented
Apr 20, 2023
And do we have read replica support on 1.x? Do we also need to insert the record for secondary replicas? |
chrajeshbabu
commented
Apr 20, 2023
We can insert record but we may not know the state and location which leads to InitMetaProcedure again. |
Apache9
commented
Apr 20, 2023
Checked the code, the condition for whether to schedule an InitMetaProcedure is So I think it only needs we have a record in master region for meta, and do not need to know its state and location? And we will first try to migrate from zookeeper, and for 1.x, if the cluster shutdown gracefully, there is no znode for meta, then we can enter the logic described above, so in this case, I think the state for meta should be CLOSED? |
chrajeshbabu
commented
Apr 20, 2023
That's correct let me check and update the patch accordingly. |
virajjasani
commented
Apr 20, 2023
@chrajeshbabu but isn't meta state unknown in your case of migration from 1.x to 2.x? Is the plan to read meta CF from filesystem directly as part of |
@Apache9 I have tried your suggestion of creating put entry of meta table in master region which gets filled in regionstates and helps to avoid calling init meta procedure there is problem with this approach. Here is the code I have tried.
|
Apache9
commented
May 9, 2023
Better post the full patch somewhere so I can check it? And maybe we need to manually schedule a TRSP to bring meta region online in this case... |
Apache-HBase
commented
Dec 15, 2025
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 15, 2025
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 15, 2025
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Dec 15, 2025
🎊 +1 overall
This message was automatically generated. |
…not always possible if we migrate from 1.x HBase