Uh oh!
There was an error while loading. Please reload this page.
HBASE-24286: HMaster won't become healthy after after cloning or crea… - #2114
HBASE-24286: HMaster won't become healthy after after cloning or crea…#2114taklwu wants to merge 1 commit into
Conversation
…ting a new cluster pointing at the same file system HBase currently does not handle `Unknown Servers` automatically and requires users to run hbck2 `scheduleRecoveries` when one see unknown servers on the HBase report UI. This became a blocker on HBase2 adoption especially when a table wasn't disabled before shutting down a HBase cluster on cloud or any dynamic environment that hostname may change frequently. Once the cluster restarts, hbase:meta will be keeping the old hostname/IPs for the previous cluster, and those region servers became `Unknown Servers` and will never be recycled. Our fix here is to trigger a repair immediately after the CatalogJanitor figured out any `Unknown Servers` with submitting a HBCKServerCrashProcedure such that regions on `Unknown Server ` can be reassigned to other online servers.
joshelser
left a comment
There was a problem hiding this comment.
Nice test to accompany this.
While I think the CatalogJanitor approach is probably an effective solution, I wonder if there's a "faster" solution we could do.
The main question is, when we don't have ZooKeeper telling us that a RegionServer has died, how can we be certain that a RegionServer won't "come back"? If we get into a situation where data was still hosted on a RegionServer we thought was dead, we would double-assign the region and that'd be a big bug.
Any thoughts on how to try to minimize the chance of us incorrectly marking a RegionServer as dead?
| if (!this.lastReport.isEmpty()) { | ||
| LOG.warn(this.lastReport.toString()); | ||
| // expires unknown servers | ||
| repairUnknownServers(); |
There was a problem hiding this comment.
I guess not an issue for master (which doesn't have the separate namespace table), but elsewhere do we still have hbase.master.namespace.init.timeout setting an upper-bound on how long we wait for hbase:namespace to get assigned? Thinking that, waiting for CatalogJanitor to run, will be a pretty "slow" solution (up to 5min wait), and we may have a master crash if the ns init timeout is 5 mins as well as the catalog janitor's interval.
Apache-HBase
commented
Jul 21, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 21, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Jul 21, 2020
💔 -1 overall
This message was automatically generated. |
petersomogyi
commented
Jun 21, 2021
I've made some testing recently using this patch to be able to start an HBase cluster on a pre-existing HBase root directory. Currently I have to use Since for some installations it might not be required to do this I'm suggesting to hide this behind a feature flag. |
joshelser
commented
Jun 21, 2021
Makes sense to me. I think that addresses some of the other concerns from @Apache9 (mentioning him to make sure that's OK with him). If @taklwu is OK with it (and can grant you edit perms), maybe you can update this PR with your changes? Or, close this and open a new one with your modifications. |
z-york
commented
Jun 21, 2021
@petersomogyi It's probably worth pinging on #2113 and subsequent PRs as that's where most of the conversation happened. I thought Stephen had offered to put this behind a config before, but maybe I'm mistaken. Anyways, it's worth revisting this issue anyways IMO. |
…ting a new cluster pointing at the same file system
HBase currently does not handle
Unknown Serversautomatically and requiresusers to run hbck2
scheduleRecoverieswhen one see unknown servers onthe HBase report UI.
This became a blocker on HBase2 adoption especially when a table wasn't
disabled before shutting down a HBase cluster on cloud or any dynamic
environment that hostname may change frequently. Once the cluster restarts,
hbase:meta will be keeping the old hostname/IPs for the previous cluster,
and those region servers became
Unknown Serversand will never be recycled.Our fix here is to trigger a repair immediately after the CatalogJanitor
figured out any
Unknown Serverswith submitting a HBCKServerCrashProceduresuch that regions on
Unknown Servercan be reassigned to other onlineservers.