Skip to content

HBASE-29292 Automatically reassign regions when WALs and ZK go missing - #6968

Closed
taklwu wants to merge 5881 commits into
apache:masterfrom
taklwu:HBASE-29292
Closed

HBASE-29292 Automatically reassign regions when WALs and ZK go missing#6968
taklwu wants to merge 5881 commits into
apache:masterfrom
taklwu:HBASE-29292

Conversation

@taklwu

@taklwutaklwu commented May 8, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

sorry for the long delay from the comments of #2114 that peter asked changes for HBASE-24282.

This patch addresses the previous comments by introducing a new feature flag, hbase.master.assign.regions.on.unknown.servers, to handle cases where an HBase cluster is restarted after WAL and ZooKeeper nodes have been removed. This is particularly useful when recreating an HBase cluster on cloud storage, where HDFS and ZooKeeper are always initialized fresh.

2005hithljand others added 30 commits July 31, 2024 09:12
@2005hithlj
HBASE-28389 HBase backup yarn queue parameter ignored (apache#6130)
68045db
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@Apache9
HBASE-28760 Exclude pom file of jaxws-ri in output tarball (apache#6140)
34898bf
Signed-off-by: Nihal Jain <nihaljain@apache.org>
@ndimiduk@charlesconnell
HBASE-28346: Expose checkQuota to Coprocessor Endpoints (apache#6066)
f0f6131
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Co-authored-by: Charles Connell <cconnell@hubspot.com>
@anmolnar
HBASE-27118 Add security headers to Thrift/HTTP server (branch-2) (ap…
be762db
…ache#6153)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Istvan Toth <stoty@apache.org>
@NihalJain
HBASE-28250 Bump jruby to 9.4.8.0 to fix snakeyaml CVE (apache#6127) (a…
d94def3
…pache#6146)
* Sync code as per irb 1.4.2
* Also provide option to try irb's new functionalities for colorize and autocomplete
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 6788ff4)
@NihalJain
HBASE-28784 Exclude samples and release-documentation zip of jaxws-ri…
a562be9
… from output tarball (apache#6157) (apache#6159)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 98e7e1b)
@NihalJain
HBASE-28786 Fix classname for command: copyreppeers in bin/hbase (apa…
0223a4f
…che#6162) (apache#6164)
Signed-off-by: Pankaj <pankajkumar@apache.org>
(cherry picked from commit 41dd87c)
@jinhyukify
HBASE-28732 Fix typo in Jenkinsfile_Github for jdk8 hadoop2 check (ap…
18ddef1
…ache#6174)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@Umeshkumar9414
HBASE-28690 Aborting Active HMaster is not rejecting reportRegionStat…
9df62ab
…eTransition if procedure is initialised by next Active master (apache#6136)
Added masterActiveTime as fencing token for remote procedures
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
@anmolnar
HBASE-28777 mTLS client hostname verification doesn't work with Optio…
4cfe8cc
…nalSslHandler (apache#6181)
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
@srinireddy2020
HBASE-28802 Log the IP when hbase.server.useip.enabled is set to true (…
9292059
…apache#6175)
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
Reviewed-by: Vineet Kumar Maheshwari <vineet.4008@gmail.com>
@rmdmattinglyRay Mattingly@ndimiduk
HBASE-28643 An unbounded backup failure message can cause an irrecove…
ce5a974
…rable state for the given backup (apache#6088)
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@rmdmattinglyRay Mattingly@ndimiduk
HBASE-28697 Don't clean bulk load system entries until backup is comp…
ac5d492
…lete (apache#6089)
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
@pankaj72981
HBASE-28793 Update hbase-thirdparty to 4.1.8 (apache#6188)
4c7bb57
Signed-off-by: Nihal Jain <nihaljain@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Vineet Kumar Maheshwari <vineet.4008@gmail.com>
@ndimiduk@charlesconnell
Backport "HBASE-28792: AsyncTableImpl should call coprocessor callbac…
18f6734
…ks in a defined order (apache#6168)" to branch-2 (apache#6200)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Co-authored-by: Charles Connell <cconnell@hubspot.com>
@ndimiduk
Revert "Backport "HBASE-28792: AsyncTableImpl should call coprocessor…
7367f50
… callbac…" (apache#6203)
This reverts commit 18f6734.
@ndimiduk@charlesconnell
HBASE-28792: AsyncTableImpl should call coprocessor callbacks in a de…
fb9d89d
…fined order (apache#6168)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Co-authored-by: Charles Connell <cconnell@hubspot.com>
@Apache9rodenli
HBASE-28669 After one RegionServer restarts, another RegionServer lea…
4da8880
…ks a connection to ZooKeeper (apache#6207)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Co-authored-by: rodenli <rodenli@tencent.com>
@guluo2016
HBASE-28778 NPE may occur when opening master-status or table.jsp or …
6a5757d
…procedure.jsp while Master is initializing (apache#6210)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@NihalJain
HBASE-28807 Remove some useless code and add some logs for CanaryTool (…
5ca5faf
…apache#6187) (apache#6214)
Co-authored-by: WangXin <1458451310@qq.com>
Co-authored-by: wangxin <wangxin9702@gmail.com>
Signed-off-by: Nihal Jain <nihaljain@apache.org>
(cherry picked from commit b161ad5)
@Apache9
HBASE-28580 Revert the deprecation for methods in WALObserver (apache…
76d28b0
…#6205) (apache#6211)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 35c7fc0)
@jhungund
HBASE-28805: Chunked persistence of backing map for persistent bucket…
0d42ad9
… cache. (apache#6183) (apache#6198)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
@ragarkar
HBASE-28811 Use region server configuration for evicting the cache wh…
6669e3c
…ile unassigning a region (apache#6197) (apache#6217)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
(cherry picked from commit 888e4dd)
@rmdmattinglyRay Mattingly@ndimiduk
HBASE-28810 Improve BackupLogCleaner naming, debug logging (apache#6195)
3f67699
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@rmdmattinglyRay Mattingly@ndimiduk
HBASE-28696 Partition BackupSystemTable queries (apache#6067)
78da9e9
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@NihalJain
HBASE-28070 Replace javax.servlet.jsp dependency with tomcat-jasper (a…
2537690
…pache#6218) (apache#5607)
- Also handle changes required for hadoop-2 assembly
Signed-off-by: Istvan Toth <stoty@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit b6394b1)
@stoty
HBASE-28725 Use thirdparty protobuf for REST interface in HBase 2.x (a…
fe3a634
…pache#6075)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
@stoty
HBASE-28725 Use thirdparty protobuf for REST interface in HBase 2.x (…
8f11a1a
…addendum: sync protobuf 3 version to hbase-thidparty)
@jhungund
HBASE-28804: Implement asynchronous retrieval of bucket-cache data fr…
d4cffea
…om persistence (apache#6182) (apache#6219)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
@Apache9@wangxin97-02
HBASE-28824 Add deprecation cycle for methods in RegionServerObserver (…
39c7535
…apache#6226) (apache#6238)
Co-authored-by: wangxin <wangxin9702@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 3390195)
Co-authored-by: WangXin <1458451310@qq.com>
Apache9and others added 16 commits April 20, 2025 13:57
@Apache9
HBASE-29259 Master crash when loading procedures (apache#6906)
38fe074
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@charlesconnell@Apache9
HBASE-29253 Avoid allocating a new closure on every row processed by …
a8ff965
…StoreScanner (apache#6901)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit abc8b43)
@virajjasani
HBASE-29251 Procedure gets stuck if the procedure state cannot be per…
741754a
…sisted (apache#6916) (apache#6910)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: gvprathyusha6 <70918688+gvprathyusha6@users.noreply.github.com>
@charlesconnell@ndimiduk
HBASE-29193: Allow ZstdByteBuffDecompressor to take direct ByteBuffer…
e110ee1
… as input and heap ByteBuffer as output, or vice versa (apache#6806)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@rmdmattingly@DieterDP-ng
HBASE-29003 Proper bulk load tracking (apache#6506) (apache#6921)
148dd9e
Signed-off-by: Ray Mattingly <rmattingly@apache.org>
The HBase backup mechanism keeps track of which HFiles
were bulk loaded, so they can be included in incremental
backups.
Before this ticket, these bulk load records were only
deleted when an incremental backup is created. This
commit adds 2 more locations:
1) after a full backup. Since a full backup already
captures all data, this meant that unnecessary HFiles
were being included in the next incremental backup.
2) after a table delete/truncate/CF-deletion. Previously,
if an HFile was loaded before a table was cleared, the next
incremental backup would effectively still include the
HFile. This lead to incorrect data being restored.
This commit also completely refactors & simplifies the
test for this functionality.
Co-authored-by: DieterDP <90392398+DieterDP-ng@users.noreply.github.com>
@hgromerHernan Gelaf-Romer@ndimiduk
HBASE-29090: Add server-side load metrics to client results (apache#6623
6162115
)
Co-authored-by: Hernan Gelaf-Romer <hgelafromer@hubspot.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ray Mattingly <rmattingly@apache.org>
@challenger71498@Apache9
HBASE-25338 Use ClassLoader of current class instead of thread contex…
5934785
…t ClassLoader (apache#6918)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit bd30367)
@ndimiduk@charlesconnell
HBASE-29218: Reduce calls to Configuration#get() in decompression path (
934a26d
apache#6857)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Co-authored-by: Charles Connell <cconnell@hubspot.com>
@rmdmattinglyRay Mattingly
HBASE-29207 The backup system table should be considered a system tab…
c78362d
…le (apache#6842) (apache#6929)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Nihal Jain <nihaljain@apache.org>
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
@charlesconnell@ndimiduk
HBASE-29204: BufferedMutatorParams#clone() should copy the requestAtt…
b90b0cc
…ributes field (apache#6823)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
@charlesconnell@ndimiduk
HBASE-29252: Reduce allocations in RowIndexSeekerV1 (apache#6902)
6535744
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
@wchevreuil
HBASE-29249 Allow for BlockCache implementations to define dynamic pr…
87b5550
…operties (apache#6948)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
(cherry picked from commit ae40069)
@mwkang@Apache9
HBASE-29254 StoreScanner returns incorrect row after flush due to top…
524825e
…Changed behavior (apache#6900)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 99bd5b5)
@rmdmattinglyRay Mattingly
HBASE-29281 Atomic request throttles are missing QuotaSettingsFactory…
259eb0b
… support (apache#6953) (apache#6957)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Co-authored-by: Ray Mattingly <rmattingly@hubspot.com>
@Apache9
HBASE-29257 SnapshotTableAction should have ways to clean up snapshots (
c07bc80
apache#6924)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 5dafa9e)
@taklwu@joshelser@ss77892
HBASE-29292 Automatically reassign regions when WALs and ZK go missing
d2b8efb
Co-authored-by: Josh Elser <elserj@apache.org>
Co-authored-by: Sergey Soldatov <ssa@apache.org>
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

This comment has been minimized.

Sign in to view
@Apache-HBase

Apache-HBase commented May 8, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 45sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ branch-2 Compile Tests _
+1 💚mvninstall3m 21sbranch-2 passed
+1 💚compile3m 7sbranch-2 passed
+1 💚checkstyle0m 40sbranch-2 passed
+1 💚spotbugs1m 39sbranch-2 passed
+1 💚spotless0m 50sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall3m 15sthe patch passed
+1 💚compile2m 59sthe patch passed
+1 💚javac2m 59sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 35sthe patch passed
+1 💚spotbugs1m 42sthe patch passed
+1 💚hadoopcheck17m 4sPatch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚spotless0m 48spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
38m 46s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6968/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6968
Optional Testsdupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
unameLinux 8d4967b8cdef 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 028d0f2
Default JavaEclipse Adoptium-11.0.23+9
Max. process+thread count79 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6968/2/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented May 8, 2025

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 48sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 26sbranch-2 passed
+1 💚compile0m 41sbranch-2 passed
+1 💚javadoc0m 23sbranch-2 passed
+1 💚shadedjars5m 17sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile0m 41sthe patch passed
+1 💚javac0m 41sthe patch passed
+1 💚javadoc0m 24sthe patch passed
+1 💚shadedjars5m 16spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit214m 13shbase-server in the patch passed.
236m 49s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6968/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#6968
Optional Testsjavac javadoc unit compile shadedjars
unameLinux d985ad25b7d8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 028d0f2
Default JavaTemurin-1.8.0_412-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6968/2/testReport/
Max. process+thread count4016 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6968/2/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@taklwu
taklwu changed the base branch from branch-2 to masterMay 9, 2025 18:08
@taklwutaklwu closed this May 9, 2025
@taklwu

taklwu commented May 9, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

let me reopen a new one against master #6981

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wchevreuilwchevreuilAwaiting requested review from wchevreuil
@joshelserjoshelserAwaiting requested review from joshelser
@petersomogyipetersomogyiAwaiting requested review from petersomogyi
@stotystotyAwaiting requested review from stoty

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

20 participants

@taklwu@Apache-HBase@2005hithlj@Apache9@ndimiduk@anmolnar@NihalJain@jinhyukify@Umeshkumar9414@srinireddy2020@rmdmattingly@pankaj72981@guluo2016@jhungund@ragarkar@stoty@hgromer@eab148@eubnara@charlesconnell