Skip to content

HBASE-26938 Compaction failures after StoreFileTracker integration (branch-2, branch-2.5) - #4334

Closed
apurtell wants to merge 1 commit into
apache:branch-2from
apurtell:HBASE-26938-branch-2
Closed

HBASE-26938 Compaction failures after StoreFileTracker integration (branch-2, branch-2.5)#4334
apurtell wants to merge 1 commit into
apache:branch-2from
apurtell:HBASE-26938-branch-2

Conversation

@apurtell

@apurtellapurtell commented Apr 9, 2022

Copy link
Copy Markdown
Contributor

One Compactor instance is reused for the lifetime of a store, and it has a writer field that at issue here.

More than one compaction cannot be concurrently selected and executed against a given store or else readers or writers of the writer field will encounter multithreaded correctness problems. Yet I am seeing concurrent selection and execution of compaction activity against the store in the test scenario.

In the test scenario I have increased the size of the small and large compaction thread pools, to 10 and 5 threads, respectively, and increased the default point for blocking files to 24, and in the scenario the store is flushing furiously. Operation under these conditions used to be reliable, but perhaps only by an accidental serialization of compaction activity prior to the SFT changes.

With this change in place the reliability and performance under the test scenario returns to previous baseline for DEFAULT SFT. No ERRORs.

Suggestions of alternative approaches to fixing this are welcome too.

@apurtell
apurtellforce-pushed the HBASE-26938-branch-2 branch from ab7ff35 to 79617adCompareApril 9, 2022 19:08
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 6sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚mvninstall5m 24sbranch-2 passed
+1 💚compile5m 0sbranch-2 passed
+1 💚checkstyle1m 5sbranch-2 passed
+1 💚spotbugs2m 46sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall5m 1sthe patch passed
+1 💚compile4m 47sthe patch passed
+1 💚javac4m 47sthe patch passed
+1 💚checkstyle1m 4sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 18sPatch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚spotbugs3m 35sthe patch passed
_ Other Tests _
+1 💚asflicense0m 23sThe patch does not generate ASF License warnings.
58m 48s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4334
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux b54dd41ede3b 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count69 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 22sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall4m 12sbranch-2 passed
+1 💚compile1m 17sbranch-2 passed
+1 💚shadedjars5m 45sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall4m 0sthe patch passed
+1 💚compile1m 14sthe patch passed
+1 💚javac1m 14sthe patch passed
+1 💚shadedjars5m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sthe patch passed
_ Other Tests _
+1 💚unit260m 42shbase-server in the patch passed.
287m 24s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 490f3afeb2f4 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/testReport/
Max. process+thread count2421 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 40sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall5m 27sbranch-2 passed
+1 💚compile1m 25sbranch-2 passed
+1 💚shadedjars6m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 51sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall4m 58sthe patch passed
+1 💚compile1m 24sthe patch passed
+1 💚javac1m 24sthe patch passed
+1 💚shadedjars6m 29spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44sthe patch passed
_ Other Tests _
+1 💚unit259m 25shbase-server in the patch passed.
291m 0s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 56a4e0dda798 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/testReport/
Max. process+thread count2221 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 48sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚mvninstall2m 23sbranch-2 passed
+1 💚compile2m 15sbranch-2 passed
+1 💚checkstyle0m 38sbranch-2 passed
+1 💚spotbugs1m 16sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall1m 57sthe patch passed
+1 💚compile2m 15sthe patch passed
+1 💚javac2m 15sthe patch passed
+1 💚checkstyle0m 36sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck7m 49sPatch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚spotbugs1m 19sthe patch passed
_ Other Tests _
+1 💚asflicense0m 9sThe patch does not generate ASF License warnings.
26m 25s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4334
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 878d73ebc6e5 5.4.0-1025-aws #25~18.04.1-Ubuntu SMP Fri Sep 11 12:03:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 24sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 34sbranch-2 passed
+1 💚compile0m 47sbranch-2 passed
+1 💚shadedjars3m 52sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 28sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 28sthe patch passed
+1 💚compile0m 47sthe patch passed
+1 💚javac0m 47sthe patch passed
+1 💚shadedjars3m 51spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 28sthe patch passed
_ Other Tests _
+1 💚unit209m 17shbase-server in the patch passed.
228m 7s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2b9aef4beca1 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/testReport/
Max. process+thread count2363 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 13sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 11sbranch-2 passed
+1 💚compile1m 0sbranch-2 passed
+1 💚shadedjars5m 23sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 34sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 15sthe patch passed
+1 💚compile1m 1sthe patch passed
+1 💚javac1m 1sthe patch passed
+1 💚shadedjars5m 23spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 35sthe patch passed
_ Other Tests _
+1 💚unit252m 20shbase-server in the patch passed.
275m 24s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux ae274b4eb8cc 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e7eb628
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/testReport/
Max. process+thread count2242 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

…ranch-2, branch-2.5)
- Throw an IllegalStateException for the "Writer exists when it should not" case in Compactor.
- If a store is already compacting, do not select it for any additional concurrent compaction.
@apurtell
apurtellforce-pushed the HBASE-26938-branch-2 branch from 79617ad to 08f8a6dCompareApril 11, 2022 18:36
@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 11sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+1 💚mvninstall2m 30sbranch-2 passed
+1 💚compile2m 19sbranch-2 passed
+1 💚checkstyle0m 41sbranch-2 passed
+1 💚spotbugs1m 20sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall2m 9sthe patch passed
+1 💚compile2m 22sthe patch passed
+1 💚javac2m 22sthe patch passed
+1 💚checkstyle0m 38sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck7m 51sPatch does not cause any errors with Hadoop 3.1.2 3.2.1.
+1 💚spotbugs1m 26sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
27m 46s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4334
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux eae1457f7b8e 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f9abe3d
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 43sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall4m 1sbranch-2 passed
+1 💚compile1m 4sbranch-2 passed
+1 💚shadedjars4m 35sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 47sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 24sthe patch passed
+1 💚compile0m 59sthe patch passed
+1 💚javac0m 59sthe patch passed
+1 💚shadedjars4m 37spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44sthe patch passed
_ Other Tests _
-1 ❌unit28m 53shbase-server in the patch failed.
52m 27s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2ad78387fca8 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f9abe3d
Default JavaAdoptOpenJDK-11.0.10+9
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/testReport/
Max. process+thread count770 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtellapurtell changed the title [WIP] [Investigation] HBASE-26938 Compaction failures after StoreFileTracker integration (branch-2, branch-2.5)HBASE-26938 Compaction failures after StoreFileTracker integration (branch-2, branch-2.5)Apr 11, 2022
@apurtell
apurtell marked this pull request as ready for review April 11, 2022 21:22
@apurtell

Copy link
Copy Markdown
ContributorAuthor

In the last precommit result TestBulkLoadReplicationHFileRefs is having issues but it does not appear related to this change, because this change only impacts compaction selection, and that unit passes for me locally.

[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.hbase.regionserver.TestBulkLoadReplicationHFileRefs
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 66.247 s - in org.apache.hadoop.hbase.regionserver.TestBulkLoadReplicationHFileRefs
[INFO] [INFO] Results:
[INFO] [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

@Apache-HBase

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 47sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 25sbranch-2 passed
+1 💚compile0m 47sbranch-2 passed
+1 💚shadedjars4m 23sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 31sbranch-2 passed
_ Patch Compile Tests _
+1 💚mvninstall3m 4sthe patch passed
+1 💚compile0m 58sthe patch passed
+1 💚javac0m 58sthe patch passed
+1 💚shadedjars4m 25spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26sthe patch passed
_ Other Tests _
+1 💚unit221m 18shbase-server in the patch passed.
243m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4334
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2cddc4666087 5.4.0-90-generic #101-Ubuntu SMP Fri Oct 15 20:00:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / f9abe3d
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/testReport/
Max. process+thread count2215 (vs. ulimit of 12500)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4334/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@apurtell

Copy link
Copy Markdown
ContributorAuthor

As mentioned on HBASE-26938, @Apache9 suggested an alternative approach that I accept and will update this PR soon.

@apurtell

Copy link
Copy Markdown
ContributorAuthor

Opened a PR for master, see #4338 . Closing this one. Will pick back the master commit after merge.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@apurtell@Apache-HBase