Skip to content

HBASE-28328 Add an option to count different types of Delete Markers in RowCounter - #6496

Merged
virajjasani merged 15 commits into
apache:branch-2from
shubham-roy:branch-2-HBASE-28328
Dec 2, 2024
Merged

HBASE-28328 Add an option to count different types of Delete Markers in RowCounter#6496
virajjasani merged 15 commits into
apache:branch-2from
shubham-roy:branch-2-HBASE-28328

Conversation

@shubham-roy

@shubham-royshubham-roy commented Nov 28, 2024

Copy link
Copy Markdown
Contributor

A flag is introduced, which, when enabled, allows RowCounter to count the various types of Delete Markers - DELETE_COLUMN, DELETE_FAMILY, DELETE_FAMILY_VERSION. It will also calculate the number of rows having a delete marker.

To enable this the scan object was modified -> if flag is set, raw scan is performed without FirstKeyOnlyFilter.

Note: Commits are cherry-picked from #6435

Shubham Roy added 15 commits November 28, 2024 09:01
Shubham Roy
Added feature to count delete markers in RowCounter.
7f834cf
(cherry picked from commit 8b9787c)
Shubham Roy
Put all new features under the flag for correct working.
8ceede0
(cherry picked from commit fa10771)
Shubham Roy
Added UT.
9e31e33
(cherry picked from commit 0cb98d8)
Shubham Roy
Fixed style.
d3a255a
(cherry picked from commit 972d3b7)
Shubham Roy
Fixed style by applying spotless plugin.
d020606
(cherry picked from commit db72e7c)
Shubham Roy
Removed banned dependency and used shaded one.
30139f4
(cherry picked from commit f2e9886)
Shubham Roy
Removed annotation.
706715a
(cherry picked from commit f3db2d5)
Shubham Roy
Fixed test.
3bf7ff3
(cherry picked from commit 1240594)
Shubham Roy
Changed filter behaviour for the scan in case of --countDeleteMarkers…
b7106ce
… flag is set.
(cherry picked from commit 5754e53)
Shubham Roy
Applied spotless plugin.
bff2212
(cherry picked from commit 0e69048)
Shubham Roy
Altered postion of setting conf property.
fc9bee1
(cherry picked from commit 7f1d9ba)
Shubham Roy
Added more test coverage for all types of delete markers.
6ebb456
(cherry picked from commit 1904ed6)
Shubham Roy
Removed pom change.
3a4f008
(cherry picked from commit 478ead4)
Shubham Roy
Added UT for range row case.
19ced25
(cherry picked from commit 3b103ef)
Shubham Roy
Added annotation.
1568fc8
(cherry picked from commit 9476622)
@shubham-royshubham-roy mentioned this pull request Nov 28, 2024
Merged
@virajjasani
virajjasani self-requested a review November 28, 2024 04:04
@Apache-HBase

Apache-HBase commented Nov 28, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 43sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗codespell0m 0scodespell was not available.
+0 🆗detsecrets0m 0sdetect-secrets was not available.
+1 💚@author0m 0sThe patch does not contain any @author tags.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
_ branch-2 Compile Tests _
+1 💚mvninstall3m 11sbranch-2 passed
+1 💚compile0m 37sbranch-2 passed
+1 💚checkstyle0m 14sbranch-2 passed
+1 💚spotbugs0m 34sbranch-2 passed
+1 💚spotless0m 48sbranch has no errors when running spotless:check.
_ Patch Compile Tests _
+1 💚mvninstall2m 59sthe patch passed
+1 💚compile0m 35sthe patch passed
+1 💚javac0m 35sthe patch passed
+1 💚blanks0m 0sThe patch has no blanks issues.
+1 💚checkstyle0m 11sthe patch passed
+1 💚xmllint0m 1sNo new issues.
+1 💚spotbugs0m 39sthe patch passed
+1 💚hadoopcheck16m 3sPatch does not cause any errors with Hadoop 2.10.2 or 3.3.6 3.4.0.
+1 💚spotless0m 45spatch has no errors when running spotless:check.
_ Other Tests _
+1 💚asflicense0m 11sThe patch does not generate ASF License warnings.
29m 12s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#6496
Optional Testsdupname asflicense javac codespell detsecrets xmllint hadoopcheck spotless compile spotbugs checkstyle hbaseanti
unameLinux 9e4bfaa37199 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 1568fc8
Default JavaEclipse Adoptium-11.0.23+9
Max. process+thread count78 (vs. ulimit of 30000)
modulesC: hbase-mapreduce U: hbase-mapreduce
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/console
versionsgit=2.34.1 maven=3.9.8 spotbugs=4.7.3 xmllint=20913
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 28, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 50sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall2m 45sbranch-2 passed
+1 💚compile0m 19sbranch-2 passed
+1 💚javadoc0m 16sbranch-2 passed
+1 💚shadedjars5m 23sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall2m 33sthe patch passed
+1 💚compile0m 19sthe patch passed
+1 💚javac0m 19sthe patch passed
+1 💚javadoc0m 14sthe patch passed
+1 💚shadedjars5m 20spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit16m 2shbase-mapreduce in the patch passed.
35m 10s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#6496
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 4aa1cf8356c1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 1568fc8
Default JavaTemurin-1.8.0_412-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/testReport/
Max. process+thread count2770 (vs. ulimit of 30000)
modulesC: hbase-mapreduce U: hbase-mapreduce
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 28, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec0m 42sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 10sbranch-2 passed
+1 💚compile0m 25sbranch-2 passed
+1 💚javadoc0m 18sbranch-2 passed
+1 💚shadedjars5m 41sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall2m 59sthe patch passed
+1 💚compile0m 23sthe patch passed
+1 💚javac0m 23sthe patch passed
+1 💚javadoc0m 16sthe patch passed
+1 💚shadedjars5m 42spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit19m 47shbase-mapreduce in the patch passed.
40m 26s
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile
GITHUB PR#6496
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 20da97949aa2 5.4.0-200-generic #220-Ubuntu SMP Fri Sep 27 13:19:16 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 1568fc8
Default JavaEclipse Adoptium-17.0.11+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/testReport/
Max. process+thread count3213 (vs. ulimit of 30000)
modulesC: hbase-mapreduce U: hbase-mapreduce
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Nov 28, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeLogfileComment
+0 🆗reexec1m 5sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+1 💚mvninstall3m 2sbranch-2 passed
+1 💚compile0m 22sbranch-2 passed
+1 💚javadoc0m 18sbranch-2 passed
+1 💚shadedjars5m 54sbranch has no errors when building our shaded downstream artifacts.
_ Patch Compile Tests _
+1 💚mvninstall3m 0sthe patch passed
+1 💚compile0m 21sthe patch passed
+1 💚javac0m 21sthe patch passed
+1 💚javadoc0m 17sthe patch passed
+1 💚shadedjars5m 37spatch has no errors when building our shaded downstream artifacts.
_ Other Tests _
+1 💚unit21m 17shbase-mapreduce in the patch passed.
42m 25s
SubsystemReport/Notes
DockerClientAPI=1.47 ServerAPI=1.47 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#6496
Optional Testsjavac javadoc unit shadedjars compile
unameLinux d784ef42e540 5.4.0-195-generic #215-Ubuntu SMP Fri Aug 2 18:28:05 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / 1568fc8
Default JavaEclipse Adoptium-11.0.23+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/testReport/
Max. process+thread count3140 (vs. ulimit of 30000)
modulesC: hbase-mapreduce U: hbase-mapreduce
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6496/1/console
versionsgit=2.34.1 maven=3.9.8
Powered byApache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

@virajjasani
virajjasani merged commit 64a2e73 into apache:branch-2Dec 2, 2024
virajjasani pushed a commit that referenced this pull request Dec 2, 2024
@shubham-roy@virajjasani
HBASE-28328Add an option to count different types of Delete Markers …
4fab1f3
…in RowCounter (#6496) (#6435)
Signed-off-by: Nihal Jain <nihaljain@apache.org>
virajjasani pushed a commit that referenced this pull request Dec 2, 2024
@shubham-roy@virajjasani
HBASE-28328Add an option to count different types of Delete Markers …
a40a0de
…in RowCounter (#6496) (#6435)
Signed-off-by: Nihal Jain <nihaljain@apache.org>
mokai87 pushed a commit to mokai87/hbase that referenced this pull request Aug 7, 2025
@shubham-roy@mokai87
HBASE-28328 Add an option to count different types of Delete Markers …
0f9f9eb
…in RowCounter (apache#6496) (apache#6435)
Signed-off-by: Nihal Jain <nihaljain@apache.org>
sanjeet006py pushed a commit to sanjeet006py/hbase that referenced this pull request Sep 26, 2025
@shubham-roytok_sfci135 tok_sfci135
HBASE-28328 Add an option to count different types of Delete Markers …
f3a12a8
…in RowCounter (apache#6496) (apache#6435)
Signed-off-by: Nihal Jain <nihaljain@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@virajjasanivirajjasaniAwaiting requested review from virajjasani

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shubham-roy@Apache-HBase@virajjasani