Skip to content

HBASE-27313: Persist list of Hfiles names for which prefetch is done - #4771

Merged
wchevreuil merged 1 commit into
apache:branch-2from
Kota-SH:branch-2
Sep 8, 2022
Merged

HBASE-27313: Persist list of Hfiles names for which prefetch is done#4771
wchevreuil merged 1 commit into
apache:branch-2from
Kota-SH:branch-2

Conversation

@Kota-SH

@Kota-SHKota-SH commented Sep 8, 2022

Copy link
Copy Markdown
Contributor

Jira link: https://issues.apache.org/jira/browse/HBASE-27313

Note:
A patch has already been merged into master, this PR is for branch-2.

Summary:
Persist list of Hfiles names for which prefetch is completed when persist for in-memory bucket cache mapping state is enabled.

We need to persist the Set, which contains a list of all the HFiles for which the prefetch has completed along with the In-memory state of the bucket cache so that when we restart and if any prefetch is scheduled again for the same HFile and we already have the in-memory state retrieved, we can skip the prefetch for it.

@Apache-HBase

Apache-HBase commented Sep 8, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 55sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ branch-2 Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for branch
+1 💚mvninstall2m 32sbranch-2 passed
+1 💚compile2m 58sbranch-2 passed
+1 💚checkstyle0m 34sbranch-2 passed
+1 💚spotless0m 38sbranch has no errors when running spotless:check.
+1 💚spotbugs2m 48sbranch-2 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 9sMaven dependency ordering for patch
+1 💚mvninstall2m 7sthe patch passed
+1 💚compile3m 5sthe patch passed
+1 💚cc3m 5sthe patch passed
+1 💚javac3m 5sthe patch passed
+1 💚checkstyle0m 46sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck10m 18sPatch does not cause any errors with Hadoop 3.2.4 3.3.4.
+1 💚hbaseprotoc1m 4sthe patch passed
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs3m 35sthe patch passed
_ Other Tests _
+1 💚asflicense0m 16sThe patch does not generate ASF License warnings.
38m 59s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#4771
JIRA IssueHBASE-27313
Optional Testsdupname asflicense cc hbaseprotoc spotless prototool javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux ad893e530b64 5.4.0-1071-aws #76~18.04.1-Ubuntu SMP Mon Mar 28 17:49:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e298782
Default JavaAdoptOpenJDK-1.8.0_282-b08
Max. process+thread count64 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 8, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 4sDocker mode activated.
-0 ⚠️yetus0m 6sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+0 🆗mvndep0m 14sMaven dependency ordering for branch
+1 💚mvninstall2m 50sbranch-2 passed
+1 💚compile1m 10sbranch-2 passed
+1 💚shadedjars4m 8sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 32sbranch-2 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 15sMaven dependency ordering for patch
+1 💚mvninstall2m 30sthe patch passed
+1 💚compile1m 8sthe patch passed
+1 💚javac1m 8sthe patch passed
+1 💚shadedjars4m 1spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 30sthe patch passed
_ Other Tests _
+1 💚unit0m 29shbase-protocol-shaded in the patch passed.
+1 💚unit187m 30shbase-server in the patch passed.
208m 47s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#4771
JIRA IssueHBASE-27313
Optional Testsunit javac javadoc shadedjars compile
unameLinux 3fe56dd4a86e 5.4.0-1081-aws #88~18.04.1-Ubuntu SMP Thu Jun 23 16:29:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e298782
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/testReport/
Max. process+thread count2977 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 8, 2022

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 4sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ branch-2 Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for branch
+1 💚mvninstall2m 26sbranch-2 passed
+1 💚compile1m 10sbranch-2 passed
+1 💚shadedjars3m 42sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sbranch-2 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 13sMaven dependency ordering for patch
+1 💚mvninstall2m 9sthe patch passed
+1 💚compile1m 7sthe patch passed
+1 💚javac1m 7sthe patch passed
+1 💚shadedjars3m 41spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 33sthe patch passed
_ Other Tests _
+1 💚unit0m 28shbase-protocol-shaded in the patch passed.
+1 💚unit201m 55shbase-server in the patch passed.
221m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile
GITHUB PR#4771
JIRA IssueHBASE-27313
Optional Testsunit javac javadoc shadedjars compile
unameLinux d013e171a160 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionbranch-2 / e298782
Default JavaAdoptOpenJDK-1.8.0_282-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/testReport/
Max. process+thread count2246 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-server U: .
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-4771/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@wchevreuil
wchevreuil merged commit a1fc483 into apache:branch-2Sep 8, 2022
vinayakphegde pushed a commit to vinayakphegde/hbase that referenced this pull request Apr 4, 2024
@Kota-SH@shankot
CDPD-42936: Backport for HBASE-27313: Persist list of Hfiles names fo…
d3637fa
…r which prefetch is done (apache#4771)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
(cherry picked from commit a1fc483)
Change-Id: I2d9fab1043625919321d2120571dc43327a42118
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@wchevreuilwchevreuilwchevreuil approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Kota-SH@Apache-HBase@wchevreuil