Skip to content

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching - #5660

Merged
bbeaudreault merged 4 commits into
apache:masterfrom
HubSpot:HBASE-28338
Feb 5, 2024
Merged

HBASE-28338 Bounded leak of FSDataInputStream buffers from checksum switching#5660
bbeaudreault merged 4 commits into
apache:masterfrom
HubSpot:HBASE-28338

Conversation

@bbeaudreault

@bbeaudreaultbbeaudreault commented Jan 31, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

This also simplifies the code quite a bit, since we no longer need reflection for old unsupported versions. Updated the test to account for the new logic and cover this case.

@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

This comment was marked as outdated.

Sign in to view
@Apache-HBase

Apache-HBase commented Jan 31, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 12sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 43smaster passed
+1 💚compile2m 28smaster passed
+1 💚checkstyle0m 36smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 29smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 43sthe patch passed
+1 💚compile2m 26sthe patch passed
+1 💚javac2m 26sthe patch passed
-0 ⚠️checkstyle0m 36shbase-server: The patch generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck4m 42sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 41spatch has no errors when running spotless:check.
+1 💚spotbugs1m 39sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
27m 6s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5660
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux d7623d3c8220 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaEclipse Adoptium-11.0.17+8
checkstylehttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count81 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 31, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 24sDocker mode activated.
-0 ⚠️yetus0m 2sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 2smaster passed
+1 💚compile0m 48smaster passed
+1 💚shadedjars5m 48sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 58sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚shadedjars5m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit224m 45shbase-server in the patch passed.
249m 13s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5660
Optional Testsjavac javadoc unit shadedjars compile
unameLinux b08c38b8a95d 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/testReport/
Max. process+thread count4712 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 31, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 36sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 31smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars5m 1sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 25smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 27sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars4m 57spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24sthe patch passed
_ Other Tests _
+1 💚unit233m 2shbase-server in the patch passed.
255m 32s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5660
Optional Testsjavac javadoc unit shadedjars compile
unameLinux bf08ae9f76fb 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaTemurin-1.8.0_352-b08
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/testReport/
Max. process+thread count5717 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/3/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 31, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 11sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall2m 43smaster passed
+1 💚compile2m 28smaster passed
+1 💚checkstyle0m 37smaster passed
+1 💚spotless0m 41sbranch has no errors when running spotless:check.
+1 💚spotbugs1m 30smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 46sthe patch passed
+1 💚compile2m 26sthe patch passed
+1 💚javac2m 26sthe patch passed
+1 💚checkstyle0m 36sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck4m 43sPatch does not cause any errors with Hadoop 3.3.6.
+1 💚spotless0m 42spatch has no errors when running spotless:check.
+1 💚spotbugs1m 38sthe patch passed
_ Other Tests _
+1 💚asflicense0m 10sThe patch does not generate ASF License warnings.
26m 52s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#5660
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti spotless checkstyle compile
unameLinux f680d3126af9 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaEclipse Adoptium-11.0.17+8
Max. process+thread count81 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versionsgit=2.34.1 maven=3.8.6 spotbugs=4.7.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 1, 2024

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 26sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 21smaster passed
+1 💚compile0m 47smaster passed
+1 💚shadedjars5m 46sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 26smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 59sthe patch passed
+1 💚compile0m 48sthe patch passed
+1 💚javac0m 48sthe patch passed
+1 💚shadedjars5m 48spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 23sthe patch passed
_ Other Tests _
+1 💚unit229m 8shbase-server in the patch passed.
253m 42s
SubsystemReport/Notes
DockerClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#5660
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 06bc5a6bf7d1 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaEclipse Adoptium-11.0.17+8
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/testReport/
Max. process+thread count4585 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Feb 1, 2024

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 37sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall2m 24smaster passed
+1 💚compile0m 40smaster passed
+1 💚shadedjars5m 1sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 24smaster passed
_ Patch Compile Tests _
+1 💚mvninstall2m 25sthe patch passed
+1 💚compile0m 40sthe patch passed
+1 💚javac0m 40sthe patch passed
+1 💚shadedjars5m 4spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
-1 ❌unit241m 48shbase-server in the patch failed.
264m 6s
SubsystemReport/Notes
DockerClientAPI=1.44 ServerAPI=1.44 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#5660
Optional Testsjavac javadoc unit shadedjars compile
unameLinux a1a7e1e17036 5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / c4e332a
Default JavaTemurin-1.8.0_352-b08
unithttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/testReport/
Max. process+thread count5357 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-5660/4/console
versionsgit=2.34.1 maven=3.8.6
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Humbedooh

Humbedooh commented Feb 1, 2024

Copy link
Copy Markdown
Member

testing jira connections here, ignore this please.

@bbeaudreault
bbeaudreault merged commit 256f10b into apache:masterFeb 5, 2024
@bbeaudreault
bbeaudreault deleted the HBASE-28338 branch February 5, 2024 13:26
bbeaudreault added a commit that referenced this pull request Feb 5, 2024
@bbeaudreault
HBASE-28338Bounded leak of FSDataInputStream buffers from checksum s…
ee6332a
…witching (#5660)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit that referenced this pull request Feb 5, 2024
@bbeaudreault
HBASE-28338Bounded leak of FSDataInputStream buffers from checksum s…
c28e285
…witching (#5660)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
bbeaudreault added a commit that referenced this pull request Feb 5, 2024
@bbeaudreault
HBASE-28338Bounded leak of FSDataInputStream buffers from checksum s…
db8ae95
…witching (#5660)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
szucsvillo pushed a commit to szucsvillo/hbase that referenced this pull request Aug 16, 2024
@petersomogyi@bbeaudreaultGitHub Enterprise
CDPD-71321: HBASE-28338 Bounded leak of FSDataInputStream buffers fro…
d0a418d
…m checksum switching (apache#5660) (apache#91)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit c28e285)
Co-authored-by: Bryan Beaudreault <bbeaudreault@apache.org>
stoty pushed a commit to stoty/hbase that referenced this pull request Nov 26, 2024
@robertkerek-cldr@bbeaudreault
CDPD-71395 HBASE-28338: Bounded leak of FSDataInputStream buffers fro…
f13f49b
…m checksum switching (apache#5660)
(cherry picked from commit c28e285)
Change-Id: I1e644c145d0a93466b1d9e259a565d19e4ec88d1
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Co-authored-by: Bryan Beaudreault <bbeaudreault@apache.org>
(cherry picked from commit bbd848a00eedd0129ac7711e620fe4ec2d903e65)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@bbeaudreault@Apache-HBase@Humbedooh@Apache9