Skip to content

HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "… - #2882

Merged
saintstack merged 1 commit into
apache:masterfrom
sunhelly:HBASE-25507
Feb 9, 2021
Merged

HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…#2882
saintstack merged 1 commit into
apache:masterfrom
sunhelly:HBASE-25507

Conversation

@sunhelly

@sunhellysunhelly commented Jan 14, 2021

Copy link
Copy Markdown
Contributor

…java.io.IOException: Invalid HFile block magic"

@Apache-HBase

Apache-HBase commented Jan 14, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 6sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 50smaster passed
+1 💚compile3m 9smaster passed
+1 💚checkstyle1m 4smaster passed
+1 💚spotbugs2m 2smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 29sthe patch passed
+1 💚compile3m 1sthe patch passed
+1 💚javac3m 1sthe patch passed
-0 ⚠️checkstyle1m 2shbase-server: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 8sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 9sthe patch passed
_ Other Tests _
+1 💚asflicense0m 15sThe patch does not generate ASF License warnings.
45m 39s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2882
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 8f189debdc19 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3488c44
Default JavaAdoptOpenJDK-1.8.0_232-b09
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count95 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 14, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 17smaster passed
+1 💚compile1m 10smaster passed
+1 💚shadedjars6m 44sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 4sthe patch passed
+1 💚compile1m 6sthe patch passed
+1 💚javac1m 6sthe patch passed
+1 💚shadedjars6m 42spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41sthe patch passed
_ Other Tests _
+1 💚unit135m 18shbase-server in the patch passed.
163m 20s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 206f8dd1b782 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3488c44
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/testReport/
Max. process+thread count3953 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 14, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 42sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 50smaster passed
+1 💚compile0m 59smaster passed
+1 💚shadedjars8m 47sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 46sthe patch passed
+1 💚compile1m 13sthe patch passed
+1 💚javac1m 13sthe patch passed
+1 💚shadedjars8m 47spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41sthe patch passed
_ Other Tests _
+1 💚unit149m 5shbase-server in the patch passed.
182m 49s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux cd8b5891ed59 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3488c44
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/testReport/
Max. process+thread count4631 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
sunhellyforce-pushed the HBASE-25507 branch 2 times, most recently from 0421634 to ef456abCompareJanuary 15, 2021 04:54
saintstack
saintstack approved these changes Jan 15, 2021
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
sfs.close();
}
} else {
Closeables.close(scanner, true);

@saintstacksaintstackJan 15, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, nice debugging. Good find.

Looking at this, I wonder if other InternalScanners can fail in similar way and leak? I see the InternalScanner is implemented in a few places... in KVHeap, and RegionScanner. Could these have the same problem?

Would it be more clean if the createFileScanners were inside a try/finally? Or will scanners act strangely if closed twice... once by internalscanner on the happy path, and then again in the outer finally?

Otherwise, nice work. Above are nits.... just stuff to consider.

@sunhellysunhellyJan 16, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @saintstack . I check the codes of KVHeap and RegionScanner, they seem OK. Thanks your suggests, I also think createFileScanners should be inside the try/finally. If we only close the scanners when InnerScanner is null, may be we can avoid to close the scanners twice. What do you think?

@saintstacksaintstackFeb 8, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me @sunhelly . Does that mean update to the PR? Thanks.

@sunhellysunhellyFeb 9, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @saintstack , the PR has been updated.
Function createFileScanners closes the store file scanners it created when encounters any exception, so it is out the try/finally. When the InternalScanner which contains all the store file scanners is not null, all the scanners will be processed when it closes. So I added a check for InternalScanner if it is null, if so, close the store file scanners manually in finally.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 5sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 27smaster passed
+1 💚compile3m 8smaster passed
+1 💚checkstyle1m 2smaster passed
+1 💚spotbugs1m 59smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 33sthe patch passed
+1 💚compile3m 21sthe patch passed
+1 💚javac3m 21sthe patch passed
-0 ⚠️checkstyle1m 19shbase-server: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 52sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 24sthe patch passed
_ Other Tests _
+1 💚asflicense0m 12sThe patch does not generate ASF License warnings.
51m 3s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2882
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux a8e40fa19710 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-1.8.0_232-b09
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 32sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 35smaster passed
+1 💚compile1m 13smaster passed
+1 💚shadedjars7m 24sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 47smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 23sthe patch passed
+1 💚compile1m 13sthe patch passed
+1 💚javac1m 13sthe patch passed
+1 💚shadedjars7m 14spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42sthe patch passed
_ Other Tests _
+1 💚unit139m 13shbase-server in the patch passed.
169m 32s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux ccd21220479b 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/testReport/
Max. process+thread count3801 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 4sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 52smaster passed
+1 💚compile0m 58smaster passed
+1 💚shadedjars6m 39sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 38smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 32sthe patch passed
+1 💚compile0m 59sthe patch passed
+1 💚javac0m 59sthe patch passed
+1 💚shadedjars6m 36spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 37sthe patch passed
_ Other Tests _
+1 💚unit143m 27shbase-server in the patch passed.
170m 18s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux c1364522bfb7 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 3cc2468
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/testReport/
Max. process+thread count4973 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/2/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 33sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall4m 30smaster passed
+1 💚compile3m 34smaster passed
+1 💚checkstyle1m 16smaster passed
+1 💚spotbugs2m 21smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 4sthe patch passed
+1 💚compile3m 32sthe patch passed
+1 💚javac3m 32sthe patch passed
-0 ⚠️checkstyle1m 13shbase-server: The patch generated 1 new + 11 unchanged - 0 fixed = 12 total (was 11)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck19m 50sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 17sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
52m 44s
SubsystemReport/Notes
DockerClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2882
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux f4f8a001b8d1 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1474ebc
Default JavaAdoptOpenJDK-1.8.0_232-b09
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 28sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall3m 27smaster passed
+1 💚compile0m 57smaster passed
+1 💚shadedjars6m 38sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 29sthe patch passed
+1 💚compile0m 58sthe patch passed
+1 💚javac0m 58sthe patch passed
+1 💚shadedjars6m 37spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit145m 54shbase-server in the patch passed.
171m 53s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 6dac9d30c7d4 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1474ebc
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/testReport/
Max. process+thread count5269 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 15, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 45sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall5m 18smaster passed
+1 💚compile1m 26smaster passed
+1 💚shadedjars8m 34sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 49smaster passed
_ Patch Compile Tests _
+1 💚mvninstall5m 13sthe patch passed
+1 💚compile1m 26sthe patch passed
+1 💚javac1m 26sthe patch passed
+1 💚shadedjars8m 7spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45sthe patch passed
_ Other Tests _
-1 ❌unit144m 7shbase-server in the patch failed.
178m 43s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 8118db4ebf78 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 1474ebc
Default JavaAdoptOpenJDK-11.0.6+10
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/testReport/
Max. process+thread count3844 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/3/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 16, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 2sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 29smaster passed
+1 💚compile3m 6smaster passed
+1 💚checkstyle1m 3smaster passed
+1 💚spotbugs1m 57smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 24sthe patch passed
+1 💚compile3m 3sthe patch passed
+1 💚javac3m 3sthe patch passed
+1 💚checkstyle1m 5sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 3sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 8sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
45m 4s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2882
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux d5044d1b6931 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 643548f
Default JavaAdoptOpenJDK-1.8.0_232-b09
Max. process+thread count96 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 16, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 27sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 24smaster passed
+1 💚compile1m 10smaster passed
+1 💚shadedjars7m 20sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 26sthe patch passed
+1 💚compile1m 15sthe patch passed
+1 💚javac1m 15sthe patch passed
+1 💚shadedjars7m 16spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 43sthe patch passed
_ Other Tests _
-1 ❌unit134m 15shbase-server in the patch failed.
164m 10s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 822d298b37e0 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 643548f
Default JavaAdoptOpenJDK-11.0.6+10
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/testReport/
Max. process+thread count4166 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 16, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 37sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 38smaster passed
+1 💚compile1m 7smaster passed
+1 💚shadedjars9m 1sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 42smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 33sthe patch passed
+1 💚compile1m 7sthe patch passed
+1 💚javac1m 7sthe patch passed
+1 💚shadedjars8m 32spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 36sthe patch passed
_ Other Tests _
+1 💚unit149m 41shbase-server in the patch passed.
182m 46s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 5ab913b7e670 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / 643548f
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/testReport/
Max. process+thread count4064 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/4/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@sunhelly
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
30b9111
…java.io.IOException: Invalid HFile block magic"
@sunhelly

sunhelly commented Jan 18, 2021

Copy link
Copy Markdown
ContributorAuthor

getScannersForCompaction close scanners when it encounters exception, so let createFileScanners outside the try/finally.

ramkrish86
ramkrish86 reviewed Jan 18, 2021
View reviewed changes
Comment threadhbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java
// and then scanner will be null, but scanners for all the hfiles should be closed,
// or else we will find leak of ESTABLISHED sockets.
if (scanner == null) {
for (StoreFileScanner sfs : scanners) {

@ramkrish86ramkrish86Jan 18, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is nice find.

@Apache-HBase

Apache-HBase commented Jan 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec1m 0sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 0sNo case conflicting files found.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ master Compile Tests _
+1 💚mvninstall3m 28smaster passed
+1 💚compile3m 6smaster passed
+1 💚checkstyle1m 3smaster passed
+1 💚spotbugs1m 58smaster passed
_ Patch Compile Tests _
+1 💚mvninstall3m 26sthe patch passed
+1 💚compile3m 8sthe patch passed
+1 💚javac3m 8sthe patch passed
+1 💚checkstyle1m 5sthe patch passed
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck17m 6sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚spotbugs2m 6sthe patch passed
_ Other Tests _
+1 💚asflicense0m 13sThe patch does not generate ASF License warnings.
45m 2s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#2882
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile
unameLinux 03c3895de8ee 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / d5debe1
Default JavaAdoptOpenJDK-1.8.0_232-b09
Max. process+thread count95 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=3.1.12
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec3m 58sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 16smaster passed
+1 💚compile1m 11smaster passed
+1 💚shadedjars7m 20sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 18sthe patch passed
+1 💚compile1m 10sthe patch passed
+1 💚javac1m 10sthe patch passed
+1 💚shadedjars7m 10spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 44sthe patch passed
_ Other Tests _
+1 💚unit139m 19shbase-server in the patch passed.
172m 13s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 4badac2a13e3 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / d5debe1
Default JavaAdoptOpenJDK-11.0.6+10
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/testReport/
Max. process+thread count3752 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jan 18, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 3sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ master Compile Tests _
+1 💚mvninstall4m 16smaster passed
+1 💚compile1m 8smaster passed
+1 💚shadedjars8m 56sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 41smaster passed
_ Patch Compile Tests _
+1 💚mvninstall4m 26sthe patch passed
+1 💚compile1m 12sthe patch passed
+1 💚javac1m 12sthe patch passed
+1 💚shadedjars8m 42spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc0m 45sthe patch passed
_ Other Tests _
+1 💚unit147m 5shbase-server in the patch passed.
180m 0s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#2882
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 2b9b652c1e0a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionmaster / d5debe1
Default JavaAdoptOpenJDK-1.8.0_232-b09
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/testReport/
Max. process+thread count4608 (vs. ulimit of 30000)
modulesC: hbase-server U: hbase-server
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-2882/5/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@saintstack
saintstack merged commit 0353909 into apache:masterFeb 9, 2021
saintstack pushed a commit that referenced this pull request Feb 9, 2021
@sunhelly@saintstack
HBASE-25507Leak of ESTABLISHED sockets when compaction encountered "…
e332c94
…java.io.IOException: Invalid HFile block magic" (#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
saintstack pushed a commit that referenced this pull request Feb 9, 2021
@sunhelly@saintstack
HBASE-25507Leak of ESTABLISHED sockets when compaction encountered "…
6849279
…java.io.IOException: Invalid HFile block magic" (#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
saintstack pushed a commit that referenced this pull request Feb 9, 2021
@sunhelly@saintstack
HBASE-25507Leak of ESTABLISHED sockets when compaction encountered "…
6d4127e
…java.io.IOException: Invalid HFile block magic" (#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
ddupg pushed a commit to ddupg/hbase that referenced this pull request Feb 19, 2021
@sunhellysunxin
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
adad29e
…java.io.IOException: Invalid HFile block magic" (apache#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
wchevreuil pushed a commit to wchevreuil/hbase that referenced this pull request Feb 24, 2021
@sunhelly@wchevreuil
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "…
b6018d1
…java.io.IOException: Invalid HFile block magic" (apache#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@saintstacksaintstacksaintstack approved these changes
@ramkrish86ramkrish86ramkrish86 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sunhelly@Apache-HBase@saintstack@ramkrish86