Uh oh!
There was an error while loading. Please reload this page.
HBASE-23956 Use less resources running tests - #1266
Merged
Merged
Conversation
Apache-HBase
commented
Mar 10, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 10, 2020
Apache-HBase
commented
Mar 10, 2020
💔 -1 overall
This message was automatically generated. |
Apache9
requested changes
Mar 10, 2020
| @@ -31,10 +30,16 @@ | |||
| */ | |||
| @InterfaceAudience.Private | |||
| class DefaultNettyEventLoopConfig { | |||
Contributor
There was a problem hiding this comment.
This is not the design...
The design is to create a new EventLoop and make the code use it.
Change a configuration to change the 'default' instance and then make the code use the 'default' instance seems strange...
And also another anti pattern is that, we even do not have a Configuration instance here...
Contributor
There was a problem hiding this comment.
Or let's just remove this class. Move the creation of event loop into the NettyRpcClientConfigHelper directly and honor the hbase.netty.worker.count configuration when creating? Though a bit strange is that the Configuration passed in may change but at least it can avoid reading the configuration from system property...
saintstack
commented
Mar 11, 2020
ContributorAuthor
Ok. I think I sorted it. Command-line arg wasn't so bad given we were passing generic for netty anyways but this is a bit better. |
Apache-HBase
commented
Mar 11, 2020
💔 -1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 11, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 11, 2020
🎊 +1 overall
This message was automatically generated. |
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests
saintstack
commented
Mar 11, 2020
ContributorAuthor
Fix checkstyle and the test failure (related) |
Apache-HBase
commented
Mar 11, 2020
🎊 +1 overall
This message was automatically generated. |
Apache-HBase
commented
Mar 11, 2020
🎊 +1 overall
This message was automatically generated. |
saintstack
commented
Mar 11, 2020
ContributorAuthor
Ok. Let me merge this. See if it helps w/ the OOME we are seeing around findbugs/spotbugs. Shout if you want amendments to the change you asked me do @Apache9 Thanks. |
asfgit pushed a commit
that referenced
this pull request
Mar 11, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
Apache-HBase
commented
Mar 11, 2020
💔 -1 overall
This message was automatically generated. |
asfgit pushed a commit
that referenced
this pull request
Mar 11, 2020
Addendum to cleanup pom warning.
asfgit pushed a commit
that referenced
this pull request
Mar 11, 2020
Addendum to cleanup pom warning.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestMultiParallel thread counts were off after parent change.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
asfgit pushed a commit
that referenced
this pull request
Mar 12, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
asfgit pushed a commit
that referenced
this pull request
Apr 15, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
asfgit pushed a commit
that referenced
this pull request
Apr 15, 2020
Addendum to cleanup pom warning.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum to cleanup pom warning.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum to cleanup pom warning.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
thangTang pushed a commit
to thangTang/hbase
that referenced
this pull request
Apr 16, 2020
Addendum: TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
ddupg pushed a commit
to ddupg/hbase
that referenced
this pull request
Apr 20, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
ddupg pushed a commit
to ddupg/hbase
that referenced
this pull request
Apr 20, 2020
Addendum to cleanup pom warning.
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Add being able to configure netty thread counts. Enable socket reuse (should not have any impact). hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java Rename the threads we create in here so they are NOT named same was threads created by Hadoop RPC. hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java Allow configuring eventloopgroup thread count (so can override for tests) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java Enable socket resuse. hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java Enable socket resuse and config for how many threads to use. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java Thread name edit; drop the redundant 'Thread' suffix. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java Make closeable and shutdown executor when called. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java Call close on HFileReplicator hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java HDFS creates lots of threads. Use less of it so less threads overall. hbase-server/src/test/resources/hbase-site.xml hbase-server/src/test/resources/hdfs-site.xml Constrain resources when running in test context. hbase-server/src/test/resources/log4j.properties Enable debug on netty to see netty configs in our log pom.xml Add system properties when we launch JVMs to constrain thread counts in tests Signed-off-by: Duo Zhang <zhangduo@apache.org>
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…o cleanup pom warning.
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…This test was depending on hard-coded number for log cleaners; the parent patch made them less in test context.
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
Addendum: Two more tests that had issues because parent cut down the number of default RegionServers launched.
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…TestMultiParallel thread counts were off after parent change.
infraio pushed a commit
to infraio/hbase
that referenced
this pull request
Aug 17, 2020
…TestNamespaceCommands was using up the limit on nioeventloopgroup.... so create them per run of client.
sanjeet006py pushed a commit
to sanjeet006py/hbase
that referenced
this pull request
Sep 26, 2025
* @W-17450732 Add jacoco argline prop * Upgrade jacoco verison
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
Rename the threads we create in here so they are NOT named same was
threads created by Hadoop RPC.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
Allow configuring eventloopgroup thread count (so can override for
tests)
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
Enable socket resuse.
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
Enable socket resuse and config for how many threads to use.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
Thread name edit; drop the redundant 'Thread' suffix.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
Make closeable and shutdown executor when called.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
Call close on HFileReplicator
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
HDFS creates lots of threads. Use less of it so less threads overall.
hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
Constrain resources when running in test context.
hbase-server/src/test/resources/log4j.properties
Enable debug on netty to see netty configs in our log
pom.xml
Add system properties when we launch JVMs to constrain thread counts in
tests