Skip to content

HBASE-26045 Master control the global throughput of all compaction servers - #3538

Closed
nyl3532016 wants to merge 2 commits into
apache:HBASE-25714from
nyl3532016:HBASE-26045
Closed

HBASE-26045 Master control the global throughput of all compaction servers#3538
nyl3532016 wants to merge 2 commits into
apache:HBASE-25714from
nyl3532016:HBASE-26045

Conversation

@nyl3532016

@nyl3532016nyl3532016 commented Jul 28, 2021•
edited
Loading

Copy link
Copy Markdown
Contributor

Compaction servers report compaction tasks to master via heartbeat report. 
The report response has throughput control message and compaction server use this message to adjust compaction throughtput.
We can set total throughtput of all compaction servers via hbase shell commands.

@Apache-HBase

Apache-HBase commented Jul 28, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec2m 9sDocker mode activated.
_ Prechecks _
+1 💚dupname0m 1sNo case conflicting files found.
+0 🆗prototool0m 0sprototool was not available.
+1 💚hbaseanti0m 0sPatch does not have any anti-patterns.
+1 💚@author0m 0sThe patch does not contain any @author tags.
_ HBASE-25714 Compile Tests _
+0 🆗mvndep0m 27sMaven dependency ordering for branch
+1 💚mvninstall5m 21sHBASE-25714 passed
+1 💚compile9m 9sHBASE-25714 passed
+1 💚checkstyle3m 33sHBASE-25714 passed
+1 💚spotbugs10m 40sHBASE-25714 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 12sMaven dependency ordering for patch
+1 💚mvninstall4m 2sthe patch passed
+1 💚compile7m 7sthe patch passed
+1 💚cc7m 7sthe patch passed
+1 💚javac7m 7sthe patch passed
-0 ⚠️checkstyle0m 28shbase-client: The patch generated 1 new + 38 unchanged - 0 fixed = 39 total (was 38)
-0 ⚠️checkstyle1m 11shbase-server: The patch generated 7 new + 6 unchanged - 0 fixed = 13 total (was 6)
-0 ⚠️rubocop0m 29sThe patch generated 20 new + 656 unchanged - 0 fixed = 676 total (was 656)
+1 💚whitespace0m 0sThe patch has no whitespace issues.
+1 💚hadoopcheck20m 24sPatch does not cause any errors with Hadoop 3.1.2 3.2.1 3.3.0.
+1 💚hbaseprotoc2m 57sthe patch passed
+1 💚spotbugs10m 12sthe patch passed
_ Other Tests _
+1 💚asflicense0m 53sThe patch does not generate ASF License warnings.
89m 57s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-general-check/output/Dockerfile
GITHUB PR#3538
Optional Testsdupname asflicense javac spotbugs hadoopcheck hbaseanti checkstyle compile cc hbaseprotoc prototool rubocop
unameLinux 6bcc747bc9d2 4.15.0-147-generic #151-Ubuntu SMP Fri Jun 18 19:21:19 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-25714 / 85f0291
Default JavaAdoptOpenJDK-1.8.0_282-b08
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt
checkstylehttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt
rubocophttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-general-check/output/diff-patch-rubocop.txt
Max. process+thread count86 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/console
versionsgit=2.17.1 maven=3.6.3 spotbugs=4.2.2 rubocop=0.80.0
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 28, 2021

Copy link
Copy Markdown

🎊 +1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 34sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-25714 Compile Tests _
+0 🆗mvndep0m 23sMaven dependency ordering for branch
+1 💚mvninstall4m 40sHBASE-25714 passed
+1 💚compile3m 55sHBASE-25714 passed
+1 💚shadedjars8m 22sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc2m 27sHBASE-25714 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall4m 24sthe patch passed
+1 💚compile3m 48sthe patch passed
+1 💚javac3m 48sthe patch passed
+1 💚shadedjars8m 10spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc2m 27sthe patch passed
_ Other Tests _
+1 💚unit1m 1shbase-protocol-shaded in the patch passed.
+1 💚unit1m 18shbase-client in the patch passed.
+1 💚unit142m 19shbase-server in the patch passed.
+1 💚unit4m 40shbase-thrift in the patch passed.
+1 💚unit7m 17shbase-shell in the patch passed.
199m 9s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile
GITHUB PR#3538
Optional Testsjavac javadoc unit shadedjars compile
unameLinux f83a3a0d36ed 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-25714 / 85f0291
Default JavaAdoptOpenJDK-11.0.10+9
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/testReport/
Max. process+thread count3694 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Jul 28, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 35sDocker mode activated.
-0 ⚠️yetus0m 5sUnprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck
_ Prechecks _
_ HBASE-25714 Compile Tests _
+0 🆗mvndep0m 22sMaven dependency ordering for branch
+1 💚mvninstall4m 25sHBASE-25714 passed
+1 💚compile3m 28sHBASE-25714 passed
+1 💚shadedjars8m 29sbranch has no errors when building our shaded downstream artifacts.
+1 💚javadoc2m 7sHBASE-25714 passed
_ Patch Compile Tests _
+0 🆗mvndep0m 16sMaven dependency ordering for patch
+1 💚mvninstall4m 5sthe patch passed
+1 💚compile3m 29sthe patch passed
+1 💚javac3m 29sthe patch passed
+1 💚shadedjars9m 22spatch has no errors when building our shaded downstream artifacts.
+1 💚javadoc2m 9sthe patch passed
_ Other Tests _
+1 💚unit0m 52shbase-protocol-shaded in the patch passed.
+1 💚unit1m 18shbase-client in the patch passed.
-1 ❌unit155m 15shbase-server in the patch failed.
+1 💚unit4m 14shbase-thrift in the patch passed.
+1 💚unit7m 26shbase-shell in the patch passed.
211m 11s
SubsystemReport/Notes
DockerClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile
GITHUB PR#3538
Optional Testsjavac javadoc unit shadedjars compile
unameLinux 5d39b79c2fbc 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build toolmaven
Personalitydev-support/hbase-personality.sh
git revisionHBASE-25714 / 85f0291
Default JavaAdoptOpenJDK-1.8.0_282-b08
unithttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt
Test Resultshttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/testReport/
Max. process+thread count3760 (vs. ulimit of 30000)
modulesC: hbase-protocol-shaded hbase-client hbase-server hbase-thrift hbase-shell U: .
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/1/console
versionsgit=2.17.1 maven=3.6.3
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

saintstack
saintstack reviewed Aug 10, 2021
View reviewed changes
Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java

/**
* update compaction server total throughput bound
* @param upperBound the total throughput upper bound of all compaction servers

@saintstacksaintstackAug 10, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we talk of compaction servers here, are we talking about the new compaction server facility or are we talking about regionservers?

@nyl3532016nyl3532016Aug 11, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can control compaction throughtput on compactionServers, can not control regionservers

@saintstacksaintstackAug 11, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most deploys will not use a compaction server suite. Is it correct then to talk of compaction servers in our Admin API? Should there be another channel for interacting with compaction servers? A CompactionServersAdmin ?

@nyl3532016nyl3532016Aug 12, 2021

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method talk to master only, compaction servers get throughput control message through periodic heartbeat report

@nyl3532016

nyl3532016 commented Aug 20, 2021

Copy link
Copy Markdown
ContributorAuthor

@Apache9@saintstack Mind help take a look at your convenience.

Apache9
Apache9 reviewed Aug 20, 2021
View reviewed changes

@Apache9Apache9 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the algorithm here does not count for ongoing compactions, instead, just divide the quota evenly across different compaction servers? If there are 10 compaction server, but only one compaction server has ongoing compactions, it still can only use 1/10 of the quota?

Comment threadhbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java Outdated
* @return the now total throughput of all compaction servers
* @throws IOException if a remote or network exception occurs
*/
Map<String, Long> updateCompactionServerTotalThroughput(Long upperBound, Long lowerBound,

@Apache9Apache9Aug 20, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have 'CompactionServer' in the method name, so this global control is only for compaction server? Region server can not have the same limit? And why Long, not long?

@nyl3532016nyl3532016Sep 10, 2021•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, We do not control regionserver's compaction throughtput in this PR (it is OK?), For HRegionserver can adjust their compaction throughtput via update_config shell commend, no need restart server. They implement ConfigurationObserver interface.

@nyl3532016

nyl3532016 commented Sep 10, 2021

Copy link
Copy Markdown
ContributorAuthor

So the algorithm here does not count for ongoing compactions, instead, just divide the quota evenly across different compaction servers? If there are 10 compaction server, but only one compaction server has ongoing compactions, it still can only use 1/10 of the quota?

OK, let me promote the algorithm according to ongoing compactions

@nyl3532016nyl3532016 changed the title HBASE-26045 Master control the global throughtput of all compaction serversHBASE-26045 Master control the global throughput of all compaction serversSep 18, 2021
@Apache-HBase

Apache-HBase commented Sep 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 0sDocker mode activated.
-1 ❌patch0m 5s#3538 does not apply to HBASE-25714. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
SubsystemReport/Notes
GITHUB PR#3538
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/2/console
versionsgit=2.17.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

2 similar comments
@Apache-HBase

Apache-HBase commented Sep 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 0sDocker mode activated.
-1 ❌patch0m 5s#3538 does not apply to HBASE-25714. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
SubsystemReport/Notes
GITHUB PR#3538
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/2/console
versionsgit=2.17.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

@Apache-HBase

Apache-HBase commented Sep 18, 2021

Copy link
Copy Markdown

💔 -1 overall

VoteSubsystemRuntimeComment
+0 🆗reexec0m 0sDocker mode activated.
-1 ❌patch0m 5s#3538 does not apply to HBASE-25714. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help.
SubsystemReport/Notes
GITHUB PR#3538
Console outputhttps://ci-hadoop.apache.org/job/HBase/job/HBase-PreCommit-GitHub-PR/job/PR-3538/2/console
versionsgit=2.17.1
Powered byApache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Apache9Apache9Apache9 left review comments
@saintstacksaintstackAwaiting requested review from saintstack

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nyl3532016@Apache-HBase@saintstack@Apache9