[feature](meta-service) Support querying and adjusting rpc qps limit on meta service by Hastyshell · Pull Request #42413 · apache/doris

Hastyshell · 2024-10-24T10:02:09Z

Proposed changes

Usage

adjust limit

curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params}

Entry	Description
param	uint64 qps_limit
behavior	set qps_limit global default value
example	`curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000`

Entry	Description
param	uint64 qps_limit, string rpc_name
behavior	set RPC specific qps_limit
example	curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster

Entry	Description
param	uint64 qps_limit, string rpc_name, string instance_id
behavior	set instance qps_limit for specific RPC
example	`ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"`

Entry	Description
param	uint64 qps_limit, string instance_id
behavior	set global qps_limit for specific instance
example	`curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"`

query limit

Entry	Description
param	none
behavior	query qps limit for all RPC interface
example	`curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit`

doris-robot · 2024-10-24T10:02:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-24T10:06:29Z

@@ -19,10 +19,13 @@

 #include <brpc/server.h>


warning: 'brpc/server.h' file not found [clang-diagnostic-error]
#include<brpc/server.h> ^

github-actions

clang-tidy made some suggestions

github-actions · 2024-10-25T09:43:52Z

@@ -17,8 +17,11 @@

 #include "rate-limiter/rate_limiter.h"


warning: 'rate-limiter/rate_limiter.h' file not found [clang-diagnostic-error]
#include"rate-limiter/rate_limiter.h" ^

Hastyshell · 2024-10-25T12:12:58Z

run buildall

Hastyshell · 2024-10-25T16:26:44Z

run buildall

Hastyshell · 2024-10-28T04:28:50Z

run buildall

Hastyshell · 2024-10-31T12:39:40Z

run buildall

Hastyshell · 2024-10-31T12:43:15Z

run buildall

github-actions

clang-tidy made some suggestions

Hastyshell · 2024-10-31T13:41:01Z

run buildall

gavinchou · 2024-11-01T14:42:49Z

Add some usage examples in description.

Hastyshell · 2024-11-05T13:16:32Z

run buildall

github-actions

clang-tidy made some suggestions

gavinchou · 2024-11-10T17:30:18Z

+ qps_limit_str, rpc_name, instance_id));
+ };
+
+ static auto parse_qps_limit =


it seems that abuse using lambdas

gavinchou · 2024-11-10T17:31:54Z

+ });
+ };
+
+ // for 8 element in true table of params, register processor cb


this part is confusing

Maybe I'll add some comments, just register different callback for different params pattern, and the "abuse" of lamda are just meant to adapt for this.

Hastyshell · 2024-11-12T12:36:33Z

run buildall

…on meta service

Hastyshell · 2024-11-13T06:20:02Z

run buildall

github-actions · 2024-11-13T08:23:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-11-13T08:23:52Z

PR approved by anyone and no changes requested.

Hastyshell · 2024-11-13T09:09:07Z

run compile

Hastyshell · 2024-11-13T09:09:17Z

run cloud_p0

Hastyshell · 2024-11-13T09:09:22Z

run p0

Hastyshell · 2024-11-13T09:09:30Z

run external

…on meta service (#42413) ## Proposed changes Usage 1. adjust limit ``` curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params} ``` | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit | |behavior | set qps_limit global default value | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name | |behavior | set RPC specific qps_limit | |example|curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name, string instance_id | |behavior | set instance qps_limit for specific RPC | |example|```ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string instance_id | |behavior | set global qps_limit for specific instance | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"```| 2. query limit | Entry | Description | | ----------- | ----------- | | param | none | |behavior | query qps limit for all RPC interface | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit```|

…c qps limit on meta service #42413 (#43884) Cherry-picked from #42413 Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>

…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

…on meta service (apache#42413) ## Proposed changes Usage 1. adjust limit ``` curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?${params} ``` | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit | |behavior | set qps_limit global default value | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name | |behavior | set RPC specific qps_limit | |example|curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string rpc_name, string instance_id | |behavior | set instance qps_limit for specific RPC | |example|```ccurl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&rpc_name=get_cluster&instance_id="doris-0"```| | Entry | Description | | ----------- | ----------- | | param | uint64 qps_limit, string instance_id | |behavior | set global qps_limit for specific instance | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/adjust_rate_limit?qps_limit=5000000&instance_id="doris-0"```| 2. query limit | Entry | Description | | ----------- | ----------- | | param | none | |behavior | query qps limit for all RPC interface | |example|```curl http://ms_ip:ms_port/MetaService/http/v1/list_rate_limit```|

…ng (apache#44362) UT failed with coredump in PR apache#42413 but was not detected by CI. Fix the problem.

github-actionsBot reviewed Oct 24, 2024
View reviewed changes

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 928d926 to 453770fCompare October 25, 2024 09:39

github-actionsBot reviewed Oct 25, 2024
View reviewed changes

Hastyshell force-pushed the dynamic-ms-rate-limit branch 2 times, most recently from cb4d256 to 063ca50Compare October 25, 2024 12:08

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 063ca50 to ebe7ddfCompare October 25, 2024 16:26

Hastyshell force-pushed the dynamic-ms-rate-limit branch from ebe7ddf to e3098abCompare October 28, 2024 04:28

Hastyshell marked this pull request as ready for review October 28, 2024 06:13

dataroaring added dev/3.0.x p0_c labels Oct 29, 2024

Hastyshell marked this pull request as draft October 29, 2024 12:13

Hastyshell force-pushed the dynamic-ms-rate-limit branch from c6532ab to e541fabCompare October 31, 2024 12:42

github-actionsBot reviewed Oct 31, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service_http.cpp
Comment threadcloud/test/meta_service_http_test.cpp

Hastyshell force-pushed the dynamic-ms-rate-limit branch from e541fab to c997f0dCompare October 31, 2024 13:40

Hastyshell marked this pull request as ready for review November 1, 2024 01:53

Hastyshell force-pushed the dynamic-ms-rate-limit branch from c997f0d to 1b6cbeaCompare November 5, 2024 13:16

github-actionsBot reviewed Nov 5, 2024
View reviewed changes

Comment threadcloud/src/meta-service/meta_service_http.cpp

gavinchou reviewed Nov 10, 2024
View reviewed changes

Comment threadcloud/src/rate-limiter/rate_limiter.cpp

gavinchou reviewed Nov 10, 2024
View reviewed changes

Comment threadcloud/src/rate-limiter/rate_limiter.cpp

luwei16 approved these changes Nov 12, 2024
View reviewed changes

Hastyshell requested a review from gavinchou November 12, 2024 12:36

Hastyshell added 3 commits November 13, 2024 14:19

[feature](meta-service) Support querying and adjusting rpc qps limit …
4f7e830
…on meta service

refactor and support adjusting by instance id
6be65b4

fix reviewd problems
70f9607

Hastyshell force-pushed the dynamic-ms-rate-limit branch from 84e350b to 70f9607Compare November 13, 2024 06:19

gavinchou approved these changes Nov 13, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 13, 2024

github-actionsBot added the reviewed label Nov 13, 2024

gavinchou merged commit 4783fc0 into apache:masterNov 13, 2024

github-actionsBot mentioned this pull request Nov 13, 2024
branch-3.0: [feature](meta-service) Support querying and adjusting rpc qps limit on meta service #42413 #43884
Merged

gavinchou pushed a commit that referenced this pull request Nov 14, 2024
branch-3.0: [feature](meta-service) Support querying and adjusting rp…
663845a
…c qps limit on meta service #42413 (#43884) Cherry-picked from #42413 Co-authored-by: Siyang Tang <tangsiyang2001@foxmail.com>

gavinchou added dev/3.0.3-merged and removed dev/3.0.x labels Nov 14, 2024

Hastyshell mentioned this pull request Nov 20, 2024
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setting #44362
Merged
16 tasks

dataroaring pushed a commit that referenced this pull request Nov 21, 2024
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…
012620c
…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

github-actionsBot pushed a commit that referenced this pull request Nov 21, 2024
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…
63cd07d
…ng (#44362) UT failed with coredump in PR #42413 but was not detected by CI. Fix the problem.

gavinchou mentioned this pull request Nov 26, 2024
Release Note 3.0.3 #44522
Closed

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[Fix](ms) Fix ci-uncaught ut coredump due to dynamic rate limit setti…
bf19a57
…ng (apache#44362) UT failed with coredump in PR apache#42413 but was not detected by CI. Fix the problem.

Uh oh!

Conversation

Hastyshell commented Oct 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 24, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 24, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotOct 25, 2024

Choose a reason for hiding this comment

Uh oh!

Hastyshell commented Oct 25, 2024

Uh oh!

Hastyshell commented Oct 25, 2024

Uh oh!

Hastyshell commented Oct 28, 2024

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hastyshell commented Oct 31, 2024

Uh oh!

gavinchou commented Nov 1, 2024

Uh oh!

Hastyshell commented Nov 5, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gavinchouNov 10, 2024

Choose a reason for hiding this comment

Uh oh!

gavinchouNov 10, 2024

Choose a reason for hiding this comment

Uh oh!

HastyshellNov 11, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hastyshell commented Nov 12, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

github-actionsBot commented Nov 13, 2024

Uh oh!

github-actionsBot commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Hastyshell commented Nov 13, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Hastyshell commented Oct 24, 2024•
edited
Loading