Skip to content

[Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread - #16639

Merged
chenlinzhong merged 19 commits into
apache:masterfrom
chenlinzhong:replace_brpc
Feb 22, 2023
Merged

[Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread#16639
chenlinzhong merged 19 commits into
apache:masterfrom
chenlinzhong:replace_brpc

Conversation

@chenlinzhong

@chenlinzhongchenlinzhong commented Feb 11, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close#16634

Problem summary

mainly include:

  1. brpc service adds two types of thread pools. The number of "light" and "heavy" thread pools is different
  2. Classify the interfaces of be. Those related to data transmission are classified as heavy interfaces and others as light interfaces
  3. Add some monitoring to the thread pool, including the queue size and the number of active threads. Use these indicators to guide the configuration of the number of threads

主要包括

  • 1.brpc服务增加2类线程池，"轻","重" 不同线程池的数量配置不一样
  • 2.对be的接口进行分类，跟数据传输有关的归类为重接口，其他为轻接口
  • 3.增加一些监控，包括be每个接口qps和线程池队列大小、活跃线程个数，根据这些指标来指导线程数的配置

image

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions[bot]
github-actionsBot reviewed Feb 11, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
chenlinzhongand others added 8 commits February 11, 2023 23:32
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
400a166
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
481dbc7
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
d0b02cd
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
a72dbb3
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
54f167d
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
31d3928
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong@github-actions
Update be/src/service/internal_service.cpp
77586a2
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@chenlinzhong
fix
56791c2
github-actions[bot]
github-actionsBot reviewed Feb 11, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Feb 11, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/service/internal_service.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Feb 11, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/service/internal_service.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Feb 11, 2023
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/service/internal_service.cpp Outdated
@github-actions

github-actionsBot commented Feb 11, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen

hello-stephen commented Feb 11, 2023•
edited
Loading

Copy link
Copy Markdown
Contributor

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.62 seconds
stream load tsv: 472 seconds loaded 74807831229 Bytes, about 151 MB/s
stream load json: 38 seconds loaded 2358488459 Bytes, about 59 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 29 seconds loaded 861443392 Bytes, about 28 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230220102018_clickbench_pr_100139.html

morningman
morningman reviewed Feb 13, 2023
View reviewed changes
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
Comment threadbe/src/service/internal_service.cpp Outdated
@morningman

morningman commented Feb 13, 2023

Copy link
Copy Markdown
Contributor

Please also update this document:
https://doris.apache.org/zh-CN/docs/dev/admin-manual/maint-monitor/monitor-metrics/metrics

dataroaring
dataroaring requested changes Feb 13, 2023
View reviewed changes
Comment threadbe/src/util/doris_metrics.cpp Outdated
Comment threadbe/src/common/config.h Outdated
Comment threadbe/src/service/internal_service.cpp
eldenmoon
eldenmoon reviewed Feb 13, 2023
View reviewed changes
Comment threadbe/src/service/internal_service.cpp Outdated
@yangzhgyangzhg changed the title [Improvement](brpc) replace brpc bthread with pthread[Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthreadFeb 13, 2023
@github-actions

github-actionsBot commented Feb 13, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actionsgithub-actionsBot added the kind/docs Categorizes issue or PR as related to documentation. label Feb 13, 2023
@chenlinzhong

chenlinzhong commented Feb 13, 2023

Copy link
Copy Markdown
ContributorAuthor

Please also update this document: https://doris.apache.org/zh-CN/docs/dev/admin-manual/maint-monitor/monitor-metrics/metrics

resolved

@github-actions

github-actionsBot commented Feb 13, 2023

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@chenlinzhong

chenlinzhong commented Feb 22, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

3 similar comments
@chenlinzhong

chenlinzhong commented Feb 22, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@chenlinzhong

chenlinzhong commented Feb 22, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@chenlinzhong

chenlinzhong commented Feb 22, 2023

Copy link
Copy Markdown
ContributorAuthor

run p0

@chenlinzhong
chenlinzhong merged commit 0e3be4e into apache:masterFeb 22, 2023
@morningmanmorningman added usercase Important user case type label dev/1.2.3 and removed dev/1.2.2 labels Feb 23, 2023
morningman pushed a commit to morningman/doris that referenced this pull request Feb 23, 2023
@chenlinzhong@morningman
[Improvement](brpc) Using a thread pool for RPC service avoiding std:…
c80cf68
…:mutex block brpc::bthread (apache#16639)
mainly include:
- brpc service adds two types of thread pools. The number of "light" and "heavy" thread pools is different
Classify the interfaces of be. Those related to data transmission are classified as heavy interfaces and others as light interfaces
- Add some monitoring to the thread pool, including the queue size and the number of active threads. Use these
- indicators to guide the configuration of the number of threads
@morningmanmorningman mentioned this pull request Feb 23, 2023
Closed
5 tasks
morningman pushed a commit that referenced this pull request Feb 23, 2023
@chenlinzhong@morningman
[Improvement](brpc) Using a thread pool for RPC service avoiding std:…
8534abc
…:mutex block brpc::bthread (#16639)
mainly include:
- brpc service adds two types of thread pools. The number of "light" and "heavy" thread pools is different
Classify the interfaces of be. Those related to data transmission are classified as heavy interfaces and others as light interfaces
- Add some monitoring to the thread pool, including the queue size and the number of active threads. Use these
- indicators to guide the configuration of the number of threads
morningman added a commit to morningman/doris that referenced this pull request Feb 27, 2023
@morningman
Revert "[Improvement](brpc) Using a thread pool for RPC service avoid…
10b594f
…ing std::mutex block brpc::bthread (apache#16639)"
This reverts commit 8534abc.
morningman added a commit that referenced this pull request Feb 27, 2023
@morningman
Revert "[Improvement](brpc) Using a thread pool for RPC service avoid…
df860e1
…ing std::mutex block brpc::bthread (#16639)" (#17198)
This reverts commit 8534abc.
This PR has some potential dead lock issue, so revert it from 1.2 branch.
yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 9, 2023
@chenlinzhong@yagagagaga
[Improvement](brpc) Using a thread pool for RPC service avoiding std:…
0569ebf
…:mutex block brpc::bthread (apache#16639)
mainly include:
- brpc service adds two types of thread pools. The number of "light" and "heavy" thread pools is different
Classify the interfaces of be. Those related to data transmission are classified as heavy interfaces and others as light interfaces
- Add some monitoring to the thread pool, including the queue size and the number of active threads. Use these - indicators to guide the configuration of the number of threads
morningman added a commit to morningman/doris that referenced this pull request Apr 10, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
23e5c4f
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
@morningmanmorningman mentioned this pull request Apr 10, 2023
Merged
5 tasks
morningman added a commit to morningman/doris that referenced this pull request Apr 13, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
d42b281
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 19, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
0213738
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 19, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
1197254
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 21, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
a39da18
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 21, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
365c8e7
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 21, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
25b4817
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit to morningman/doris that referenced this pull request Apr 21, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs
5592a69
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
morningman added a commit that referenced this pull request Apr 21, 2023
@morningman
[refactor] intro pthread pool for brpc and hadoop libhdfs (#18513)
890a8bb
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (#17435)
@chenlinzhongchenlinzhong mentioned this pull request Apr 24, 2023
Closed
5 tasks
mongo360 pushed a commit to mongo360/doris that referenced this pull request Jul 12, 2023
@morningmancaiconghui1
[refactor] intro pthread pool for brpc and hadoop libhdfs (apache#18513)
0b813a2
1. cherry-pick [Improvement](brpc) Using a thread pool for RPC service avoiding std::mutex block brpc::bthread (apache#16639)
2. cherry-pick [fix](brpc) solve bthread hang problem (apache#17206)
3. cherry-pick [deps](libhdfs) add official hadoop libhdfs for x86 (apache#17435)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman left review comments
@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@eldenmooneldenmooneldenmoon left review comments
@dataroaringdataroaringdataroaring approved these changes
@yinzhijianyinzhijianyinzhijian left review comments

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/1.2.5-mergedkind/docsCategorizes issue or PR as related to documentation.kind/testreviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Enhancement] replace brpc bthread with pthead

6 participants

@chenlinzhong@hello-stephen@morningman@yinzhijian@eldenmoon@dataroaring