[fix](broker-read) refactor broker reading process to avoid null broker connection by Hastyshell · Pull Request #26050 · apache/doris

Hastyshell · 2023-10-28T06:31:06Z

Proposed changes

issue: #26052
introduced: #16021

Move the broker related RPC into BrokerFileSystem to make the client under protection, and should not expose the connection to the reader.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

github-actions · 2023-10-28T06:49:55Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-28T06:50:26Z

clang-tidy review says "All clean, LGTM! 👍"

Hastyshell · 2023-10-28T06:59:53Z

run buildall

Hastyshell · 2023-10-28T07:02:42Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2023-10-28T07:11:09Z

clang-tidy review says "All clean, LGTM! 👍"

Hastyshell · 2023-10-28T10:40:07Z

run buildall

github-actions · 2023-10-28T10:48:45Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-28T11:23:36Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.17 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162261855 Bytes

…gSiyang2001/doris into avoid-null-broker-connection

Hastyshell · 2023-10-29T16:15:38Z

run buildall

github-actions · 2023-10-29T16:21:15Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-10-29T16:23:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-29T17:07:33Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.93 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162361266 Bytes

github-actions · 2023-11-01T07:46:11Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

github-actions · 2023-11-01T07:58:09Z

PR approved by at least one committer and no changes requested.

…er connection (apache#26050)

…er connection (#26050) (#26349)

…er connection (apache#26050)

…er connection (apache#26050) (apache#26349)

…er connection (apache#26050)

refactor broker reading process
2ecb7f4

github-actionsBot reviewed Oct 28, 2023
View reviewed changes

Comment threadbe/src/io/fs/broker_file_system.cpp Outdated

Hastyshell added the dev/2.0.3 label Oct 28, 2023

Hastyshell force-pushed the avoid-null-broker-connection branch from a4d9289 to 38aa90aCompare October 28, 2023 06:59

format
3213a01

Hastyshell force-pushed the avoid-null-broker-connection branch from 38aa90a to 3213a01Compare October 28, 2023 07:02

github-actionsBot reviewed Oct 28, 2023
View reviewed changes

Comment threadbe/src/io/fs/broker_file_reader.h

Hastyshell added 2 commits October 28, 2023 18:39

refactor broker reading process
cf26897

format
9899a6a

Hastyshell force-pushed the avoid-null-broker-connection branch from 3213a01 to 9899a6aCompare October 28, 2023 10:39

Hastyshell added 2 commits October 30, 2023 00:07

Merge branch 'avoid-null-broker-connection' ofhttps://github.com/Tan…
e35f130
…gSiyang2001/doris into avoid-null-broker-connection

check is alive
53474d0

Hastyshell force-pushed the avoid-null-broker-connection branch from 0f9821b to 53474d0Compare October 29, 2023 16:15

wm1581066 added the p0_c label Oct 30, 2023

wm1581066 requested review from dataroaring and morningman October 30, 2023 03:57

platoneko approved these changes Nov 1, 2023
View reviewed changes

github-actionsBot added the reviewed label Nov 1, 2023

dataroaring approved these changes Nov 1, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2023

dataroaring merged commit 4644191 into apache:masterNov 1, 2023

dataroaring changed the title ~~[refactor](broker-read) refactor broker reading process to avoid null broker connection~~[fix](broker-read) refactor broker reading process to avoid null broker connectionNov 1, 2023

Hastyshell added a commit to Hastyshell/doris that referenced this pull request Nov 3, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
ac07bc3
…er connection (apache#26050)

Hastyshell mentioned this pull request Nov 3, 2023
[cherry-pick](broker-read) refactor broker reading process to avoid null broker connection (#26050) #26349
Merged

xiaokang pushed a commit that referenced this pull request Nov 3, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
b7680fd
…er connection (#26050) (#26349)

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 3, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
ec3c318
…er connection (apache#26050)

seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
03799d6
…er connection (apache#26050)

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
6f3bc6d
…er connection (apache#26050) (apache#26349)

xiaokang mentioned this pull request Dec 4, 2023
Release Note 2.0.3 #27909
Closed

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
[fix](broker-read) refactor broker reading process to avoid null brok…
467e710
…er connection (apache#26050)

Uh oh!

Conversation

Hastyshell commented Oct 28, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot commented Oct 28, 2023

Uh oh!

github-actionsBot commented Oct 28, 2023

Uh oh!

Hastyshell commented Oct 28, 2023

Uh oh!

Hastyshell commented Oct 28, 2023

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actionsBot commented Oct 28, 2023

Uh oh!

Hastyshell commented Oct 28, 2023

Uh oh!

github-actionsBot commented Oct 28, 2023

Uh oh!

doris-robot commented Oct 28, 2023

Uh oh!

Hastyshell commented Oct 29, 2023

Uh oh!

github-actionsBot commented Oct 29, 2023

Uh oh!

github-actionsBot commented Oct 29, 2023

Uh oh!

doris-robot commented Oct 29, 2023

Uh oh!

github-actionsBot commented Nov 1, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Nov 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Hastyshell commented Oct 28, 2023•
edited
Loading