[Improvement] (pipeline) Cancel related query if backend restarts or dead by RoanHeNaN · Pull Request #23863 · apache/doris

RoanHeNaN · 2023-09-04T10:05:03Z

Proposed changes

This is the 2nd pr of #23704 , now we can cancel query if related backend restarts or dead. Privious pr is #23582

RoanHeNaN · 2023-09-04T10:06:58Z

run buildall

github-actions · 2023-09-04T10:11:08Z

clang-tidy review says "All clean, LGTM! 👍"

RoanHeNaN · 2023-09-04T10:13:33Z

run buildall

github-actions · 2023-09-04T10:19:18Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-09-04T11:01:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.55 seconds
stream load tsv: 544 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162062480 Bytes

RoanHeNaN · 2023-09-04T13:49:38Z

run buildall

github-actions · 2023-09-04T13:56:13Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-04T14:43:47Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.32 seconds
stream load tsv: 546 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162061647 Bytes

RoanHeNaN · 2023-09-05T01:39:31Z

run buildall

doris-robot · 2023-09-05T02:13:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.82 seconds
stream load tsv: 548 seconds loaded 74807831229 Bytes, about 130 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162020467 Bytes

RoanHeNaN · 2023-09-05T02:29:32Z

run buildall

github-actions · 2023-09-05T02:34:29Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-05T02:37:27Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-05T03:30:54Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.93 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17161936732 Bytes

RoanHeNaN · 2023-09-05T03:48:03Z

run buildall

github-actions · 2023-09-05T03:54:58Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-05T04:29:19Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 48.45 seconds
stream load tsv: 531 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162061315 Bytes

github-actions · 2023-09-05T07:10:47Z

clang-tidy review says "All clean, LGTM! 👍"

RoanHeNaN · 2023-09-05T07:17:14Z

run FE UT

RoanHeNaN · 2023-09-06T06:27:15Z

run buildall

doris-robot · 2023-09-06T07:09:35Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.68 seconds
stream load tsv: 536 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162120996 Bytes

yiguolei · 2023-09-06T07:25:58Z

+ for (Backend be : currentBackends) {
+ curBeMap.put(be.getId(), be);
+ }
+ lock();


use try (lock()) like RAII in cpp

but here we do not have potential thrown exception? can we use try {lock()} finally {unlock();} either?

yiguolei · 2023-09-06T11:05:06Z

+ }
+ lock();
+
+ for (Long id : idToBackend.keySet()) {


这里为什么要判断这个idToBackend，按道理说直接BackendExecStates 就行了吧？

嗯，这里可以删掉，只用 BackendExecStates 跟 PipelineExecContext 也是可以的

这里为什么要判断这个idToBackend，按道理说直接BackendExecStates 就行了吧？
idToBackend 是用来记录某个 coordinator 在 prepare阶段时候所有 be 信息的 snapshot。感觉我们这里索引数据结构太多了，很容易用乱，需要一个 multi_index container 这种数据结构

github-actions · 2023-09-06T11:51:57Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-06T11:52:47Z

clang-tidy review says "All clean, LGTM! 👍"

RoanHeNaN · 2023-09-06T12:17:59Z

run buildall

doris-robot · 2023-09-06T12:51:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.61 seconds
stream load tsv: 538 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162267662 Bytes

RoanHeNaN · 2023-09-07T03:46:50Z

run buildall

github-actions · 2023-09-07T03:52:50Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-07T04:34:37Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.83 seconds
stream load tsv: 540 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162167310 Bytes

github-actions · 2023-09-08T02:35:58Z

clang-tidy review says "All clean, LGTM! 👍"

RoanHeNaN · 2023-09-08T03:45:46Z

run buildall

github-actions · 2023-09-08T03:52:52Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-08T04:21:13Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.3 seconds
stream load tsv: 533 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162197187 Bytes

yiguolei

LGTM

github-actions · 2023-09-08T11:00:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-08T11:00:50Z

PR approved by anyone and no changes requested.

xinyiZzz

LGTM

RoanHeNaN changed the title ~~[improment] (pipeline) Cancel related query if backend restarts or dead.~~[improvement] (pipeline) Cancel related query if backend restarts or deadSep 4, 2023

RoanHeNaN changed the title ~~[improvement] (pipeline) Cancel related query if backend restarts or dead~~[Improvement] (pipeline) Cancel related query if backend restarts or deadSep 4, 2023

RoanHeNaN force-pushed the cancel-be-pr branch from 3ece65e to be08cd4Compare September 4, 2023 13:49

RoanHeNaN force-pushed the cancel-be-pr branch 2 times, most recently from ea1b7f3 to ea3f858Compare September 5, 2023 02:29

RoanHeNaN force-pushed the cancel-be-pr branch from ea3f858 to 8fd30d6Compare September 5, 2023 03:47

yiguolei reviewed Sep 5, 2023
View reviewed changes

Comment threadbe/src/agent/heartbeat_server.cpp Outdated

yiguolei reviewed Sep 5, 2023
View reviewed changes

Comment threadgensrc/thrift/HeartbeatService.thrift Outdated

RoanHeNaN force-pushed the cancel-be-pr branch from 8fd30d6 to 9d83143Compare September 5, 2023 07:00

RoanHeNaN force-pushed the cancel-be-pr branch from 9d83143 to 45802f5Compare September 6, 2023 06:21

yiguolei added the kind/meta-version-change Categorizes issue or PR as related to changing meta version label Sep 6, 2023

yiguolei reviewed Sep 6, 2023
View reviewed changes

RoanHeNaN force-pushed the cancel-be-pr branch from 68b74d4 to 9ccb355Compare September 6, 2023 11:46

RoanHeNaN force-pushed the cancel-be-pr branch from 9ccb355 to d886d98Compare September 7, 2023 03:46

RoanHeNaN force-pushed the cancel-be-pr branch from d886d98 to b152b2fCompare September 8, 2023 02:28

RoanHeNaN added 4 commits September 8, 2023 11:45

cancel-be need
bbf05cb

cancel need 2
364bcaf

format
ec0be76

modify review
9827147

RoanHeNaN force-pushed the cancel-be-pr branch from b152b2f to 9827147Compare September 8, 2023 03:45

yiguolei approved these changes Sep 8, 2023
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 8, 2023

xinyiZzz approved these changes Sep 8, 2023
View reviewed changes

yiguolei merged commit 5c2f9eb into apache:masterSep 8, 2023

Uh oh!

Conversation

RoanHeNaN commented Sep 4, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

RoanHeNaN commented Sep 4, 2023

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

RoanHeNaN commented Sep 4, 2023

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

hello-stephen commented Sep 4, 2023

Uh oh!

RoanHeNaN commented Sep 4, 2023

Uh oh!

github-actionsBot commented Sep 4, 2023

Uh oh!

doris-robot commented Sep 4, 2023

Uh oh!

RoanHeNaN commented Sep 5, 2023

Uh oh!

doris-robot commented Sep 5, 2023

Uh oh!

RoanHeNaN commented Sep 5, 2023

Uh oh!

github-actionsBot commented Sep 5, 2023

Uh oh!

github-actionsBot commented Sep 5, 2023

Uh oh!

doris-robot commented Sep 5, 2023

Uh oh!

RoanHeNaN commented Sep 5, 2023

Uh oh!

github-actionsBot commented Sep 5, 2023

Uh oh!

Uh oh!

Uh oh!

doris-robot commented Sep 5, 2023

Uh oh!

github-actionsBot commented Sep 5, 2023

Uh oh!

RoanHeNaN commented Sep 5, 2023

Uh oh!

RoanHeNaN commented Sep 6, 2023

Uh oh!

doris-robot commented Sep 6, 2023

Uh oh!

yiguoleiSep 6, 2023

Choose a reason for hiding this comment

Uh oh!

RoanHeNaNSep 6, 2023

Choose a reason for hiding this comment

Uh oh!

yiguoleiSep 6, 2023

Choose a reason for hiding this comment

Uh oh!

RoanHeNaNSep 6, 2023

Choose a reason for hiding this comment

Uh oh!

RoanHeNaNSep 6, 2023

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 6, 2023

Uh oh!

github-actionsBot commented Sep 6, 2023

Uh oh!

RoanHeNaN commented Sep 6, 2023

Uh oh!

doris-robot commented Sep 6, 2023

Uh oh!

RoanHeNaN commented Sep 7, 2023

Uh oh!

github-actionsBot commented Sep 7, 2023

Uh oh!

doris-robot commented Sep 7, 2023

Uh oh!

github-actionsBot commented Sep 8, 2023

RoanHeNaN commented Sep 4, 2023•
edited
Loading