[opt](pipeline) Make pipeline fragment context send_report asynchronized by liutang123 · Pull Request #23142 · apache/doris

liutang123 · 2023-08-17T15:07:41Z

Proposed changes

Issue Number: close #xxx

Add PipelineContextReportExecutor class to do report asyncrinized.
Add FragmentMgr::trigger_pipeline_context_report func to trigger report.
use FragmentMgr::trigger_pipeline_context_report as _report_status_cb of PipelineFragmentContext.
change Status of ReportStatusRequest from reference to constant.
Add exec_status when doing exchange and propagate error info to downstream.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-08-17T15:15:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-17T15:18:25Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-17T15:55:02Z

run build all

github-actions · 2023-08-18T08:20:30Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-18T09:33:21Z

run build all

liutang123 · 2023-08-19T10:23:24Z

run buildall

hello-stephen · 2023-08-19T11:01:50Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.94 seconds
stream load tsv: 541 seconds loaded 74807831229 Bytes, about 131 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 35.2 seconds inserted 10000000 Rows, about 284K ops/s
storage size: 17162169795 Bytes

liutang123 · 2023-08-19T15:38:30Z

run p0

github-actions · 2023-08-19T15:44:44Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-20T10:03:20Z

run p0

github-actions · 2023-08-20T10:10:27Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-21T04:39:03Z

run p0

github-actions · 2023-08-21T04:45:43Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-21T05:54:12Z

run p0

liutang123 · 2023-08-21T13:27:05Z

run p0

github-actions · 2023-08-21T13:33:07Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-21T14:19:44Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-21T14:24:45Z

run p0

github-actions · 2023-08-22T02:32:04Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-08-22T03:14:20Z

run buildall

github-actions · 2023-08-22T03:46:30Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-26T17:11:57Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.99 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162569697 Bytes

liutang123 · 2023-09-27T07:29:34Z

run buildall

liutang123 · 2023-09-27T09:24:34Z

run buildall

doris-robot · 2023-09-27T10:26:43Z

TeamCity be ut coverage result:
Function Coverage: 36.31% (8139/22413)
Line Coverage: 28.47% (65156/228852)
Region Coverage: 27.43% (33784/123155)
Branch Coverage: 24.09% (17244/71574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/341700e70885d82b919ffca74372f9704fb56640_341700e70885d82b919ffca74372f9704fb56640/report/index.html

doris-robot · 2023-09-27T10:52:51Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.52 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162349515 Bytes

liutang123 · 2023-09-27T12:53:01Z

run buildall

github-actions · 2023-09-27T12:56:09Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-27T13:52:46Z

TeamCity be ut coverage result:
Function Coverage: 36.30% (8134/22410)
Line Coverage: 28.45% (65098/228788)
Region Coverage: 27.40% (33728/123101)
Branch Coverage: 24.06% (17214/71532)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3839fc6e7788d659ee599e8664214badfe3baf94_3839fc6e7788d659ee599e8664214badfe3baf94/report/index.html

doris-robot · 2023-09-27T14:20:48Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162059357 Bytes

HappenLee · 2023-09-28T02:20:18Z

 }

+Status RuntimeState::query_status() {
+ auto st = _query_ctx->exec_status();


why RETURN_IF_ERROR(_query_ctx->exec_status());

When canceling pipeline fragment context or updating the status of pipeline fragment context, query context will be changed at the same time.
We use this status to try_close or close sink.

HappenLee

LGTM

github-actions · 2023-09-28T09:29:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-28T09:29:14Z

PR approved by anyone and no changes requested.

Gabriel39

LGTM

…zed (apache#23142)

liutang123 force-pushed the pipeline-sync-report branch from e9c319e to 2b51133Compare August 17, 2023 15:10

liutang123 force-pushed the pipeline-sync-report branch from 2b51133 to 7a855f2Compare August 18, 2023 08:14

yiguolei reviewed Aug 19, 2023
View reviewed changes

Comment threadbe/src/pipeline/pipeline_fragment_context.cpp Outdated

liutang123 force-pushed the pipeline-sync-report branch from 3891684 to 776deddCompare August 21, 2023 13:26

liutang123 self-assigned this Aug 21, 2023

liutang123 force-pushed the pipeline-sync-report branch from 776dedd to ec727e8Compare August 21, 2023 14:12

liutang123 force-pushed the pipeline-sync-report branch from ec727e8 to 977d03dCompare August 22, 2023 02:24

liutang123 force-pushed the pipeline-sync-report branch from 977d03d to 2ac0696Compare August 22, 2023 03:40

liutang123 force-pushed the pipeline-sync-report branch from 2ac0696 to f62d016Compare August 22, 2023 06:46

liutang123 force-pushed the pipeline-sync-report branch from 96cbbc8 to bbc7d5dCompare September 27, 2023 07:03

liutang123 force-pushed the pipeline-sync-report branch from bbc7d5d to 341700eCompare September 27, 2023 08:36

liutang123 added 8 commits September 27, 2023 20:44

pipeline report
3e4cd1a

pipeline report async
793d246

refactor
b779092

fix pipeline x
bc00bc8

fix code style
f90b900

fix compile
0bf7de3

fix memory_order
4b0b246

fix cr
3839fc6

liutang123 force-pushed the pipeline-sync-report branch from 341700e to 3839fc6Compare September 27, 2023 12:45

HappenLee reviewed Sep 28, 2023
View reviewed changes

HappenLee approved these changes Sep 28, 2023
View reviewed changes

github-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 28, 2023

Gabriel39 approved these changes Sep 28, 2023
View reviewed changes

Gabriel39 merged commit 864a0f9 into apache:masterSep 28, 2023

vinlee19 pushed a commit to vinlee19/doris that referenced this pull request Oct 7, 2023
[opt](pipeline) Make pipeline fragment context send_report asynchroni…
b9eb3a7
…zed (apache#23142)

RoanHeNaN mentioned this pull request Dec 19, 2023
[fix](pipeline) Fix query is not cancelled when be report error status #28669
Merged

Uh oh!

Conversation

liutang123 commented Aug 17, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actionsBot commented Aug 17, 2023

Uh oh!

github-actionsBot commented Aug 17, 2023

Uh oh!

liutang123 commented Aug 17, 2023

Uh oh!

github-actionsBot commented Aug 18, 2023

Uh oh!

liutang123 commented Aug 18, 2023

Uh oh!

Uh oh!

liutang123 commented Aug 19, 2023

Uh oh!

hello-stephen commented Aug 19, 2023

Uh oh!

liutang123 commented Aug 19, 2023

Uh oh!

github-actionsBot commented Aug 19, 2023

Uh oh!

liutang123 commented Aug 20, 2023

Uh oh!

github-actionsBot commented Aug 20, 2023

Uh oh!

liutang123 commented Aug 21, 2023

Uh oh!

github-actionsBot commented Aug 21, 2023

Uh oh!

liutang123 commented Aug 21, 2023

Uh oh!

liutang123 commented Aug 21, 2023

Uh oh!

github-actionsBot commented Aug 21, 2023

Uh oh!

github-actionsBot commented Aug 21, 2023

Uh oh!

liutang123 commented Aug 21, 2023

Uh oh!

github-actionsBot commented Aug 22, 2023

Uh oh!

liutang123 commented Aug 22, 2023

Uh oh!

github-actionsBot commented Aug 22, 2023

Uh oh!

doris-robot commented Sep 26, 2023

Uh oh!

liutang123 commented Sep 27, 2023

Uh oh!

liutang123 commented Sep 27, 2023

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

liutang123 commented Sep 27, 2023

Uh oh!

github-actionsBot commented Sep 27, 2023

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

doris-robot commented Sep 27, 2023

Uh oh!

HappenLeeSep 28, 2023

Choose a reason for hiding this comment

Uh oh!

liutang123Sep 28, 2023

Choose a reason for hiding this comment

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 28, 2023

Uh oh!

github-actionsBot commented Sep 28, 2023

liutang123 commented Aug 17, 2023•
edited
Loading