Skip to content

[fix](pipeline) Fix query hang if limited rows is reached - #35466

Merged
yiguolei merged 1 commit into
apache:masterfrom
Gabriel39:fix_0527
May 28, 2024
Merged

[fix](pipeline) Fix query hang if limited rows is reached#35466
yiguolei merged 1 commit into
apache:masterfrom
Gabriel39:fix_0527

Conversation

@Gabriel39

@Gabriel39Gabriel39 commented May 27, 2024•
edited by yiguolei
Loading

Copy link
Copy Markdown
Contributor

Proposed changes

Some operators has limit condition, the source operator should notify the sink operator that limit reached.
Although FE has limit logic but it not always send .

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39

Gabriel39 commented May 27, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented May 27, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40258 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11cd4d6af44510695e3652679fd3d1bcddd66f03, data reload: false
------ Round 1 ----------------------------------
q1	17591	4620	4276	4276
q2	2021	193	183	183
q3	10886	1218	1245	1218
q4	10505	799	842	799
q5	7646	2741	2665	2665
q6	227	137	142	137
q7	972	606	605	605
q8	9278	2131	2144	2131
q9	9089	6674	6641	6641
q10	9233	3754	3739	3739
q11	464	236	228	228
q12	487	225	224	224
q13	17776	2966	3039	2966
q14	256	215	230	215
q15	516	472	470	470
q16	482	389	398	389
q17	978	740	695	695
q18	8124	7396	7489	7396
q19	3469	1473	1542	1473
q20	650	304	303	303
q21	5027	3897	3233	3233
q22	342	272	278	272
Total cold run time: 116019 ms
Total hot run time: 40258 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4359	4211	4236	4211
q2	373	265	269	265
q3	2999	2726	2709	2709
q4	1857	1599	1572	1572
q5	5270	5333	5338	5333
q6	207	126	128	126
q7	2143	1748	1786	1748
q8	3173	3343	3343	3343
q9	8488	8456	8416	8416
q10	3901	3692	3706	3692
q11	576	486	477	477
q12	807	625	594	594
q13	17463	2969	2976	2969
q14	295	248	282	248
q15	509	483	489	483
q16	494	413	410	410
q17	1801	1478	1473	1473
q18	7765	7595	7512	7512
q19	1668	1575	1595	1575
q20	2028	1787	1778	1778
q21	5043	4736	4767	4736
q22	573	490	515	490
Total cold run time: 71792 ms
Total hot run time: 54160 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 173466 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11cd4d6af44510695e3652679fd3d1bcddd66f03, data reload: false
query1	935	387	376	376
query2	6444	2509	2385	2385
query3	6662	204	205	204
query4	19888	17506	17440	17440
query5	4168	445	447	445
query6	269	170	159	159
query7	4586	301	296	296
query8	248	187	196	187
query9	8513	2365	2374	2365
query10	464	286	259	259
query11	10526	10092	9889	9889
query12	136	91	85	85
query13	1633	367	370	367
query14	8655	7250	7621	7250
query15	210	174	166	166
query16	7803	262	274	262
query17	1841	537	539	537
query18	1970	277	282	277
query19	220	152	154	152
query20	93	87	89	87
query21	194	128	134	128
query22	4304	4036	3960	3960
query23	33713	33213	33148	33148
query24	11981	2941	2924	2924
query25	671	368	369	368
query26	1821	159	160	159
query27	3015	329	343	329
query28	7704	2001	2011	2001
query29	1111	606	621	606
query30	290	148	151	148
query31	958	791	741	741
query32	84	54	56	54
query33	762	275	262	262
query34	1059	494	491	491
query35	758	609	607	607
query36	1059	927	930	927
query37	294	66	75	66
query38	2906	2803	2743	2743
query39	887	810	798	798
query40	281	127	127	127
query41	45	44	41	41
query42	104	97	99	97
query43	609	550	576	550
query44	1217	730	753	730
query45	177	168	160	160
query46	1089	746	747	746
query47	1854	1755	1785	1755
query48	377	307	297	297
query49	1204	390	395	390
query50	782	408	397	397
query51	6873	6732	6733	6732
query52	103	94	95	94
query53	366	290	299	290
query54	982	442	424	424
query55	78	73	73	73
query56	264	279	252	252
query57	1149	1052	1069	1052
query58	239	216	216	216
query59	3354	3338	3449	3338
query60	286	270	258	258
query61	93	85	84	84
query62	654	460	441	441
query63	315	292	294	292
query64	9915	2182	1738	1738
query65	3212	3180	3199	3180
query66	1390	335	343	335
query67	15153	14878	14890	14878
query68	4572	538	531	531
query69	461	279	281	279
query70	1156	1061	1145	1061
query71	415	283	270	270
query72	7661	5517	5393	5393
query73	734	325	325	325
query74	6025	5593	5599	5593
query75	3492	2656	2653	2653
query76	2924	1020	1014	1014
query77	617	269	278	269
query78	10271	9745	9793	9745
query79	2191	515	536	515
query80	1647	449	449	449
query81	516	224	220	220
query82	785	98	97	97
query83	308	170	169	169
query84	260	89	90	89
query85	1573	292	365	292
query86	471	303	321	303
query87	3252	3133	3084	3084
query88	4194	2422	2433	2422
query89	480	401	394	394
query90	1992	205	190	190
query91	124	98	97	97
query92	59	50	50	50
query93	2122	509	497	497
query94	1267	197	191	191
query95	407	316	320	316
query96	595	275	270	270
query97	3204	3006	3011	3006
query98	251	221	223	221
query99	1243	846	858	846
Total cold run time: 276269 ms
Total hot run time: 173466 ms

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.42 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11cd4d6af44510695e3652679fd3d1bcddd66f03, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.07	0.07
query5	0.49	0.49	0.48
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.53	0.48	0.50
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.14	0.13	0.12
query13	0.59	0.59	0.59
query14	0.76	0.81	0.77
query15	0.83	0.81	0.81
query16	0.38	0.36	0.37
query17	0.93	1.03	1.00
query18	0.21	0.23	0.24
query19	1.76	1.71	1.72
query20	0.02	0.01	0.01
query21	15.71	0.65	0.66
query22	4.09	7.21	1.83
query23	18.32	1.34	1.25
query24	1.90	0.22	0.22
query25	0.14	0.09	0.08
query26	0.25	0.17	0.17
query27	0.08	0.08	0.08
query28	13.27	1.01	0.99
query29	13.16	3.27	3.34
query30	0.24	0.06	0.06
query31	2.89	0.38	0.38
query32	3.27	0.47	0.47
query33	2.90	2.91	2.88
query34	17.27	4.40	4.44
query35	4.51	4.47	4.48
query36	0.65	0.46	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.92 s
Total hot run time: 30.42 s

@doris-robot

doris-robot commented May 27, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 35.75% (9007/25193)
Line Coverage: 27.36% (74547/272493)
Region Coverage: 26.57% (38562/145111)
Branch Coverage: 23.44% (19665/83892)
Coverage Report: http://coverage.selectdb-in.cc/coverage/11cd4d6af44510695e3652679fd3d1bcddd66f03_11cd4d6af44510695e3652679fd3d1bcddd66f03/report/index.html

yiguolei
yiguolei approved these changes May 28, 2024
View reviewed changes

@yiguoleiyiguolei left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label May 28, 2024
@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented May 28, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguolei
yiguolei merged commit 85c011d into apache:masterMay 28, 2024
yiguolei pushed a commit that referenced this pull request May 28, 2024
@Gabriel39@Doris-Extras
[fix](pipeline) Fix query hang if limited rows is reached (#35466)
2310915
## Proposed changes
Some operators has limit condition, the source operator should notify
the sink operator that limit reached.
Although FE has limit logic but it not always send .
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
@Gabriel39Gabriel39 mentioned this pull request May 28, 2024
Merged
dataroaring pushed a commit that referenced this pull request May 28, 2024
@Gabriel39@dataroaring
[fix](pipeline) Fix query hang if limited rows is reached (#35466)
64482b2
## Proposed changes
Some operators has limit condition, the source operator should notify
the sink operator that limit reached.
Although FE has limit logic but it not always send .
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
Gabriel39 added a commit that referenced this pull request May 31, 2024
@Gabriel39
[fix](pipeline) Fix query hang up if limited rows is reached (#35513)
a11a590
Follow-up for #35466.
We should assure closed tasks will not block other tasks.
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request May 31, 2024
@Gabriel39
[fix](pipeline) Fix query hang up if limited rows is reached (apache#…
2ee88bd
…35513)
Follow-up for apache#35466.
We should assure closed tasks will not block other tasks.
@Gabriel39Gabriel39 mentioned this pull request May 31, 2024
Merged
yiguolei pushed a commit that referenced this pull request May 31, 2024
@Gabriel39
[fix](pipeline) Fix query hang up if limited rows is reached (#35513) (…
b864aa7
…#35746)
Follow-up for #35466.
We should assure closed tasks will not block other tasks.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@Gabriel39@dataroaring
[fix](pipeline) Fix query hang up if limited rows is reached (#35513)
5b002b0
Follow-up for #35466.
We should assure closed tasks will not block other tasks.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@Gabriel39@seawinde
[fix](pipeline) Fix query hang up if limited rows is reached (apache#…
a43ecaa
…35513)
Follow-up for apache#35466.
We should assure closed tasks will not block other tasks.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Gabriel39@HappenLee
[fix](pipeline) Fix query hang if limited rows is reached (apache#35466)
0c08f48
## Proposed changes
Some operators has limit condition, the source operator should notify
the sink operator that limit reached.
Although FE has limit logic but it not always send .
## Further comments
If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@Gabriel39@HappenLee
[fix](pipeline) Fix query hang up if limited rows is reached (apache#…
8b540c6
…35513)
Follow-up for apache#35466.
We should assure closed tasks will not block other tasks.
@Gabriel39
Gabriel39 deleted the fix_0527 branch July 30, 2026 06:27
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDtBiteTheDDDDtBiteTheDDDDt approved these changes
@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.0-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Gabriel39@doris-robot@BiteTheDDDDt@yiguolei@dataroaring