Skip to content

[Bug](partition_topn) fix partition_topn not reset output rows after do_partition_topn_sort - #40761

Merged
yiguolei merged 2 commits into
apache:masterfrom
zhangstar333:test_branch
Sep 14, 2024
Merged

[Bug](partition_topn) fix partition_topn not reset output rows after do_partition_topn_sort#40761
yiguolei merged 2 commits into
apache:masterfrom
zhangstar333:test_branch

Conversation

@zhangstar333

@zhangstar333zhangstar333 commented Sep 12, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

should reset all variables about output rows after do_partition_topn_sort and get_next from sorter
if we not reset all variables. it's will be regard as have output many rows
but the sorter still have remaing many data need to output.

@doris-robot

doris-robot commented Sep 12, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333

zhangstar333 commented Sep 12, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Sep 12, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.89% (9461/25647)
Line Coverage: 28.26% (77808/275354)
Region Coverage: 27.66% (40170/145241)
Branch Coverage: 24.27% (20416/84110)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8fc334f6ef2f300f81f80afdc887f4293c4c977f_8fc334f6ef2f300f81f80afdc887f4293c4c977f/report/index.html

@doris-robot

doris-robot commented Sep 12, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 43020 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fc334f6ef2f300f81f80afdc887f4293c4c977f, data reload: false
------ Round 1 ----------------------------------
q1	17611	7316	7234	7234
q2	2053	187	184	184
q3	10637	1381	1384	1381
q4	10233	949	1024	949
q5	8697	3215	3176	3176
q6	248	156	150	150
q7	1062	641	641	641
q8	9743	2061	2024	2024
q9	6937	6356	6443	6356
q10	7271	2537	2512	2512
q11	440	247	248	247
q12	402	220	224	220
q13	17753	3019	2994	2994
q14	288	248	256	248
q15	584	538	516	516
q16	516	438	433	433
q17	992	937	956	937
q18	7428	6798	6804	6798
q19	1394	1233	1229	1229
q20	620	342	326	326
q21	3871	3504	3492	3492
q22	1097	983	973	973
Total cold run time: 109877 ms
Total hot run time: 43020 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7181	7206	7217	7206
q2	362	259	240	240
q3	3086	3086	3089	3086
q4	2071	2074	1978	1978
q5	5652	5598	5719	5598
q6	246	154	152	152
q7	2217	1797	1765	1765
q8	3357	3472	3396	3396
q9	8729	8753	8784	8753
q10	3526	3579	3623	3579
q11	567	496	492	492
q12	803	617	623	617
q13	5774	3219	3219	3219
q14	305	293	282	282
q15	588	544	556	544
q16	500	473	471	471
q17	1825	1800	1759	1759
q18	8458	7979	7774	7774
q19	1734	1761	1730	1730
q20	2098	1868	1873	1868
q21	5744	5637	5570	5570
q22	1091	1048	1005	1005
Total cold run time: 65914 ms
Total hot run time: 61084 ms

@yiguoleiyiguolei added dev/2.1.x usercase Important user case type label p0_b dev/3.0.x labels Sep 12, 2024
@zhangstar333zhangstar333 mentioned this pull request Sep 13, 2024
Merged
@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 43060 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8fc334f6ef2f300f81f80afdc887f4293c4c977f, data reload: false
------ Round 1 ----------------------------------
q1	17623	7383	7208	7208
q2	2050	190	187	187
q3	10457	1331	1377	1331
q4	10157	970	980	970
q5	7722	3227	3154	3154
q6	244	155	158	155
q7	1052	641	628	628
q8	9461	2090	2010	2010
q9	8096	6318	6359	6318
q10	7071	2525	2569	2525
q11	453	253	253	253
q12	434	229	231	229
q13	17761	3038	3043	3038
q14	290	244	262	244
q15	578	518	521	518
q16	549	433	431	431
q17	996	968	957	957
q18	7583	6876	6792	6792
q19	1380	1254	1245	1245
q20	604	353	338	338
q21	3991	3581	3553	3553
q22	1077	976	981	976
Total cold run time: 109629 ms
Total hot run time: 43060 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7207	7178	7208	7178
q2	341	237	235	235
q3	2956	2922	2884	2884
q4	1987	1945	1966	1945
q5	5495	5457	5462	5457
q6	237	146	151	146
q7	2067	1691	1691	1691
q8	3275	3319	3335	3319
q9	8479	8542	8442	8442
q10	3405	3456	3460	3456
q11	576	477	491	477
q12	756	597	585	585
q13	10220	3033	3064	3033
q14	290	279	265	265
q15	567	537	522	522
q16	495	460	461	460
q17	1771	1720	1718	1718
q18	8059	7567	7663	7567
q19	1736	1711	1706	1706
q20	2068	1816	1842	1816
q21	5629	5507	5407	5407
q22	1113	982	1002	982
Total cold run time: 68729 ms
Total hot run time: 59291 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196225 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8fc334f6ef2f300f81f80afdc887f4293c4c977f, data reload: false
query1	838	391	381	381
query2	6010	1691	1703	1691
query3	6327	210	217	210
query4	27624	24137	24226	24137
query5	3362	542	553	542
query6	274	171	172	171
query7	4450	304	301	301
query8	288	217	211	211
query9	8308	2581	2610	2581
query10	449	288	287	287
query11	16079	15784	15548	15548
query12	153	103	107	103
query13	1699	414	391	391
query14	9729	7044	7252	7044
query15	210	174	173	173
query16	6850	480	468	468
query17	1085	594	590	590
query18	1879	302	297	297
query19	198	147	155	147
query20	122	112	111	111
query21	175	116	108	108
query22	4373	4248	4322	4248
query23	34825	33889	34010	33889
query24	9449	3135	3088	3088
query25	636	418	409	409
query26	1198	166	165	165
query27	2546	291	291	291
query28	5926	2093	2086	2086
query29	865	429	438	429
query30	273	165	163	163
query31	1004	772	806	772
query32	94	59	58	58
query33	664	311	316	311
query34	908	494	493	493
query35	877	718	747	718
query36	1047	916	896	896
query37	155	87	89	87
query38	3951	3934	3897	3897
query39	1518	1438	1426	1426
query40	264	123	118	118
query41	50	48	49	48
query42	124	104	102	102
query43	498	424	432	424
query44	1202	799	782	782
query45	198	169	172	169
query46	1110	833	828	828
query47	1855	1760	1795	1760
query48	378	296	295	295
query49	865	468	460	460
query50	942	443	447	443
query51	7141	6899	6960	6899
query52	106	94	92	92
query53	269	190	188	188
query54	666	488	477	477
query55	79	81	82	81
query56	315	284	292	284
query57	1218	1089	1076	1076
query58	264	252	254	252
query59	2769	2736	2660	2660
query60	321	294	294	294
query61	127	127	246	127
query62	908	677	679	677
query63	223	191	192	191
query64	4525	698	672	672
query65	3284	3194	3186	3186
query66	1247	299	291	291
query67	15966	15810	15504	15504
query68	2923	586	603	586
query69	413	269	279	269
query70	1197	1091	1081	1081
query71	344	288	281	281
query72	5377	4021	4066	4021
query73	770	334	330	330
query74	9315	9066	9087	9066
query75	3346	2733	2738	2733
query76	1585	1356	1336	1336
query77	438	330	325	325
query78	9902	9376	9653	9376
query79	1143	905	895	895
query80	1012	829	818	818
query81	545	273	268	268
query82	847	267	257	257
query83	219	190	191	190
query84	250	113	107	107
query85	769	410	396	396
query86	451	308	316	308
query87	4363	4351	4297	4297
query88	4161	4072	4076	4072
query89	387	376	377	376
query90	1629	322	354	322
query91	126	126	124	124
query92	79	79	75	75
query93	1073	1068	1032	1032
query94	831	385	392	385
query95	510	439	439	439
query96	474	467	475	467
query97	3128	3166	3162	3162
query98	239	227	225	225
query99	1506	1308	1316	1308
Total cold run time: 272684 ms
Total hot run time: 196225 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8fc334f6ef2f300f81f80afdc887f4293c4c977f, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.50	0.50
query6	1.15	0.72	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.58	0.52	0.51
query10	0.60	0.59	0.58
query11	0.16	0.12	0.12
query12	0.15	0.13	0.13
query13	0.63	0.61	0.62
query14	1.47	1.47	1.48
query15	0.90	0.87	0.88
query16	0.37	0.37	0.36
query17	1.07	1.06	1.05
query18	0.16	0.16	0.17
query19	1.97	1.81	1.76
query20	0.01	0.01	0.01
query21	15.39	0.68	0.69
query22	3.73	6.97	1.82
query23	17.98	1.42	1.29
query24	2.26	0.22	0.22
query25	0.18	0.08	0.08
query26	0.30	0.19	0.19
query27	0.08	0.08	0.07
query28	13.17	1.14	1.11
query29	12.54	3.31	3.28
query30	0.24	0.05	0.06
query31	2.85	0.42	0.42
query32	3.21	0.49	0.50
query33	3.03	3.05	3.07
query34	15.45	4.35	4.31
query35	4.34	4.34	4.37
query36	0.69	0.49	0.49
query37	0.18	0.17	0.16
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.10	0.04	0.04
query42	0.06	0.06	0.04
query43	0.04	0.04	0.05
Total cold run time: 108.02 s
Total hot run time: 31.59 s

@zhangstar333

zhangstar333 commented Sep 13, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 43077 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb24dee01977105442ebdbdd6b3e7de04db8a120, data reload: false
------ Round 1 ----------------------------------
q1	18060	7482	7316	7316
q2	2380	206	191	191
q3	10861	1357	1395	1357
q4	10153	1007	1068	1007
q5	7720	3201	3159	3159
q6	246	157	157	157
q7	1061	642	620	620
q8	9454	2045	2071	2045
q9	6775	6284	6276	6276
q10	7028	2527	2535	2527
q11	425	250	264	250
q12	407	238	233	233
q13	17761	3029	3057	3029
q14	287	248	258	248
q15	573	548	532	532
q16	522	436	442	436
q17	983	970	929	929
q18	7248	6831	6700	6700
q19	1388	1247	1251	1247
q20	616	342	332	332
q21	3952	3582	3507	3507
q22	1083	1001	979	979
Total cold run time: 108983 ms
Total hot run time: 43077 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7165	7153	7204	7153
q2	362	232	234	232
q3	2951	2919	2892	2892
q4	1958	1943	1937	1937
q5	5442	5433	5425	5425
q6	237	148	149	148
q7	2037	1685	1664	1664
q8	3229	3302	3328	3302
q9	8378	8434	8449	8434
q10	3403	3448	3448	3448
q11	575	463	475	463
q12	782	610	589	589
q13	7083	3064	3056	3056
q14	304	271	284	271
q15	566	512	520	512
q16	504	457	433	433
q17	1755	1739	1711	1711
q18	8049	7600	7546	7546
q19	1757	1699	1705	1699
q20	2052	1821	1818	1818
q21	5686	5390	5392	5390
q22	1118	998	974	974
Total cold run time: 65393 ms
Total hot run time: 59097 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.89% (9465/25654)
Line Coverage: 28.26% (77844/275430)
Region Coverage: 27.67% (40197/145295)
Branch Coverage: 24.28% (20430/84140)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb24dee01977105442ebdbdd6b3e7de04db8a120_bb24dee01977105442ebdbdd6b3e7de04db8a120/report/index.html

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 194699 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb24dee01977105442ebdbdd6b3e7de04db8a120, data reload: false
query1	906	376	366	366
query2	6459	1732	1728	1728
query3	6691	213	222	213
query4	25926	23833	23975	23833
query5	5051	550	545	545
query6	268	173	173	173
query7	4601	301	312	301
query8	289	229	244	229
query9	8506	2607	2594	2594
query10	472	284	278	278
query11	16165	15524	15664	15524
query12	154	105	101	101
query13	1691	401	378	378
query14	11737	6367	7125	6367
query15	232	170	176	170
query16	7506	472	482	472
query17	1545	584	574	574
query18	1929	300	291	291
query19	202	149	151	149
query20	121	110	119	110
query21	214	106	102	102
query22	4694	4487	4400	4400
query23	34440	33783	33742	33742
query24	9779	3103	3064	3064
query25	666	428	417	417
query26	1203	159	161	159
query27	2801	289	290	289
query28	6795	2137	2093	2093
query29	955	433	433	433
query30	301	164	152	152
query31	983	760	803	760
query32	110	55	60	55
query33	741	313	306	306
query34	907	485	488	485
query35	908	741	747	741
query36	1025	903	899	899
query37	173	90	84	84
query38	4020	3865	3853	3853
query39	1450	1411	1425	1411
query40	213	120	119	119
query41	50	49	51	49
query42	122	99	97	97
query43	500	435	435	435
query44	1239	792	779	779
query45	198	182	174	174
query46	1122	839	817	817
query47	1905	1815	1809	1809
query48	370	299	293	293
query49	1134	461	464	461
query50	922	433	433	433
query51	7164	6975	6845	6845
query52	106	94	92	92
query53	258	189	187	187
query54	799	471	476	471
query55	84	78	75	75
query56	298	285	272	272
query57	1231	1072	1088	1072
query58	376	244	237	237
query59	2695	2544	2487	2487
query60	326	278	276	276
query61	105	105	100	100
query62	938	662	660	660
query63	224	187	192	187
query64	5353	688	669	669
query65	3309	3185	3139	3139
query66	1391	304	301	301
query67	15916	15484	15517	15484
query68	3258	602	586	586
query69	423	284	278	278
query70	1184	1079	1097	1079
query71	341	285	276	276
query72	6108	4044	4050	4044
query73	762	329	334	329
query74	9464	9042	9055	9042
query75	3380	2759	2790	2759
query76	1939	1345	1344	1344
query77	435	322	319	319
query78	9877	9486	9279	9279
query79	983	897	892	892
query80	983	830	814	814
query81	512	264	275	264
query82	1363	268	264	264
query83	221	195	202	195
query84	245	117	109	109
query85	734	403	393	393
query86	334	320	321	320
query87	4401	4460	4306	4306
query88	4161	4104	4075	4075
query89	394	375	381	375
query90	1809	328	324	324
query91	125	126	125	125
query92	87	78	75	75
query93	1093	1090	1075	1075
query94	885	364	379	364
query95	517	424	443	424
query96	476	478	473	473
query97	3215	3148	3115	3115
query98	244	233	226	226
query99	1549	1312	1295	1295
Total cold run time: 282466 ms
Total hot run time: 194699 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb24dee01977105442ebdbdd6b3e7de04db8a120, data reload: false
query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.06	0.06
query5	0.50	0.49	0.50
query6	1.13	0.73	0.74
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.56	0.52	0.50
query10	0.57	0.59	0.56
query11	0.15	0.12	0.11
query12	0.16	0.13	0.12
query13	0.62	0.60	0.60
query14	1.47	1.47	1.45
query15	0.90	0.88	0.89
query16	0.36	0.36	0.37
query17	1.05	1.02	1.04
query18	0.22	0.21	0.21
query19	1.97	1.82	1.78
query20	0.01	0.01	0.01
query21	15.43	0.68	0.68
query22	4.10	8.35	1.14
query23	17.87	1.33	1.32
query24	2.28	0.22	0.22
query25	0.19	0.08	0.08
query26	0.29	0.19	0.17
query27	0.08	0.08	0.07
query28	13.17	1.14	1.11
query29	12.52	3.34	3.34
query30	0.25	0.05	0.06
query31	2.87	0.42	0.41
query32	3.23	0.49	0.49
query33	3.03	3.05	3.12
query34	15.44	4.31	4.31
query35	4.34	4.34	4.36
query36	0.68	0.48	0.50
query37	0.19	0.16	0.17
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.14	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.29 s
Total hot run time: 30.92 s

HappenLee
HappenLee reviewed Sep 14, 2024
View reviewed changes
Comment threadbe/src/pipeline/exec/partition_sort_sink_operator.h
HappenLee
HappenLee reviewed Sep 14, 2024
View reviewed changes
Comment threadbe/src/pipeline/exec/partition_sort_sink_operator.h
HappenLee
HappenLee approved these changes Sep 14, 2024
View reviewed changes

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2024
@github-actions

github-actionsBot commented Sep 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Sep 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguolei
yiguolei merged commit 19e65c3 into apache:masterSep 14, 2024
yiguolei pushed a commit that referenced this pull request Sep 14, 2024
@zhangstar333
[cherry-pick](branch-21) fix partition_topn not reset output rows aft…
9a79edc
…er do_partition_topn_sort (#40761) (#40792)
## Proposed changes
cherry-pick from master #40761
<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@zhangstar333@dataroaring
[Bug](partition_topn) fix partition_topn not reset output rows after …
9f22401
…do_partition_topn_sort (#40761)
## Proposed changes
should reset all variables about output rows after
do_partition_topn_sort and get_next from sorter
if we not reset all variables. it's will be regard as have output many
rows
but the sorter still have remaing many data need to output.
<!--Describe your changes.-->
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
@yiguoleiyiguolei mentioned this pull request Nov 6, 2024
Closed
hubgeter pushed a commit to hubgeter/doris that referenced this pull request Jan 15, 2025
@kaka11chen
[cherry-pick](branch-21) fix partition_topn not reset output rows aft…
54e4011
…er do_partition_topn_sort (apache#40761) (apache#40792) (apache#268)
@yiguoleiyiguolei mentioned this pull request Jan 19, 2025
Closed
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@zhangstar333@HappenLee
[Bug](partition_topn) fix partition_topn not reset output rows after …
0239f8a
…do_partition_topn_sort (apache#40761)
## Proposed changes
should reset all variables about output rows after
do_partition_topn_sort and get_next from sorter
if we not reset all variables. it's will be regard as have output many
rows
but the sorter still have remaing many data need to output.
<!--Describe your changes.-->
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDtBiteTheDDDDtBiteTheDDDDt approved these changes
@HappenLeeHappenLeeHappenLee approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.2-mergedp0_breviewedusercaseImportant user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zhangstar333@doris-robot@BiteTheDDDDt@HappenLee@yiguolei@dataroaring