Skip to content

[Bug](partition-topn) fix partition-topn calculate partition input rows have error - #39100

Merged
HappenLee merged 1 commit into
apache:masterfrom
zhangstar333:fix_partition_nums
Aug 10, 2024
Merged

[Bug](partition-topn) fix partition-topn calculate partition input rows have error#39100
HappenLee merged 1 commit into
apache:masterfrom
zhangstar333:fix_partition_nums

Conversation

@zhangstar333

@zhangstar333zhangstar333 commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

  1. fix the _sorted_partition_input_rows calculate have error, it's should only update the rows which have been emplace into hash table, not include the rows which is pass through.

  2. add some counter in profile could get some info of about input/output rows have been do partition-topn.

@doris-robot

doris-robot commented Aug 8, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333

zhangstar333 commented Aug 8, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Aug 8, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Aug 8, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
------ Round 1 ----------------------------------
q1	17629	4299	4249	4249
q2	2021	176	186	176
q3	10495	1131	1073	1073
q4	10145	696	743	696
q5	7498	2524	2471	2471
q6	220	139	141	139
q7	959	594	600	594
q8	9219	1884	1890	1884
q9	8642	6553	6500	6500
q10	7046	2180	2199	2180
q11	460	235	246	235
q12	503	216	221	216
q13	17756	3001	2976	2976
q14	288	245	237	237
q15	523	489	502	489
q16	511	382	379	379
q17	959	664	748	664
q18	8052	7440	7459	7440
q19	5749	1004	1060	1004
q20	659	325	328	325
q21	5225	4199	4652	4199
q22	1088	1003	1021	1003
Total cold run time: 115647 ms
Total hot run time: 39129 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4489	4264	4208	4208
q2	372	261	275	261
q3	2852	2638	2601	2601
q4	1977	1662	1732	1662
q5	5513	5531	5468	5468
q6	233	137	134	134
q7	2078	1716	1748	1716
q8	3288	3427	3397	3397
q9	8779	8695	8836	8695
q10	3518	3268	3268	3268
q11	622	507	502	502
q12	767	623	621	621
q13	17109	3206	3199	3199
q14	319	280	294	280
q15	530	495	496	495
q16	499	453	445	445
q17	1806	1526	1483	1483
q18	7890	7898	7809	7809
q19	1765	1593	1609	1593
q20	2139	1875	1929	1875
q21	5477	5157	5421	5157
q22	1092	1038	1021	1021
Total cold run time: 73114 ms
Total hot run time: 55890 ms

@doris-robot

doris-robot commented Aug 8, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 201782 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
query1	949	415	392	392
query2	6437	2072	1931	1931
query3	6631	205	216	205
query4	31393	23105	22965	22965
query5	3609	497	475	475
query6	274	184	182	182
query7	4581	301	289	289
query8	250	206	202	202
query9	8471	2375	2368	2368
query10	958	882	861	861
query11	17757	14992	14882	14882
query12	128	94	92	92
query13	1631	379	368	368
query14	10360	7185	7645	7185
query15	381	363	324	324
query16	7676	464	501	464
query17	1450	589	543	543
query18	2033	361	398	361
query19	266	200	184	184
query20	126	122	118	118
query21	206	103	107	103
query22	4408	4356	4342	4342
query23	34333	33668	33890	33668
query24	10992	3063	2825	2825
query25	570	363	393	363
query26	699	148	146	146
query27	2152	276	286	276
query28	6023	1992	2002	1992
query29	768	400	395	395
query30	250	148	143	143
query31	963	760	738	738
query32	97	52	53	52
query33	613	271	266	266
query34	876	456	474	456
query35	979	822	847	822
query36	1102	920	935	920
query37	132	76	81	76
query38	4449	4205	4193	4193
query39	1403	1395	1386	1386
query40	189	115	113	113
query41	46	48	45	45
query42	113	94	95	94
query43	502	484	470	470
query44	1090	725	734	725
query45	399	364	381	364
query46	1120	799	761	761
query47	1816	1798	1762	1762
query48	374	299	314	299
query49	848	442	445	442
query50	793	413	422	413
query51	6888	6685	6617	6617
query52	102	91	89	89
query53	248	182	178	178
query54	906	441	440	440
query55	74	75	72	72
query56	257	247	237	237
query57	1113	1064	1078	1064
query58	220	223	231	223
query59	3017	2856	2855	2855
query60	291	255	256	255
query61	96	111	98	98
query62	804	648	642	642
query63	203	178	174	174
query64	9241	2517	1896	1896
query65	3210	3141	3120	3120
query66	748	330	325	325
query67	15307	14765	14661	14661
query68	5042	552	535	535
query69	410	424	413	413
query70	1172	1112	1107	1107
query71	490	272	264	264
query72	18410	16594	16122	16122
query73	782	325	326	325
query74	9107	8760	8748	8748
query75	3416	2572	2613	2572
query76	3128	982	1004	982
query77	593	304	317	304
query78	9524	9069	8934	8934
query79	1778	515	509	509
query80	1922	480	477	477
query81	628	229	226	226
query82	684	128	127	127
query83	288	148	144	144
query84	260	78	74	74
query85	1338	320	304	304
query86	459	277	287	277
query87	4638	4420	4536	4420
query88	4171	2472	2465	2465
query89	390	282	282	282
query90	1700	196	189	189
query91	139	119	118	118
query92	60	47	53	47
query93	2506	535	534	534
query94	785	285	304	285
query95	355	259	259	259
query96	609	279	272	272
query97	3252	3174	3022	3022
query98	228	201	196	196
query99	1491	1266	1262	1262
Total cold run time: 306146 ms
Total hot run time: 201782 ms

@doris-robot

doris-robot commented Aug 8, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.10	0.09
query5	0.50	0.49	0.49
query6	1.12	0.73	0.76
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.56	0.49	0.48
query10	0.55	0.54	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.59
query14	0.76	0.79	0.78
query15	0.85	0.80	0.80
query16	0.38	0.35	0.34
query17	1.03	0.96	1.02
query18	0.22	0.22	0.22
query19	1.86	1.68	1.66
query20	0.03	0.01	0.01
query21	15.39	0.74	0.65
query22	4.63	7.45	1.45
query23	18.28	1.37	1.24
query24	2.09	0.23	0.21
query25	0.14	0.09	0.07
query26	0.29	0.21	0.21
query27	0.45	0.23	0.22
query28	13.26	1.02	1.01
query29	12.65	3.38	3.39
query30	0.25	0.06	0.05
query31	2.85	0.38	0.38
query32	3.28	0.47	0.46
query33	2.89	2.85	2.91
query34	17.06	4.30	4.35
query35	4.46	4.42	4.38
query36	0.65	0.49	0.48
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 30.02 s

@zhangstar333

zhangstar333 commented Aug 9, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 9, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 39041 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
------ Round 1 ----------------------------------
q1	17628	4366	4388	4366
q2	2030	175	183	175
q3	10467	1192	1081	1081
q4	10126	669	733	669
q5	7492	2534	2485	2485
q6	226	143	143	143
q7	965	586	590	586
q8	9216	1889	1906	1889
q9	8651	6506	6527	6506
q10	7060	2137	2150	2137
q11	460	244	248	244
q12	513	218	218	218
q13	17749	2966	2933	2933
q14	286	239	230	230
q15	523	488	489	488
q16	496	384	380	380
q17	955	643	666	643
q18	7881	7564	7438	7438
q19	2217	988	1107	988
q20	655	328	323	323
q21	5532	4528	4099	4099
q22	1096	1034	1020	1020
Total cold run time: 112224 ms
Total hot run time: 39041 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4450	4234	4285	4234
q2	375	265	270	265
q3	2816	2614	2559	2559
q4	1873	1632	1716	1632
q5	5560	5582	5474	5474
q6	223	143	142	142
q7	2145	1734	1684	1684
q8	3214	3574	3392	3392
q9	8776	8705	8665	8665
q10	3562	3311	3290	3290
q11	605	500	524	500
q12	809	676	663	663
q13	17360	3162	3154	3154
q14	317	278	282	278
q15	539	487	500	487
q16	499	439	459	439
q17	1814	1578	1483	1483
q18	7972	7905	7858	7858
q19	1728	1554	1616	1554
q20	2163	1895	1847	1847
q21	7911	5351	5312	5312
q22	1109	1022	1025	1022
Total cold run time: 75820 ms
Total hot run time: 55934 ms

@doris-robot

doris-robot commented Aug 9, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 204083 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
query1	974	407	397	397
query2	6462	1847	1872	1847
query3	6637	206	221	206
query4	33741	23287	23136	23136
query5	3655	498	504	498
query6	271	179	167	167
query7	4582	296	289	289
query8	234	202	215	202
query9	8593	2402	2356	2356
query10	930	932	885	885
query11	17688	14890	14913	14890
query12	134	100	93	93
query13	1670	396	373	373
query14	10291	8008	8106	8008
query15	426	343	338	338
query16	7823	464	480	464
query17	1723	608	601	601
query18	2145	435	391	391
query19	254	253	224	224
query20	123	111	111	111
query21	213	118	115	115
query22	4391	4386	4279	4279
query23	34214	33464	33862	33464
query24	9877	2971	2982	2971
query25	549	382	391	382
query26	694	154	154	154
query27	2134	281	285	281
query28	5651	2020	1996	1996
query29	746	408	413	408
query30	253	152	152	152
query31	961	762	724	724
query32	87	54	52	52
query33	618	282	277	277
query34	860	473	478	473
query35	948	833	840	833
query36	1060	926	953	926
query37	133	78	76	76
query38	4282	4097	4139	4097
query39	1445	1397	1370	1370
query40	197	117	121	117
query41	46	44	45	44
query42	118	97	97	97
query43	496	471	460	460
query44	1091	738	745	738
query45	409	373	388	373
query46	1110	809	778	778
query47	1811	1773	1755	1755
query48	385	301	307	301
query49	837	422	436	422
query50	804	403	417	403
query51	6866	6671	6665	6665
query52	108	88	89	88
query53	253	183	181	181
query54	824	451	460	451
query55	76	77	75	75
query56	266	280	240	240
query57	1122	1041	1095	1041
query58	227	230	238	230
query59	2849	2826	2726	2726
query60	294	265	268	265
query61	93	90	93	90
query62	769	636	641	636
query63	210	187	183	183
query64	9266	2477	1925	1925
query65	3207	3152	3141	3141
query66	759	328	332	328
query67	15232	14709	14614	14614
query68	4637	561	557	557
query69	406	391	410	391
query70	1151	1066	1119	1066
query71	364	282	285	282
query72	20889	17389	17382	17382
query73	790	328	335	328
query74	9207	8823	8718	8718
query75	3752	2606	2644	2606
query76	2670	1041	973	973
query77	656	317	306	306
query78	9619	9976	8880	8880
query79	2406	545	517	517
query80	1403	513	512	512
query81	600	228	237	228
query82	894	148	140	140
query83	349	163	147	147
query84	279	87	81	81
query85	1836	312	358	312
query86	463	281	300	281
query87	4722	4516	4494	4494
query88	3851	2492	2523	2492
query89	402	289	293	289
query90	1823	207	205	205
query91	143	133	116	116
query92	61	51	52	51
query93	2671	534	533	533
query94	897	289	299	289
query95	371	267	259	259
query96	640	277	276	276
query97	3249	3055	3080	3055
query98	228	206	205	205
query99	1523	1239	1251	1239
Total cold run time: 310149 ms
Total hot run time: 204083 ms

@doris-robot

doris-robot commented Aug 9, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be736576ec7f5f71f8128d974653bee8d09caf16, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.66	0.09	0.08
query5	0.51	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.48	0.49
query10	0.54	0.53	0.54
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.61	0.58
query14	0.76	0.80	0.78
query15	0.84	0.84	0.81
query16	0.36	0.36	0.37
query17	1.00	1.02	0.95
query18	0.23	0.22	0.21
query19	1.89	1.75	1.69
query20	0.01	0.01	0.01
query21	15.39	0.75	0.65
query22	3.79	6.79	2.22
query23	18.28	1.39	1.30
query24	2.09	0.23	0.21
query25	0.15	0.08	0.08
query26	0.30	0.21	0.21
query27	0.45	0.22	0.22
query28	13.36	1.02	1.01
query29	12.65	3.39	3.34
query30	0.24	0.06	0.05
query31	2.90	0.39	0.39
query32	3.29	0.48	0.47
query33	2.93	2.96	2.85
query34	17.12	4.33	4.38
query35	4.40	4.41	4.39
query36	0.66	0.46	0.50
query37	0.19	0.16	0.15
query38	0.15	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.51 s
Total hot run time: 30.87 s

HappenLee
HappenLee approved these changes Aug 9, 2024
View reviewed changes

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Aug 9, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 9, 2024
@github-actions

github-actionsBot commented Aug 9, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@HappenLee
HappenLee merged commit 0e9951f into apache:masterAug 10, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@zhangstar333@dataroaring
[Bug](partition-topn) fix partition-topn calculate partition input ro…
a2c16e2
…ws have error (#39100)
1. fix the _sorted_partition_input_rows calculate have error, it's
should only update the rows which have been emplace into hash table, not
include the rows which is pass through.
2. add some counter in profile could get some info of about input/output
rows have been do partition-topn.
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Aug 13, 2024
@zhangstar333
[cherry-pick](branch-21) fix partition-topn calculate partition input…
49f8eec
… rows have error (apache#39100)
@zhangstar333zhangstar333 mentioned this pull request Aug 13, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 13, 2024
@zhangstar333
[cherry-pick](branch-21) fix partition-topn calculate partition input…
7e7729c
… rows have error (#39100) (#39281)
## Proposed changes
cherry-pick from master: #39100 <!--Describe your changes.-->
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
@zhangstar333@wyxxxcat
[Bug](partition-topn) fix partition-topn calculate partition input ro…
49105a9
…ws have error (apache#39100)
1. fix the _sorted_partition_input_rows calculate have error, it's
should only update the rows which have been emplace into hash table, not
include the rows which is pass through.
2. add some counter in profile could get some info of about input/output
rows have been do partition-topn.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
@zhangstar333@dataroaring
[Bug](partition-topn) fix partition-topn calculate partition input ro…
3924de8
…ws have error (#39100)
1. fix the _sorted_partition_input_rows calculate have error, it's
should only update the rows which have been emplace into hash table, not
include the rows which is pass through.
2. add some counter in profile could get some info of about input/output
rows have been do partition-topn.
@zhangstar333zhangstar333 mentioned this pull request Aug 27, 2024
Merged
HappenLee pushed a commit that referenced this pull request Aug 31, 2024
@zhangstar333
[cherry-pick](branch-20) fix partition-topn calculate partition input…
5e82c2b
… rows have error (#39100) (#39581) (#40003)
cherry-pick from master #39100#39581
@gavinchougavinchou mentioned this pull request Oct 13, 2024
Closed
zhangstar333 added a commit to zhangstar333/incubator-doris that referenced this pull request Oct 17, 2024
@zhangstar333
[Bug](partition-topn) fix partition-topn calculate partition input ro…
8718bfe
…ws have error (apache#39100)
1. fix the _sorted_partition_input_rows calculate have error, it's
should only update the rows which have been emplace into hash table, not
include the rows which is pass through.
2. add some counter in profile could get some info of about input/output
rows have been do partition-topn.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@zhangstar333@HappenLee
[Bug](partition-topn) fix partition-topn calculate partition input ro…
dd460f8
…ws have error (apache#39100)
1. fix the _sorted_partition_input_rows calculate have error, it's
should only update the rows which have been emplace into hash table, not
include the rows which is pass through.
2. add some counter in profile could get some info of about input/output
rows have been do partition-topn.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@BiteTheDDDDtBiteTheDDDDtBiteTheDDDDt approved these changes
@HappenLeeHappenLeeHappenLee approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.15-mergeddev/2.1.6-mergeddev/3.0.2-mergeddoingreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@zhangstar333@doris-robot@BiteTheDDDDt@HappenLee@yiguolei@wm1581066@dataroaring