Skip to content

[opt](nereids) improve prune partition with lots of in (xxx) - #46261

Merged
924060929 merged 1 commit into
apache:masterfrom
924060929:opt-prune-partition-in-predicte
Jan 2, 2025
Merged

[opt](nereids) improve prune partition with lots of in (xxx)#46261
924060929 merged 1 commit into
apache:masterfrom
924060929:opt-prune-partition-in-predicte

Conversation

@924060929

@924060929924060929 commented Jan 2, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

improve prune partition with lots of in (xxx)

this can reduce from 35s to 300ms when there have 4000 options

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929

924060929 commented Jan 2, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jan 2, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32403 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99572e9807571b4fc3f197af6b0374ad45359042, data reload: false
------ Round 1 ----------------------------------
q1	17604	6102	6013	6013
q2	2046	300	167	167
q3	10436	1189	724	724
q4	10290	865	441	441
q5	8826	2157	1978	1978
q6	204	180	145	145
q7	902	773	596	596
q8	9249	1352	1177	1177
q9	5285	5011	4931	4931
q10	6755	2318	1862	1862
q11	481	280	252	252
q12	337	359	213	213
q13	17807	3612	2929	2929
q14	223	225	214	214
q15	554	508	501	501
q16	615	614	583	583
q17	584	849	341	341
q18	6732	6445	6246	6246
q19	2266	950	582	582
q20	302	305	188	188
q21	3037	2152	2014	2014
q22	367	332	306	306
Total cold run time: 104902 ms
Total hot run time: 32403 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6321	6220	6217	6217
q2	239	333	225	225
q3	2269	2653	2312	2312
q4	1381	1805	1373	1373
q5	4330	4742	4855	4742
q6	187	183	147	147
q7	2047	2013	1832	1832
q8	2626	2740	2689	2689
q9	7268	7300	7329	7300
q10	3057	3418	2795	2795
q11	568	498	492	492
q12	674	746	671	671
q13	3370	3760	3180	3180
q14	289	316	296	296
q15	564	509	515	509
q16	640	702	664	664
q17	1243	1719	1261	1261
q18	7634	7446	7355	7355
q19	874	1157	1126	1126
q20	1995	1977	1866	1866
q21	5780	5388	4901	4901
q22	613	632	588	588
Total cold run time: 53969 ms
Total hot run time: 52541 ms

@doris-robot

doris-robot commented Jan 2, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99572e9807571b4fc3f197af6b0374ad45359042, data reload: false
query1	1285	962	896	896
query2	6400	2462	2397	2397
query3	11101	4829	4737	4737
query4	33142	23724	23456	23456
query5	4737	609	471	471
query6	289	205	190	190
query7	3987	488	309	309
query8	281	251	231	231
query9	9328	2631	2623	2623
query10	467	317	247	247
query11	18293	15400	15248	15248
query12	157	106	105	105
query13	1552	534	401	401
query14	10311	7617	7034	7034
query15	245	214	178	178
query16	7887	601	454	454
query17	1527	745	587	587
query18	2017	423	307	307
query19	213	191	165	165
query20	128	117	114	114
query21	203	142	125	125
query22	4600	4542	4316	4316
query23	34498	33750	33390	33390
query24	6510	2285	2314	2285
query25	488	450	390	390
query26	800	284	150	150
query27	2153	486	346	346
query28	5938	2455	2432	2432
query29	641	562	437	437
query30	219	185	150	150
query31	963	920	844	844
query32	102	63	63	63
query33	489	376	333	333
query34	779	868	546	546
query35	811	833	779	779
query36	1038	1081	1012	1012
query37	126	116	88	88
query38	4275	4279	4244	4244
query39	1517	1497	1453	1453
query40	229	120	108	108
query41	48	50	45	45
query42	128	106	106	106
query43	549	579	514	514
query44	1357	828	835	828
query45	182	175	166	166
query46	882	1063	667	667
query47	2040	2001	1942	1942
query48	396	414	324	324
query49	722	506	395	395
query50	697	694	399	399
query51	7306	7384	7299	7299
query52	110	105	93	93
query53	230	263	183	183
query54	498	484	423	423
query55	85	79	83	79
query56	275	268	260	260
query57	1253	1212	1177	1177
query58	230	225	226	225
query59	3227	3351	3110	3110
query60	278	272	253	253
query61	110	106	112	106
query62	862	827	754	754
query63	242	206	191	191
query64	3655	1013	645	645
query65	3304	3274	3280	3274
query66	775	408	309	309
query67	16498	15814	15648	15648
query68	9704	769	509	509
query69	486	306	257	257
query70	1215	1132	1128	1128
query71	445	294	254	254
query72	6212	3922	3875	3875
query73	676	775	369	369
query74	10128	9178	9217	9178
query75	4708	3175	2637	2637
query76	5213	1172	769	769
query77	1017	371	279	279
query78	11125	10158	9372	9372
query79	5980	889	587	587
query80	717	515	508	508
query81	504	264	222	222
query82	328	153	122	122
query83	187	164	143	143
query84	282	89	69	69
query85	749	368	301	301
query86	361	324	307	307
query87	4659	4389	4430	4389
query88	3940	2253	2221	2221
query89	430	326	292	292
query90	2139	235	191	191
query91	134	136	104	104
query92	70	54	48	48
query93	3302	876	535	535
query94	676	381	273	273
query95	331	262	250	250
query96	491	621	274	274
query97	2752	2800	2687	2687
query98	220	209	195	195
query99	1645	1520	1439	1439
Total cold run time: 306491 ms
Total hot run time: 196840 ms

@doris-robot

doris-robot commented Jan 2, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.06 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99572e9807571b4fc3f197af6b0374ad45359042, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.41	0.42
query6	1.17	0.64	0.64
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.51	0.51
query10	0.56	0.57	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.58
query14	2.72	2.84	2.73
query15	0.89	0.82	0.82
query16	0.39	0.39	0.39
query17	1.06	1.06	1.04
query18	0.22	0.20	0.21
query19	1.98	1.89	1.98
query20	0.02	0.01	0.02
query21	15.37	0.93	0.59
query22	0.77	0.85	0.68
query23	15.16	1.41	0.60
query24	3.82	1.03	0.58
query25	0.15	0.23	0.05
query26	0.29	0.14	0.14
query27	0.05	0.06	0.05
query28	13.80	1.52	1.05
query29	12.60	3.97	3.23
query30	0.25	0.10	0.06
query31	2.83	0.59	0.37
query32	3.22	0.54	0.48
query33	3.06	3.16	3.10
query34	16.94	5.16	4.52
query35	4.53	4.51	4.50
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.92 s
Total hot run time: 31.06 s

@github-actions

github-actionsBot commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025
@github-actions

github-actionsBot commented Jan 2, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@924060929
924060929 merged commit 5fc52ba into apache:masterJan 2, 2025
github-actionsBot pushed a commit that referenced this pull request Jan 2, 2025
@924060929Your Name
[opt](nereids) improve prune partition with lots ofin (xxx)(#46261)
eaa8e5f
improve prune partition with lots of `in (xxx)`
this can reduce from 35s to 300ms when there have 4000 options
@github-actionsgithub-actionsBot mentioned this pull request Jan 2, 2025
Merged
github-actionsBot pushed a commit that referenced this pull request Jan 2, 2025
@924060929Your Name
[opt](nereids) improve prune partition with lots ofin (xxx)(#46261)
d7cf0de
improve prune partition with lots of `in (xxx)`
this can reduce from 35s to 300ms when there have 4000 options
@github-actionsgithub-actionsBot mentioned this pull request Jan 2, 2025
Merged
924060929 pushed a commit that referenced this pull request Jan 2, 2025
@github-actions
branch-2.1: [opt](nereids) improve prune partition with lots of `in (…
c385293
…xxx)` #46261 (#46286)
improve prune partition with lots of `in (xxx)`
this can reduce from 35s to 300ms when there have 4000 options
morrySnow pushed a commit that referenced this pull request Jan 3, 2025
@github-actions@924060929
branch-3.0: [opt](nereids) improve prune partition with lots of `in (…
04b18d9
…xxx)` #46261 (#46285)
Cherry-picked from #46261
Co-authored-by: 924060929 <lanhuajian@selectdb.com>
@gavinchougavinchou mentioned this pull request Feb 18, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnowmorrySnowmorrySnow approved these changes
+1 more reviewer
@feiniaofeiafeifeiniaofeiafeifeiniaofeiafei approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.8-mergeddev/3.0.4-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@924060929@hello-stephen@doris-robot@feiniaofeiafei@morrySnow