Skip to content

[Fix](Nereids) fix use cbo rule hint unused because of logic reverse - #39715

Merged
morrySnow merged 1 commit into
apache:masterfrom
LiBinfeng-01:fix_cbo_rule_hint
Aug 23, 2024
Merged

[Fix](Nereids) fix use cbo rule hint unused because of logic reverse#39715
morrySnow merged 1 commit into
apache:masterfrom
LiBinfeng-01:fix_cbo_rule_hint

Conversation

@LiBinfeng-01

@LiBinfeng-01LiBinfeng-01 commented Aug 21, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

use_cbo_rule_hint show used but it does not be used
intro by: #35925

@doris-robot

doris-robot commented Aug 21, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@LiBinfeng-01

LiBinfeng-01 commented Aug 21, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morrySnow

morrySnow commented Aug 21, 2024

Copy link
Copy Markdown
Contributor

add intro pr in desc

@github-actions

github-actionsBot commented Aug 21, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 21, 2024
@github-actions

github-actionsBot commented Aug 21, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Aug 21, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38011 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
------ Round 1 ----------------------------------
q1	17885	4446	4301	4301
q2	2048	211	210	210
q3	11789	1048	1049	1048
q4	10554	852	793	793
q5	7753	2857	2799	2799
q6	267	160	158	158
q7	1010	652	643	643
q8	9587	2113	2095	2095
q9	8761	6580	6572	6572
q10	7063	2273	2179	2179
q11	503	275	264	264
q12	420	248	252	248
q13	18946	2977	3007	2977
q14	293	245	253	245
q15	548	506	528	506
q16	526	410	414	410
q17	988	694	713	694
q18	7259	6697	6780	6697
q19	6600	1091	1060	1060
q20	683	349	348	348
q21	3784	2786	2710	2710
q22	1154	1061	1054	1054
Total cold run time: 118421 ms
Total hot run time: 38011 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4575	4324	4290	4290
q2	391	292	298	292
q3	2887	2697	2728	2697
q4	1987	1740	1700	1700
q5	5680	5719	5654	5654
q6	254	154	158	154
q7	2240	1788	1769	1769
q8	3302	3509	3464	3464
q9	8759	8667	8768	8667
q10	3659	3386	3300	3300
q11	630	513	516	513
q12	841	696	673	673
q13	16248	3204	3066	3066
q14	320	298	286	286
q15	570	516	520	516
q16	507	457	484	457
q17	1821	1559	1561	1559
q18	8310	7970	7780	7780
q19	3402	1588	1493	1493
q20	2181	1937	1857	1857
q21	14276	5255	5403	5255
q22	1224	1101	1075	1075
Total cold run time: 84064 ms
Total hot run time: 56517 ms

@doris-robot

doris-robot commented Aug 21, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196363 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
query1	1323	917	887	887
query2	6679	2012	1907	1907
query3	10824	3976	3800	3800
query4	55530	25649	23347	23347
query5	5872	721	717	717
query6	522	214	211	211
query7	6089	342	336	336
query8	541	447	442	442
query9	8926	2578	2543	2543
query10	576	348	342	342
query11	17110	15021	15274	15021
query12	193	140	136	136
query13	1627	426	432	426
query14	11122	7508	7304	7304
query15	253	198	194	194
query16	7441	501	560	501
query17	1165	633	620	620
query18	1784	363	340	340
query19	303	170	169	169
query20	155	136	136	136
query21	251	146	142	142
query22	4513	4238	4184	4184
query23	34432	33670	33962	33670
query24	6157	3007	3030	3007
query25	572	438	438	438
query26	698	186	186	186
query27	1707	302	319	302
query28	4020	2222	2214	2214
query29	691	468	455	455
query30	232	193	194	193
query31	1056	865	832	832
query32	111	97	83	83
query33	505	344	348	344
query34	898	508	503	503
query35	904	753	761	753
query36	1110	959	968	959
query37	161	109	105	105
query38	3969	3778	3844	3778
query39	1542	1487	1468	1468
query40	236	151	154	151
query41	139	139	140	139
query42	147	122	117	117
query43	545	510	521	510
query44	1120	788	798	788
query45	223	193	197	193
query46	1148	833	781	781
query47	1914	1822	1824	1822
query48	395	342	339	339
query49	917	584	582	582
query50	877	465	469	465
query51	7096	7077	7195	7077
query52	123	106	113	106
query53	298	228	228	228
query54	602	511	494	494
query55	96	92	89	89
query56	353	309	309	309
query57	1231	1142	1117	1117
query58	297	299	297	297
query59	3003	2996	2726	2726
query60	349	331	318	318
query61	148	145	150	145
query62	808	721	699	699
query63	269	227	229	227
query64	3192	1874	1870	1870
query65	3288	3192	3202	3192
query66	1032	682	715	682
query67	15512	14993	14984	14984
query68	5122	570	575	570
query69	611	317	323	317
query70	1267	1125	1155	1125
query71	516	320	320	320
query72	7449	2374	2121	2121
query73	830	359	357	357
query74	9235	8801	8803	8801
query75	3631	2697	2711	2697
query76	2596	1063	991	991
query77	824	451	489	451
query78	9853	9127	9184	9127
query79	7635	564	569	564
query80	1363	617	607	607
query81	617	264	266	264
query82	1304	162	156	156
query83	405	217	214	214
query84	297	99	99	99
query85	1183	359	360	359
query86	430	308	302	302
query87	4377	4213	4225	4213
query88	5424	2453	2471	2453
query89	458	340	332	332
query90	2145	236	235	235
query91	152	127	123	123
query92	84	115	76	76
query93	6135	545	559	545
query94	969	331	335	331
query95	396	307	298	298
query96	642	285	278	278
query97	3247	3071	3117	3071
query98	248	238	229	229
query99	1740	1321	1295	1295
Total cold run time: 328107 ms
Total hot run time: 196363 ms

@doris-robot

doris-robot commented Aug 21, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.08	0.08
query5	0.51	0.52	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.49	0.49
query10	0.55	0.55	0.54
query11	0.16	0.12	0.13
query12	0.16	0.13	0.13
query13	0.63	0.60	0.58
query14	0.78	0.79	0.79
query15	0.83	0.82	0.83
query16	0.36	0.37	0.39
query17	1.05	0.97	1.02
query18	0.22	0.20	0.20
query19	1.85	1.71	1.69
query20	0.01	0.01	0.02
query21	15.40	0.66	0.65
query22	4.26	6.51	2.27
query23	18.33	1.39	1.32
query24	2.05	0.25	0.22
query25	0.16	0.09	0.09
query26	0.26	0.19	0.18
query27	0.08	0.08	0.08
query28	13.20	1.03	1.02
query29	12.65	3.34	3.29
query30	0.44	0.24	0.19
query31	2.81	0.39	0.40
query32	3.25	0.49	0.48
query33	2.98	3.00	2.95
query34	17.19	4.43	4.44
query35	4.47	4.47	4.45
query36	0.66	0.48	0.50
query37	0.20	0.18	0.17
query38	0.17	0.16	0.17
query39	0.06	0.05	0.06
query40	0.18	0.14	0.15
query41	0.11	0.07	0.07
query42	0.08	0.07	0.07
query43	0.06	0.06	0.06
Total cold run time: 109.94 s
Total hot run time: 31.43 s

@LiBinfeng-01

LiBinfeng-01 commented Aug 22, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 37837 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
------ Round 1 ----------------------------------
q1	17616	4455	4296	4296
q2	2009	180	171	171
q3	11697	929	1093	929
q4	10520	745	654	654
q5	7758	2834	2798	2798
q6	222	137	134	134
q7	987	613	598	598
q8	9501	2063	2091	2063
q9	6946	6531	6518	6518
q10	6996	2198	2232	2198
q11	470	243	239	239
q12	392	226	230	226
q13	18819	3023	2987	2987
q14	279	246	239	239
q15	525	478	486	478
q16	484	388	381	381
q17	984	662	676	662
q18	7269	6902	6890	6890
q19	1387	1030	1042	1030
q20	684	331	330	330
q21	4150	3008	3097	3008
q22	1130	1019	1008	1008
Total cold run time: 110825 ms
Total hot run time: 37837 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4383	4388	4309	4309
q2	368	271	272	271
q3	2913	2622	2660	2622
q4	1900	1672	1633	1633
q5	5587	5678	5703	5678
q6	221	129	132	129
q7	2198	1841	1841	1841
q8	3247	3428	3455	3428
q9	8798	8814	8801	8801
q10	3630	3397	3370	3370
q11	612	519	488	488
q12	826	701	660	660
q13	13985	3197	3255	3197
q14	312	287	287	287
q15	543	492	487	487
q16	491	457	448	448
q17	1827	1557	1523	1523
q18	8083	7822	7837	7822
q19	1751	1633	1500	1500
q20	2133	1949	1870	1870
q21	5815	5467	5574	5467
q22	1138	1052	1032	1032
Total cold run time: 70761 ms
Total hot run time: 56863 ms

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 190890 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
query1	1270	871	868	868
query2	6292	1871	1904	1871
query3	10589	3677	3908	3677
query4	59944	23864	23071	23071
query5	5587	518	512	512
query6	427	167	168	167
query7	6046	298	293	293
query8	292	227	225	225
query9	8757	2452	2441	2441
query10	483	281	271	271
query11	18430	15011	15171	15011
query12	164	106	114	106
query13	1570	382	375	375
query14	11141	7471	7371	7371
query15	230	166	175	166
query16	7531	474	457	457
query17	1151	571	576	571
query18	1846	299	309	299
query19	287	148	161	148
query20	123	120	111	111
query21	210	102	102	102
query22	4742	4467	4525	4467
query23	34153	34106	33136	33136
query24	5930	2822	2876	2822
query25	527	378	400	378
query26	696	155	153	153
query27	1805	281	280	280
query28	3723	2018	2000	2000
query29	667	409	425	409
query30	236	152	147	147
query31	930	770	731	731
query32	75	55	57	55
query33	452	288	281	281
query34	870	483	477	477
query35	844	754	727	727
query36	1036	917	899	899
query37	146	83	83	83
query38	3956	3826	3851	3826
query39	1457	1387	1392	1387
query40	191	115	118	115
query41	47	45	46	45
query42	118	98	102	98
query43	507	472	469	469
query44	1087	746	758	746
query45	198	166	162	162
query46	1100	759	775	759
query47	1838	1771	1792	1771
query48	378	289	302	289
query49	777	420	423	420
query50	836	410	418	410
query51	7231	6980	7219	6980
query52	99	87	89	87
query53	258	181	179	179
query54	572	446	444	444
query55	76	77	77	77
query56	291	264	256	256
query57	1166	1086	1078	1078
query58	212	232	229	229
query59	3045	2754	2917	2754
query60	293	278	310	278
query61	111	98	99	98
query62	769	643	674	643
query63	214	186	182	182
query64	3290	1744	1778	1744
query65	3242	3167	3189	3167
query66	668	332	326	326
query67	15441	15159	15068	15068
query68	3006	608	572	572
query69	394	313	307	307
query70	1153	1106	1017	1017
query71	363	276	276	276
query72	2496	2118	2107	2107
query73	718	326	331	326
query74	9183	8830	8824	8824
query75	3384	2703	2618	2618
query76	1529	1117	976	976
query77	530	324	322	322
query78	9884	9022	8965	8965
query79	1045	540	538	538
query80	725	508	530	508
query81	459	235	221	221
query82	291	150	135	135
query83	173	146	148	146
query84	257	74	79	74
query85	684	289	285	285
query86	306	305	290	290
query87	4478	4283	4305	4283
query88	3137	2345	2352	2345
query89	377	286	292	286
query90	1863	197	196	196
query91	122	98	99	98
query92	59	51	54	51
query93	1057	533	544	533
query94	620	294	313	294
query95	330	269	268	268
query96	593	268	275	268
query97	3174	3156	3082	3082
query98	228	217	202	202
query99	1540	1232	1286	1232
Total cold run time: 302828 ms
Total hot run time: 190890 ms

@doris-robot

doris-robot commented Aug 22, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 42a8389249bedc1ef2b3604e0a880b6c6878fd54, data reload: false
query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.49	0.48
query6	1.13	0.71	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.50	0.49
query10	0.54	0.52	0.53
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.59
query14	0.75	0.79	0.80
query15	0.85	0.82	0.82
query16	0.36	0.37	0.38
query17	1.05	1.04	1.06
query18	0.21	0.21	0.21
query19	1.96	1.68	1.79
query20	0.01	0.01	0.00
query21	15.39	0.68	0.67
query22	4.00	6.74	2.66
query23	18.30	1.34	1.25
query24	2.17	0.22	0.22
query25	0.16	0.09	0.07
query26	0.26	0.18	0.18
query27	0.08	0.08	0.07
query28	13.23	1.00	1.00
query29	12.65	3.42	3.45
query30	0.24	0.06	0.05
query31	2.89	0.40	0.39
query32	3.25	0.48	0.47
query33	2.95	3.00	2.97
query34	16.78	4.40	4.38
query35	4.46	4.44	4.43
query36	0.66	0.48	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.05	0.03
Total cold run time: 109.17 s
Total hot run time: 31.46 s

@morrySnow
morrySnow merged commit 9712d65 into apache:masterAug 23, 2024
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
@LiBinfeng-01@dataroaring
[Fix](Nereids) fix use cbo rule hint unused because of logic reverse (#…
03e7e52
…39715)
use_cbo_rule_hint show used but it does not be used
intro by: #35925
LiBinfeng-01 added a commit to LiBinfeng-01/doris that referenced this pull request Oct 15, 2024
@LiBinfeng-01
[Fix](Nereids) fix use cbo rule hint unused because of logic reverse (a…
78eacbb
…pache#39715)
use_cbo_rule_hint show used but it does not be used
intro by: apache#35925
@LiBinfeng-01LiBinfeng-01 mentioned this pull request Oct 15, 2024
Merged
morrySnow pushed a commit that referenced this pull request Oct 17, 2024
@LiBinfeng-01
[Feat](Nereids) support use cbo rule hint#35925#39715#40167#40958(…
80d7523
…#41869)
pick: #35925#39715#40167#40958
Add feat of force use/nouse cbo rule hint and fix pr
introduce
when not using this hint, cbo rules like INFER_SET_OPERATOR_DISTINCT
would generate two plans and compare their cost
and nereids optimizer would decide which is better. But when we want to
control the behavior of cbo rules we could use this force cbo rule hint
usage example
explain shape plan
select /*+ USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */
*
from t1
union
select * from t2;
the USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) hint would force rule
INFER_SET_OPERATOR_DISTINCT to be used
and generate plan like, which hashAgg below union is generated by this
rule:
-- !with_hint_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t1]
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t2]
Hint log:
Used: INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
When we want to force disable this rule, we could use
explain shape plan select /*+
NO_USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */ * from t1 union select *
from t2;
which would generate plan with this rule:
-- !with_hint_no_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------PhysicalOlapScan[t1]
--------------PhysicalOlapScan[t2]
Hint log:
Used: NO_INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
change sessionvariable enableNereidsRules to varType.remove
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@LiBinfeng-01@HappenLee
[Fix](Nereids) fix use cbo rule hint unused because of logic reverse (a…
6cc6162
…pache#39715)
use_cbo_rule_hint show used but it does not be used
intro by: apache#35925
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@engleflyengleflyenglefly approved these changes
@morrySnowmorrySnowmorrySnow approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.2-mergedp0_breviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@LiBinfeng-01@doris-robot@morrySnow@englefly@dataroaring