[enhancement](Nereids) Enable parse sql from sql cache by 924060929 · Pull Request #33262 · apache/doris

924060929 · 2024-04-04T17:27:35Z

Proposed changes

Before this pr, the query must pass through parser, analyzer, rewriter, optimizer and translator, then we can check whether this query can use sql cache, if the query is too long, or the number of join tables too big, the plan time usually >= 500ms.

This pr reduce this time by skip the fashion plan path, because we can reuse the previous physical plan and query result if no any changed. In some cases we should not parse sql from sql cache, e.g. table structure changed, data changed, user policies changed, privileges changed, contains non-deterministic functions, and user variables changed.

In my test case: query a view which has lots of join and union, and the tables has empty partition, the query latency is about 3ms. if not parse sql from sql cache, the plan time is about 550ms

Features

use Config.sql_cache_manage_num to control how many sql cache be reused in on fe
if explain plan appear some plans contains LogicalSqlCache or PhysicalSqlCache, it means the query can use sql cache, like this:

mysql>set enable_sql_cache=true;
Query OK, 0 rows affected (0.00 sec)
mysql> explain physical plan select*fromtest.t;
+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+----------------------------------------------------------------------------------+
| cost =3.135 |
| PhysicalResultSink[53] ( outputExprs=[c1#0, c2#1] ) |
| +--PhysicalDistribute[50]@0 ( stats=3, distributionSpec=DistributionSpecGather ) |
| +--PhysicalOlapScan[t]@0 ( stats=3 ) |+----------------------------------------------------------------------------------+4 rows inset (0.02 sec)
mysql>select*fromtest.t;
+------+------+
| c1 | c2 |
+------+------+
| 1 | 2 |
| -2 | -2 |
| NULL | 30 |
+------+------+3 rows inset (0.05 sec)
mysql> explain physical plan select*fromtest.t;
+-------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+-------------------------------------------------------------------------------------------+
| cost =0.0 |
| PhysicalSqlCache[2] ( queryId=78511f515cda466b-95385d892d6c68d0, backend=127.0.0.1:9050 ) |
| +--PhysicalResultSink[52] ( outputExprs=[c1#0, c2#1] ) |
| +--PhysicalDistribute[49]@0 ( stats=3, distributionSpec=DistributionSpecGather ) |
| +--PhysicalOlapScan[t]@0 ( stats=3 ) |+-------------------------------------------------------------------------------------------+5 rows inset (0.01 sec)

doris-robot · 2024-04-04T17:27:39Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

924060929 · 2024-04-04T18:38:08Z

run buildall

doris-robot · 2024-04-04T19:18:54Z

TPC-H: Total hot run time: 38368 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f231da42e48dff435f0d6827f480151c3997935a, data reload: false
------ Round 1 ----------------------------------
q1	17638	4053	4067	4053
q2	2015	183	181	181
q3	10474	1198	1312	1198
q4	10189	854	940	854
q5	7464	2989	2926	2926
q6	221	136	134	134
q7	1130	607	591	591
q8	9408	1977	2025	1977
q9	6762	6200	6138	6138
q10	8444	3489	3495	3489
q11	421	241	235	235
q12	388	217	212	212
q13	17783	2895	2930	2895
q14	276	239	245	239
q15	524	485	483	483
q16	501	386	388	386
q17	959	904	894	894
q18	7226	6414	6329	6329
q19	2694	1534	1519	1519
q20	570	315	299	299
q21	3501	3116	3053	3053
q22	362	283	302	283
Total cold run time: 108950 ms
Total hot run time: 38368 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4059	4028	4064	4028
q2	323	216	228	216
q3	2967	2949	2918	2918
q4	1833	1871	1868	1868
q5	5228	5182	5210	5182
q6	206	125	124	124
q7	2244	1790	1797	1790
q8	3185	3272	3245	3245
q9	8461	8461	8452	8452
q10	3741	3955	4008	3955
q11	548	465	467	465
q12	726	555	596	555
q13	16950	3077	3091	3077
q14	301	273	284	273
q15	525	508	491	491
q16	501	450	458	450
q17	1777	1724	1745	1724
q18	8186	7865	7598	7598
q19	1702	1679	1711	1679
q20	2028	1893	1868	1868
q21	5120	4949	4792	4792
q22	502	428	443	428
Total cold run time: 71113 ms
Total hot run time: 55178 ms

doris-robot · 2024-04-04T19:30:21Z

TPC-DS: Total hot run time: 182044 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f231da42e48dff435f0d6827f480151c3997935a, data reload: false
query1	1207	1124	1109	1109
query2	6318	1925	1949	1925
query3	6656	227	218	218
query4	24325	21615	21574	21574
query5	4166	417	406	406
query6	272	199	172	172
query7	4605	318	303	303
query8	229	176	183	176
query9	8493	2194	2207	2194
query10	456	258	269	258
query11	15180	14564	14596	14564
query12	131	90	87	87
query13	1635	383	377	377
query14	8562	6985	6976	6976
query15	214	172	191	172
query16	6753	279	266	266
query17	982	588	579	579
query18	1835	308	280	280
query19	191	148	151	148
query20	90	86	83	83
query21	204	134	129	129
query22	4941	4766	4774	4766
query23	33290	32904	32640	32640
query24	11496	3240	3166	3166
query25	686	385	418	385
query26	1916	164	160	160
query27	3185	364	377	364
query28	7403	1849	1832	1832
query29	1338	602	595	595
query30	312	173	182	173
query31	974	760	729	729
query32	103	57	55	55
query33	696	251	250	250
query34	1315	490	508	490
query35	886	735	717	717
query36	995	910	883	883
query37	268	74	71	71
query38	3619	3587	3640	3587
query39	1633	1635	1583	1583
query40	235	131	129	129
query41	45	43	42	42
query42	111	100	101	100
query43	497	432	427	427
query44	1207	740	757	740
query45	282	290	257	257
query46	1119	829	805	805
query47	1970	1884	1845	1845
query48	382	308	300	300
query49	962	356	376	356
query50	820	423	405	405
query51	6862	6714	6675	6675
query52	106	90	102	90
query53	367	286	280	280
query54	282	241	220	220
query55	81	77	75	75
query56	244	225	252	225
query57	1276	1171	1195	1171
query58	235	207	228	207
query59	2723	2720	2540	2540
query60	244	225	229	225
query61	87	86	87	86
query62	670	434	448	434
query63	307	285	281	281
query64	5725	3136	3133	3133
query65	3033	2992	3004	2992
query66	1311	318	314	314
query67	15176	15012	14878	14878
query68	5881	572	573	572
query69	545	303	303	303
query70	1191	1087	1103	1087
query71	492	278	276	276
query72	6360	2563	2390	2390
query73	775	331	324	324
query74	6748	6262	6282	6262
query75	3365	2297	2281	2281
query76	4156	1206	1221	1206
query77	522	249	296	249
query78	10905	10047	10014	10014
query79	3294	530	535	530
query80	1330	432	413	413
query81	494	234	226	226
query82	988	96	91	91
query83	254	173	169	169
query84	270	84	88	84
query85	1471	282	276	276
query86	469	304	283	283
query87	3640	3484	3526	3484
query88	3807	2353	2370	2353
query89	480	375	370	370
query90	1980	180	180	180
query91	128	105	102	102
query92	60	47	51	47
query93	5423	528	526	526
query94	1019	188	178	178
query95	418	344	314	314
query96	605	277	276	276
query97	2660	2473	2458	2458
query98	226	212	218	212
query99	1225	834	874	834
Total cold run time: 285100 ms
Total hot run time: 182044 ms

doris-robot · 2024-04-04T19:35:47Z

ClickBench: Total hot run time: 30.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f231da42e48dff435f0d6827f480151c3997935a, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.47	0.49
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.57	0.51	0.52
query10	0.57	0.57	0.56
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.78	0.80
query15	0.87	0.84	0.83
query16	0.35	0.35	0.36
query17	0.96	0.99	0.96
query18	0.26	0.27	0.25
query19	1.77	1.69	1.71
query20	0.02	0.01	0.01
query21	15.42	0.65	0.64
query22	4.11	6.19	2.53
query23	17.78	1.27	1.18
query24	1.41	0.20	0.19
query25	0.16	0.09	0.08
query26	0.28	0.16	0.17
query27	0.08	0.07	0.08
query28	13.92	0.97	0.95
query29	12.54	3.28	3.24
query30	0.26	0.07	0.06
query31	2.87	0.40	0.40
query32	3.27	0.48	0.47
query33	2.85	2.88	2.90
query34	15.50	4.34	4.33
query35	4.39	4.36	4.39
query36	0.68	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.18	0.16	0.14
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.03 s
Total hot run time: 30.75 s

doris-robot · 2024-04-04T19:38:52Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f231da42e48dff435f0d6827f480151c3997935a with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 17.0 seconds inserted 10000000 Rows, about 588K ops/s

924060929 · 2024-04-05T10:35:25Z

run buildall

doris-robot · 2024-04-05T11:10:38Z

TPC-H: Total hot run time: 38365 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 75cbd4e2325715e10753cbedb1686bd1a2211902, data reload: false
------ Round 1 ----------------------------------
q1	17660	4049	4079	4049
q2	2019	185	177	177
q3	10506	1217	1343	1217
q4	10208	815	967	815
q5	7472	2967	2915	2915
q6	219	136	133	133
q7	1095	604	612	604
q8	9398	2084	2036	2036
q9	6837	6209	6134	6134
q10	8453	3492	3482	3482
q11	420	231	245	231
q12	383	209	206	206
q13	17779	2876	2911	2876
q14	269	230	239	230
q15	532	479	469	469
q16	512	388	368	368
q17	961	910	906	906
q18	7204	6376	6411	6376
q19	1598	1542	1529	1529
q20	583	310	305	305
q21	3464	3015	3130	3015
q22	350	292	303	292
Total cold run time: 107922 ms
Total hot run time: 38365 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4090	4062	4054	4054
q2	319	219	223	219
q3	2941	2922	2927	2922
q4	1838	1844	1821	1821
q5	5197	5183	5168	5168
q6	211	123	125	123
q7	2216	1774	1817	1774
q8	3197	3275	3255	3255
q9	8459	8556	8484	8484
q10	3737	3892	3991	3892
q11	563	458	482	458
q12	752	582	576	576
q13	16936	3053	3101	3053
q14	320	289	284	284
q15	521	492	517	492
q16	490	449	451	449
q17	1789	1739	1717	1717
q18	8169	7859	7695	7695
q19	1699	1679	1698	1679
q20	2046	1786	1826	1786
q21	5109	4918	4922	4918
q22	525	435	461	435
Total cold run time: 71124 ms
Total hot run time: 55254 ms

doris-robot · 2024-04-05T11:22:10Z

TPC-DS: Total hot run time: 181646 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 75cbd4e2325715e10753cbedb1686bd1a2211902, data reload: false
query1	1213	1113	1113	1113
query2	6362	2027	1920	1920
query3	6672	205	201	201
query4	25052	21420	21291	21291
query5	4207	419	421	419
query6	282	189	193	189
query7	4607	299	296	296
query8	222	169	174	169
query9	8472	2305	2298	2298
query10	457	240	249	240
query11	15027	14508	14512	14508
query12	134	87	89	87
query13	1638	366	373	366
query14	8696	7043	6928	6928
query15	216	170	179	170
query16	6772	263	267	263
query17	1000	577	561	561
query18	1832	287	276	276
query19	209	159	165	159
query20	94	88	87	87
query21	201	141	132	132
query22	5005	4807	4823	4807
query23	33232	33205	32571	32571
query24	10547	3150	3180	3150
query25	722	424	432	424
query26	1962	184	164	164
query27	3112	385	406	385
query28	7233	1918	1887	1887
query29	1281	620	629	620
query30	363	182	179	179
query31	1031	718	726	718
query32	94	56	52	52
query33	655	241	238	238
query34	1202	511	523	511
query35	866	710	720	710
query36	1011	859	858	858
query37	243	75	76	75
query38	3669	3649	3591	3591
query39	1648	1628	1548	1548
query40	225	132	129	129
query41	46	50	43	43
query42	106	105	105	105
query43	467	413	418	413
query44	1229	722	742	722
query45	287	290	276	276
query46	1093	832	814	814
query47	1942	1846	1838	1838
query48	394	311	304	304
query49	998	371	368	368
query50	829	419	424	419
query51	6973	6824	6844	6824
query52	95	100	94	94
query53	363	292	287	287
query54	261	233	224	224
query55	80	78	77	77
query56	236	223	240	223
query57	1253	1172	1169	1169
query58	238	219	209	209
query59	2654	2467	2276	2276
query60	247	230	232	230
query61	90	89	88	88
query62	668	438	453	438
query63	308	279	284	279
query64	5807	3097	3147	3097
query65	3043	2995	2988	2988
query66	1351	327	312	312
query67	15212	14761	14766	14761
query68	6319	561	569	561
query69	546	313	314	313
query70	1173	1078	1080	1078
query71	524	267	271	267
query72	6270	2624	2411	2411
query73	812	320	322	320
query74	6723	6325	6333	6325
query75	3528	2317	2265	2265
query76	4724	1146	1226	1146
query77	562	245	251	245
query78	10737	10082	10123	10082
query79	7766	530	526	526
query80	1073	432	434	432
query81	499	234	234	234
query82	763	95	97	95
query83	239	170	166	166
query84	268	88	86	86
query85	1325	284	282	282
query86	448	278	289	278
query87	3712	3542	3471	3471
query88	4082	2369	2370	2369
query89	551	372	384	372
query90	1953	180	181	180
query91	136	104	104	104
query92	60	49	53	49
query93	6234	519	527	519
query94	955	185	183	183
query95	417	327	329	327
query96	602	283	275	275
query97	2667	2473	2479	2473
query98	234	204	215	204
query99	1331	840	860	840
Total cold run time: 290969 ms
Total hot run time: 181646 ms

doris-robot · 2024-04-05T11:27:36Z

ClickBench: Total hot run time: 29.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 75cbd4e2325715e10753cbedb1686bd1a2211902, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.49	0.48
query6	1.13	0.64	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.52	0.51
query10	0.55	0.57	0.57
query11	0.16	0.12	0.11
query12	0.14	0.11	0.12
query13	0.62	0.58	0.59
query14	0.77	0.77	0.80
query15	0.85	0.84	0.82
query16	0.34	0.36	0.35
query17	0.96	0.98	0.97
query18	0.24	0.25	0.25
query19	1.82	1.66	1.75
query20	0.02	0.01	0.01
query21	15.43	0.63	0.63
query22	3.80	8.11	1.36
query23	17.99	1.30	1.31
query24	1.53	0.20	0.19
query25	0.16	0.07	0.08
query26	0.25	0.16	0.16
query27	0.08	0.08	0.07
query28	13.88	0.96	0.94
query29	12.53	3.25	3.23
query30	0.26	0.06	0.06
query31	2.86	0.38	0.37
query32	3.32	0.48	0.47
query33	2.88	2.93	2.90
query34	15.48	4.34	4.35
query35	4.36	4.35	4.38
query36	0.68	0.46	0.46
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.03	0.03	0.03
query40	0.17	0.16	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 106.93 s
Total hot run time: 29.56 s

doris-robot · 2024-04-05T11:30:40Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 75cbd4e2325715e10753cbedb1686bd1a2211902 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 15.7 seconds inserted 10000000 Rows, about 636K ops/s

924060929 · 2024-04-07T03:38:32Z

run buildall

924060929 · 2024-04-07T03:51:23Z

run buildall

doris-robot · 2024-04-07T04:25:53Z

TPC-H: Total hot run time: 38298 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a764541e252934c60ef945b78480776f7dcd7214, data reload: false
------ Round 1 ----------------------------------
q1	17679	4056	4052	4052
q2	2012	180	179	179
q3	10497	1143	1314	1143
q4	10211	833	908	833
q5	7472	2946	2914	2914
q6	217	131	130	130
q7	1078	604	619	604
q8	9406	1977	2012	1977
q9	6793	6167	6165	6165
q10	8489	3516	3466	3466
q11	421	247	233	233
q12	418	215	203	203
q13	17769	2876	2887	2876
q14	269	235	228	228
q15	523	475	469	469
q16	500	378	367	367
q17	950	924	909	909
q18	7323	6438	6343	6343
q19	1721	1525	1527	1525
q20	554	329	312	312
q21	3484	3125	3070	3070
q22	356	302	300	300
Total cold run time: 108142 ms
Total hot run time: 38298 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4067	4012	4026	4012
q2	328	219	221	219
q3	2942	2941	2944	2941
q4	1842	1823	1830	1823
q5	5214	5214	5197	5197
q6	210	122	120	120
q7	2242	1838	1758	1758
q8	3200	3259	3255	3255
q9	8452	8466	8461	8461
q10	3729	3917	3971	3917
q11	576	451	466	451
q12	724	578	589	578
q13	16882	3084	3100	3084
q14	314	277	263	263
q15	524	474	490	474
q16	482	460	464	460
q17	1782	1741	1726	1726
q18	8221	7833	7518	7518
q19	2015	1676	1688	1676
q20	2025	1824	1820	1820
q21	5054	4964	4875	4875
q22	502	408	446	408
Total cold run time: 71327 ms
Total hot run time: 55036 ms

doris-robot · 2024-04-07T04:37:16Z

TPC-DS: Total hot run time: 181634 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a764541e252934c60ef945b78480776f7dcd7214, data reload: false
query1	1211	1120	1110	1110
query2	6289	1937	1948	1937
query3	6665	222	215	215
query4	25020	21405	21583	21405
query5	4185	392	420	392
query6	271	182	169	169
query7	4605	300	290	290
query8	226	167	170	167
query9	8483	2208	2212	2208
query10	446	251	245	245
query11	14946	14368	14587	14368
query12	134	102	88	88
query13	1643	391	375	375
query14	8508	6930	6799	6799
query15	203	174	170	170
query16	6727	259	259	259
query17	1017	564	557	557
query18	1824	277	274	274
query19	191	147	147	147
query20	90	89	85	85
query21	191	130	125	125
query22	5022	4800	4793	4793
query23	34173	32611	32946	32611
query24	11815	3150	3118	3118
query25	719	402	412	402
query26	1897	163	163	163
query27	3190	368	371	368
query28	7430	1888	1835	1835
query29	1240	631	604	604
query30	298	179	175	175
query31	1004	740	780	740
query32	96	57	60	57
query33	671	239	248	239
query34	1266	504	492	492
query35	841	717	721	717
query36	995	893	890	890
query37	262	72	74	72
query38	3614	3599	3577	3577
query39	1674	1630	1614	1614
query40	229	131	126	126
query41	46	46	44	44
query42	114	112	105	105
query43	472	417	416	416
query44	1170	739	733	733
query45	270	255	273	255
query46	1135	839	810	810
query47	1964	1888	1888	1888
query48	392	306	305	305
query49	929	383	371	371
query50	838	411	409	409
query51	6951	6801	6757	6757
query52	103	91	89	89
query53	356	294	297	294
query54	267	235	236	235
query55	88	76	76	76
query56	251	231	231	231
query57	1270	1160	1171	1160
query58	250	212	214	212
query59	2654	2509	2351	2351
query60	259	245	242	242
query61	118	107	111	107
query62	690	468	450	450
query63	310	280	284	280
query64	5841	3204	3130	3130
query65	3019	2977	2974	2974
query66	1323	309	317	309
query67	15208	14855	14823	14823
query68	4538	557	574	557
query69	483	296	293	293
query70	1156	1139	1112	1112
query71	483	269	270	269
query72	6115	2557	2417	2417
query73	719	321	322	321
query74	6611	6293	6330	6293
query75	3075	2336	2285	2285
query76	3115	1078	1237	1078
query77	526	249	254	249
query78	10715	10217	10166	10166
query79	3241	542	520	520
query80	2024	430	442	430
query81	518	243	233	233
query82	1206	96	100	96
query83	329	180	178	178
query84	281	90	91	90
query85	1578	334	327	327
query86	459	298	296	296
query87	3674	3437	3543	3437
query88	3696	2373	2369	2369
query89	478	371	370	370
query90	1988	186	187	186
query91	139	113	115	113
query92	65	51	52	51
query93	3952	516	529	516
query94	1094	183	185	183
query95	532	319	312	312
query96	605	272	272	272
query97	2657	2546	2516	2516
query98	227	214	213	213
query99	1285	857	852	852
Total cold run time: 283163 ms
Total hot run time: 181634 ms

doris-robot · 2024-04-07T04:42:43Z

ClickBench: Total hot run time: 29.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a764541e252934c60ef945b78480776f7dcd7214, data reload: false
query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.06
query5	0.48	0.48	0.49
query6	1.13	0.65	0.66
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.52
query10	0.56	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.60	0.59
query14	0.77	0.78	0.78
query15	0.86	0.84	0.84
query16	0.35	0.35	0.34
query17	0.96	0.99	0.99
query18	0.25	0.25	0.25
query19	1.79	1.72	1.71
query20	0.01	0.01	0.01
query21	15.42	0.65	0.66
query22	4.06	7.50	1.34
query23	18.08	1.34	1.31
query24	1.49	0.20	0.19
query25	0.15	0.08	0.08
query26	0.28	0.17	0.18
query27	0.08	0.07	0.07
query28	13.88	0.98	0.95
query29	12.53	3.30	3.27
query30	0.25	0.06	0.06
query31	2.85	0.40	0.40
query32	3.27	0.48	0.47
query33	2.86	2.82	2.89
query34	15.48	4.35	4.34
query35	4.38	4.37	4.41
query36	0.68	0.48	0.48
query37	0.18	0.17	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.03
query40	0.20	0.15	0.15
query41	0.09	0.05	0.04
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.22 s
Total hot run time: 29.72 s

doris-robot · 2024-04-07T04:45:50Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a764541e252934c60ef945b78480776f7dcd7214 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 16.7 seconds inserted 10000000 Rows, about 598K ops/s

924060929 · 2024-04-07T04:54:19Z

run buildall

924060929 · 2024-04-07T05:06:04Z

run buildall

doris-robot · 2024-04-07T05:40:31Z

TPC-H: Total hot run time: 38087 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0e77c449931e17bb986973f95f160f2d17668096, data reload: false
------ Round 1 ----------------------------------
q1	17648	4066	4037	4037
q2	2016	180	171	171
q3	10493	1144	1279	1144
q4	10207	809	943	809
q5	7474	2914	2927	2914
q6	217	134	132	132
q7	1086	607	610	607
q8	9394	1894	2012	1894
q9	6987	6199	6118	6118
q10	8453	3530	3492	3492
q11	420	238	233	233
q12	377	217	211	211
q13	17778	2879	2902	2879
q14	269	241	249	241
q15	527	483	470	470
q16	500	402	376	376
q17	945	899	895	895
q18	7220	6437	6309	6309
q19	1831	1517	1510	1510
q20	576	293	296	293
q21	3523	3118	3063	3063
q22	350	289	309	289
Total cold run time: 108291 ms
Total hot run time: 38087 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4090	3998	4030	3998
q2	319	219	217	217
q3	2925	2932	2910	2910
q4	1851	1863	1826	1826
q5	5177	5192	5179	5179
q6	206	124	125	124
q7	2230	1770	1765	1765
q8	3187	3246	3227	3227
q9	8422	8439	8436	8436
q10	3731	3857	4110	3857
q11	582	448	463	448
q12	759	589	628	589
q13	17075	3079	3212	3079
q14	320	269	278	269
q15	540	481	487	481
q16	475	430	432	430
q17	1737	1719	1716	1716
q18	8333	7799	7614	7614
q19	1709	1709	1718	1709
q20	2088	1809	1783	1783
q21	5350	5044	5050	5044
q22	503	423	420	420
Total cold run time: 71609 ms
Total hot run time: 55121 ms

doris-robot · 2024-04-07T05:51:47Z

TPC-DS: Total hot run time: 180834 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0e77c449931e17bb986973f95f160f2d17668096, data reload: false
query1	1207	1118	1116	1116
query2	6182	1923	1962	1923
query3	6663	223	227	223
query4	23470	21726	21457	21457
query5	4183	392	416	392
query6	271	180	191	180
query7	4585	300	296	296
query8	227	178	177	177
query9	8478	2221	2191	2191
query10	467	256	264	256
query11	14907	14660	14434	14434
query12	141	89	86	86
query13	1628	379	388	379
query14	8456	6905	6857	6857
query15	208	176	173	173
query16	6786	270	267	267
query17	1003	584	616	584
query18	1868	303	306	303
query19	195	150	150	150
query20	90	86	87	86
query21	194	122	126	122
query22	4927	4798	4764	4764
query23	33332	32498	32565	32498
query24	9935	3198	3188	3188
query25	661	391	380	380
query26	1488	172	163	163
query27	3239	371	378	371
query28	7192	1879	1847	1847
query29	1103	605	592	592
query30	312	178	171	171
query31	1032	800	728	728
query32	95	55	52	52
query33	639	269	242	242
query34	1116	520	521	520
query35	840	720	721	720
query36	1005	882	874	874
query37	151	71	75	71
query38	3613	3535	3637	3535
query39	1114	1015	1101	1015
query40	207	128	129	128
query41	48	44	43	43
query42	120	101	108	101
query43	471	438	436	436
query44	1190	755	740	740
query45	275	258	272	258
query46	1127	863	834	834
query47	1954	1865	1825	1825
query48	392	319	301	301
query49	865	362	369	362
query50	812	410	425	410
query51	6821	6825	6776	6776
query52	119	96	93	93
query53	356	292	287	287
query54	274	247	226	226
query55	86	78	80	78
query56	236	234	238	234
query57	1262	1171	1184	1171
query58	247	213	211	211
query59	2603	2584	2337	2337
query60	260	232	230	230
query61	90	89	87	87
query62	657	449	453	449
query63	318	282	288	282
query64	4953	3175	3113	3113
query65	3028	3002	3001	3001
query66	964	337	317	317
query67	15179	14867	14767	14767
query68	4651	581	562	562
query69	521	291	300	291
query70	1108	1123	1149	1123
query71	411	279	282	279
query72	6534	2538	2383	2383
query73	726	321	321	321
query74	6637	6243	6218	6218
query75	2950	2302	2239	2239
query76	3200	1192	1226	1192
query77	521	252	252	252
query78	10804	10120	10092	10092
query79	3598	542	535	535
query80	1752	431	426	426
query81	510	233	233	233
query82	1209	96	97	96
query83	306	175	173	173
query84	263	90	87	87
query85	1480	294	271	271
query86	476	316	278	278
query87	3650	3458	3490	3458
query88	3847	2348	2364	2348
query89	503	370	374	370
query90	1911	187	190	187
query91	128	100	103	100
query92	56	46	46	46
query93	5439	529	521	521
query94	1048	179	183	179
query95	413	315	311	311
query96	621	273	269	269
query97	2677	2462	2477	2462
query98	237	218	215	215
query99	1361	847	837	837
Total cold run time: 277465 ms
Total hot run time: 180834 ms

doris-robot · 2024-04-07T05:57:13Z

ClickBench: Total hot run time: 30.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0e77c449931e17bb986973f95f160f2d17668096, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.69	0.06	0.06
query5	0.49	0.49	0.50
query6	1.15	0.65	0.66
query7	0.01	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.51	0.51
query10	0.56	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.60
query14	0.78	0.78	0.79
query15	0.86	0.86	0.86
query16	0.35	0.36	0.35
query17	0.96	1.02	1.01
query18	0.25	0.25	0.24
query19	1.80	1.71	1.74
query20	0.02	0.02	0.01
query21	15.40	0.65	0.65
query22	3.58	6.19	1.83
query23	17.98	1.26	1.25
query24	1.47	0.19	0.20
query25	0.15	0.08	0.08
query26	0.28	0.17	0.17
query27	0.08	0.08	0.08
query28	13.83	0.97	0.95
query29	12.58	3.30	3.30
query30	0.26	0.06	0.06
query31	2.85	0.42	0.39
query32	3.27	0.49	0.48
query33	2.89	2.85	2.90
query34	15.49	4.31	4.33
query35	4.40	4.37	4.36
query36	0.70	0.47	0.47
query37	0.18	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.15
query41	0.09	0.04	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 106.73 s
Total hot run time: 30.22 s

doris-robot · 2024-04-07T06:00:19Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 0e77c449931e17bb986973f95f160f2d17668096 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 15.9 seconds inserted 10000000 Rows, about 628K ops/s

924060929 · 2024-04-07T08:48:58Z

run buildall

github-actions · 2024-04-07T08:55:55Z

clang-tidy review says "All clean, LGTM! 👍"

924060929 · 2024-04-07T08:57:06Z

run buildall

…cache=true (#42032) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by #33262

…che=true (#42032) (#42037) fix Is Cached is not Yes in Profile when enable_sql_cache=true, introduced by #33262

…che=true (#42032) (#42034) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by #33262

…yy-MM-dd HH:mm:ss') (#44631) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') which introduced by #33262 the wrong result: is `yyyy-MM-dd HH:mm:ss`

…yy-MM-dd HH:mm:ss') (apache#44631) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') which introduced by apache#33262 the wrong result: is `yyyy-MM-dd HH:mm:ss` (cherry picked from commit d592e27)

…yy-MM-dd HH:mm:ss') (#44631) (#44810) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') which introduced by #33262 the wrong result: is `yyyy-MM-dd HH:mm:ss`

…yy-MM-dd HH:mm:ss') (#44631) (#44811) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') which introduced by #33262 the wrong result: is `yyyy-MM-dd HH:mm:ss`

1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join` 2. fix failed test `explain`, this bug only exists in master branch, introduced by #40202 3. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by #33262 4. regression test add `foreachFrontends`, `foreachBackends`, `retry` function

1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join` 2. fix failed test `explain`, this bug only exists in master branch, introduced by apache#40202 3. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by apache#33262 4. regression test add `foreachFrontends`, `foreachBackends`, `retry` function (cherry picked from commit e718c0b)

1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join` 2. fix failed test `explain`, this bug only exists in master branch, introduced by #40202 3. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by #33262 4. regression test add `foreachFrontends`, `foreachBackends`, `retry` function

…46506) 1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join` 2. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by #33262 3. regression test add `foreachFrontends`, `foreachBackends`, `retry` function

1. use retry to fix unstable test `colocate_union_numbers`, `prune_bucket_with_bucket_shuffle_join` 2. fix failed test `explain`, this bug only exists in master branch, introduced by apache#40202 3. fix sql cache bug which use stale cache after drop table and create(table id changed), test in `parse_sql_from_sql_cache`, introduced by apache#33262 4. regression test add `foreachFrontends`, `foreachBackends`, `retry` function (cherry picked from commit e718c0b)

…erException (#48902) fix prepare statement with sql cache throw NullPointerException, introduced by #33262. forbid sql cache when useServerPrepStmts=true, the mysql protocol has different between COM_QUERY and COM_STMT_EXECUTE, the sql cache use the result of COM_QUERY, so we can not provide the result of sql cache for COM_STMT_EXECUTE

…erException (apache#48902) fix prepare statement with sql cache throw NullPointerException, introduced by apache#33262. forbid sql cache when useServerPrepStmts=true, the mysql protocol has different between COM_QUERY and COM_STMT_EXECUTE, the sql cache use the result of COM_QUERY, so we can not provide the result of sql cache for COM_STMT_EXECUTE (cherry picked from commit 74ad0e3)

…erException (apache#48902) fix prepare statement with sql cache throw NullPointerException, introduced by apache#33262. forbid sql cache when useServerPrepStmts=true, the mysql protocol has different between COM_QUERY and COM_STMT_EXECUTE, the sql cache use the result of COM_QUERY, so we can not provide the result of sql cache for COM_STMT_EXECUTE

…che=true (apache#42032) (apache#42034) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by apache#33262

zhangm365 · 2025-10-24T08:30:19Z

hello, how to invalidate cache?
@924060929 thanks.

@0

Before this pr, the query must pass through parser, analyzer, rewriter, optimizer and translator, then we can check whether this query can use sql cache, if the query is too long, or the number of join tables too big, the plan time usually >= 500ms. This pr reduce this time by skip the fashion plan path, because we can reuse the previous physical plan and query result if no any changed. In some cases we should not parse sql from sql cache, e.g. table structure changed, data changed, user policies changed, privileges changed, contains non-deterministic functions, and user variables changed. In my test case: query a view which has lots of join and union, and the tables has empty partition, the query latency is about 3ms. if not parse sql from sql cache, the plan time is about 550ms ## Features 1. use Config.sql_cache_manage_num to control how many sql cache be reused in on fe 2. if explain plan appear some plans contains `LogicalSqlCache` or `PhysicalSqlCache`, it means the query can use sql cache, like this: ```sql mysql> set enable_sql_cache=true; Query OK, 0 rows affected (0.00 sec) mysql> explain physical plan select * from test.t; +----------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +----------------------------------------------------------------------------------+ | cost = 3.135 | | PhysicalResultSink[53] ( outputExprs=[c1#0, c2#1] ) | | +--PhysicalDistribute[50]@0 ( stats=3, distributionSpec=DistributionSpecGather ) | | +--PhysicalOlapScan[t]@0 ( stats=3 ) | +----------------------------------------------------------------------------------+ 4 rows in set (0.02 sec) mysql> select * from test.t; +------+------+ | c1 | c2 | +------+------+ | 1 | 2 | | -2 | -2 | | NULL | 30 | +------+------+ 3 rows in set (0.05 sec) mysql> explain physical plan select * from test.t; +-------------------------------------------------------------------------------------------+ | Explain String(Nereids Planner) | +-------------------------------------------------------------------------------------------+ | cost = 0.0 | | PhysicalSqlCache[2] ( queryId=78511f515cda466b-95385d892d6c68d0, backend=127.0.0.1:9050 ) | | +--PhysicalResultSink[52] ( outputExprs=[c1#0, c2#1] ) | | +--PhysicalDistribute[49]@0 ( stats=3, distributionSpec=DistributionSpecGather ) | | +--PhysicalOlapScan[t]@0 ( stats=3 ) | +-------------------------------------------------------------------------------------------+ 5 rows in set (0.01 sec) ```

…ontends (apache#33722) fix some sql cache consistence bug between multiple frontends which introduced by [enhancement](Nereids) Enable parse sql from sql cache apache#33262, fix by use row policy as the part of sql cache key. support dynamic update the num of fe manage sql cache key

…cache=true (apache#42032) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by apache#33262

…yy-MM-dd HH:mm:ss') (apache#44631) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') which introduced by apache#33262 the wrong result: is `yyyy-MM-dd HH:mm:ss`

924060929 force-pushed the optimize_cache branch from fae1c39 to 00448f5Compare April 4, 2024 18:18

924060929 force-pushed the optimize_cache branch from f231da4 to 75cbd4eCompare April 5, 2024 10:35

924060929 force-pushed the optimize_cache branch from 75cbd4e to 9562c3aCompare April 7, 2024 03:38

924060929 force-pushed the optimize_cache branch from a764541 to 62c26ddCompare April 7, 2024 04:54

morrySnow added dev/2.1.3-merged and removed 2.1.0-conflict labels May 23, 2024

924060929 mentioned this pull request Oct 17, 2024
[fix](Nereids) fix Is Cached is not Yes in Profile when enable_sql_cache=true #42032
Merged

924060929 added a commit that referenced this pull request Oct 18, 2024
[fix](Nereids) fix Is Cached is not Yes in Profile when enable_sql_…
1cd3d9d
…cache=true (#42032) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by #33262

924060929 added a commit that referenced this pull request Oct 18, 2024
[fix](Nereids) fix Is Cached is not Yes in Profile when enable_sql_ca…
c27e792
…che=true (#42032) (#42037) fix Is Cached is not Yes in Profile when enable_sql_cache=true, introduced by #33262

924060929 added a commit that referenced this pull request Oct 18, 2024
[fix](Nereids) fix Is Cached is not Yes in Profile when enable_sql_ca…
1236cfd
…che=true (#42032) (#42034) fix `Is Cached` is not Yes in Profile when enable_sql_cache=true, introduced by #33262

924060929 mentioned this pull request Nov 27, 2024
[fix](sql_cache) fix sql cache result wrong of from_unixtime(col, 'yyyy-MM-dd HH:mm:ss') #44631
Merged
16 tasks

justfortaste mentioned this pull request Dec 26, 2024
[Bug] preparestmt not support sql_cache #46008
Closed
3 tasks

924060929 mentioned this pull request Jan 6, 2025
[fix](nereids) fix sql cache bug and some tests #46443
Merged
16 tasks

924060929 mentioned this pull request Mar 11, 2025
[fix](sql cache) fix prepare statement with sql cache throw NullPointerException #48902
Merged
16 tasks

Uh oh!

Conversation

924060929 commented Apr 4, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Features

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

924060929 commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

doris-robot commented Apr 4, 2024

Uh oh!

924060929 commented Apr 5, 2024

Uh oh!

doris-robot commented Apr 5, 2024

Uh oh!

doris-robot commented Apr 5, 2024

Uh oh!

doris-robot commented Apr 5, 2024

Uh oh!

doris-robot commented Apr 5, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

doris-robot commented Apr 7, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

github-actionsBot commented Apr 7, 2024

Uh oh!

924060929 commented Apr 7, 2024

Uh oh!

zhangm365 commented Oct 24, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

924060929 commented Apr 4, 2024•
edited
Loading