[opt](cloud) Support cached cloud partition version for high frequency query by gavinchou · Pull Request #50439 · apache/doris

gavinchou · 2025-04-26T09:17:00Z

What problem does this PR solve?

Support new session variable to get rid of frequent get_version request to meta-service.
set cloud_partition_version_cache_ttl_ms with a suitable value as needed, it means a partition's version may be cached on FE at most X milliseconds.
pro: reduce RPC/IO to meta-service and fdb, increase query throughput and reduce query latency (plan phase)
con: visibility may be delayed when doing point query; data consistency (snapshot) may be broken when doing query wi involved multiple partitions and there is continuous data import;

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-04-26T09:17:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

gavinchou · 2025-04-26T09:17:09Z

run buildall

gavinchou · 2025-04-26T13:55:29Z

run buildall

…y query

gavinchou · 2025-04-26T14:15:19Z

run buildall

doris-robot · 2025-04-26T14:35:10Z

TPC-H: Total hot run time: 34544 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eda01cedced2a1bd7ccaae6cebfa0157f91c7f31, data reload: false
------ Round 1 ----------------------------------
q1	25953	5141	5076	5076
q2	2057	291	182	182
q3	10524	1226	682	682
q4	10230	987	525	525
q5	7924	2433	2390	2390
q6	187	164	132	132
q7	910	754	619	619
q8	9365	1286	1122	1122
q9	7751	5169	5197	5169
q10	6865	2345	1901	1901
q11	492	284	274	274
q12	354	356	227	227
q13	18489	3751	3209	3209
q14	225	223	216	216
q15	545	502	507	502
q16	443	451	406	406
q17	605	858	368	368
q18	7567	7256	7178	7178
q19	2309	1028	541	541
q20	334	324	227	227
q21	4172	3754	2538	2538
q22	1154	1112	1060	1060
Total cold run time: 118455 ms
Total hot run time: 34544 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5305	5145	4993	4993
q2	263	345	242	242
q3	2311	2857	2528	2528
q4	1521	1954	1520	1520
q5	4553	4474	4339	4339
q6	209	165	125	125
q7	2000	1921	1736	1736
q8	2573	2520	2547	2520
q9	7251	7030	7308	7030
q10	2984	3174	2755	2755
q11	566	519	500	500
q12	698	758	603	603
q13	3445	3918	3336	3336
q14	282	299	281	281
q15	555	472	470	470
q16	470	521	479	479
q17	1156	1532	1418	1418
q18	7887	7500	7426	7426
q19	793	758	913	758
q20	1907	1971	1838	1838
q21	5255	4832	4889	4832
q22	1158	1057	1041	1041
Total cold run time: 53142 ms
Total hot run time: 50770 ms

doris-robot · 2025-04-26T14:47:05Z

TPC-DS: Total hot run time: 193061 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eda01cedced2a1bd7ccaae6cebfa0157f91c7f31, data reload: false
query1	1422	1071	1063	1063
query2	6381	1938	1942	1938
query3	11199	4612	4674	4612
query4	25728	23686	23475	23475
query5	4912	602	454	454
query6	300	197	243	197
query7	3980	482	289	289
query8	294	244	239	239
query9	8540	2562	2574	2562
query10	503	299	260	260
query11	15527	15078	14795	14795
query12	162	114	106	106
query13	1574	517	398	398
query14	8691	6065	6197	6065
query15	215	198	167	167
query16	7504	642	495	495
query17	1149	775	604	604
query18	2120	424	332	332
query19	212	190	189	189
query20	129	130	120	120
query21	208	127	110	110
query22	4803	4687	4505	4505
query23	34416	33727	33545	33545
query24	8543	2398	2419	2398
query25	525	454	397	397
query26	1149	287	155	155
query27	2878	508	331	331
query28	4445	2177	2131	2131
query29	692	551	434	434
query30	280	215	198	198
query31	965	868	809	809
query32	72	64	62	62
query33	533	387	326	326
query34	797	889	523	523
query35	823	852	769	769
query36	1000	1030	946	946
query37	111	100	78	78
query38	4234	4313	4111	4111
query39	1489	1443	1459	1443
query40	211	117	105	105
query41	56	50	54	50
query42	117	113	106	106
query43	508	527	482	482
query44	1332	839	855	839
query45	185	171	171	171
query46	836	1023	646	646
query47	1911	1922	1803	1803
query48	379	422	301	301
query49	761	495	423	423
query50	667	689	419	419
query51	4260	4195	4147	4147
query52	106	107	102	102
query53	217	263	185	185
query54	596	585	511	511
query55	82	78	80	78
query56	314	315	286	286
query57	1184	1214	1132	1132
query58	262	264	274	264
query59	2927	2975	2817	2817
query60	334	346	298	298
query61	130	123	121	121
query62	812	729	662	662
query63	221	192	181	181
query64	4029	1046	678	678
query65	4502	4474	4363	4363
query66	1094	401	301	301
query67	15939	15654	15238	15238
query68	8510	893	519	519
query69	470	298	264	264
query70	1217	1102	1092	1092
query71	470	311	304	304
query72	5821	4696	4686	4686
query73	706	582	352	352
query74	8881	9266	8905	8905
query75	3861	3196	2681	2681
query76	3755	1180	751	751
query77	777	362	286	286
query78	10163	10248	9384	9384
query79	1946	804	577	577
query80	598	495	433	433
query81	462	260	217	217
query82	427	127	98	98
query83	278	243	223	223
query84	276	108	85	85
query85	796	354	305	305
query86	333	317	286	286
query87	4425	4435	4319	4319
query88	3407	2260	2244	2244
query89	400	318	278	278
query90	1908	205	246	205
query91	135	135	106	106
query92	80	59	54	54
query93	1543	934	580	580
query94	656	392	303	303
query95	362	288	288	288
query96	477	566	274	274
query97	3176	3204	3165	3165
query98	218	216	201	201
query99	1446	1433	1256	1256
Total cold run time: 280457 ms
Total hot run time: 193061 ms

doris-robot · 2025-04-26T14:52:18Z

ClickBench: Total hot run time: 29.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eda01cedced2a1bd7ccaae6cebfa0157f91c7f31, data reload: false
query1	0.03	0.03	0.03
query2	0.12	0.10	0.11
query3	0.24	0.19	0.19
query4	1.60	0.18	0.19
query5	0.59	0.61	0.57
query6	1.18	0.72	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.52	0.52
query10	0.57	0.57	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.59
query14	1.17	1.20	1.17
query15	0.88	0.85	0.84
query16	0.38	0.37	0.39
query17	1.01	1.00	1.00
query18	0.21	0.20	0.20
query19	1.86	1.80	1.80
query20	0.01	0.02	0.01
query21	15.41	0.88	0.53
query22	0.75	1.34	0.63
query23	14.85	1.39	0.65
query24	7.16	1.83	1.10
query25	0.57	0.24	0.13
query26	0.62	0.16	0.15
query27	0.05	0.05	0.05
query28	9.74	0.83	0.44
query29	12.57	3.99	3.26
query30	0.26	0.08	0.06
query31	2.83	0.59	0.37
query32	3.23	0.54	0.47
query33	3.08	3.06	3.08
query34	15.83	5.18	4.49
query35	4.60	4.54	4.56
query36	0.67	0.50	0.49
query37	0.08	0.06	0.07
query38	0.05	0.05	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.13
query41	0.09	0.03	0.03
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 104.07 s
Total hot run time: 29.86 s

gavinchou · 2025-04-26T19:31:43Z

run buildall

gavinchou · 2025-04-26T19:32:46Z

run buildall

doris-robot · 2025-04-26T20:10:18Z

TPC-H: Total hot run time: 34065 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d025c528065775642501a38aa45e825ba7c02de9, data reload: false
------ Round 1 ----------------------------------
q1	25449	5081	5021	5021
q2	2066	279	181	181
q3	10389	1217	690	690
q4	10212	998	504	504
q5	7538	2357	2251	2251
q6	178	163	130	130
q7	893	735	621	621
q8	9310	1428	1074	1074
q9	6908	5046	5094	5046
q10	6851	2302	1911	1911
q11	465	284	276	276
q12	342	350	226	226
q13	17787	3622	3103	3103
q14	230	235	214	214
q15	535	492	486	486
q16	432	457	397	397
q17	598	844	368	368
q18	7690	7418	7280	7280
q19	2030	992	568	568
q20	330	337	229	229
q21	3781	3357	2441	2441
q22	1044	1048	1048	1048
Total cold run time: 115058 ms
Total hot run time: 34065 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5270	5072	5099	5072
q2	237	329	242	242
q3	2115	2609	2319	2319
q4	1437	1824	1358	1358
q5	4418	4391	4383	4383
q6	220	163	124	124
q7	1999	1942	1750	1750
q8	2564	2516	2515	2515
q9	7230	7073	7057	7057
q10	3014	3220	2738	2738
q11	584	502	483	483
q12	667	765	607	607
q13	3496	3931	3340	3340
q14	296	303	304	303
q15	517	478	475	475
q16	466	491	486	486
q17	1137	1626	1342	1342
q18	7661	7458	7359	7359
q19	756	725	819	725
q20	2016	1987	1825	1825
q21	5261	4936	4865	4865
q22	1088	1072	1044	1044
Total cold run time: 52449 ms
Total hot run time: 50412 ms

doris-robot · 2025-04-26T20:22:07Z

TPC-DS: Total hot run time: 192107 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d025c528065775642501a38aa45e825ba7c02de9, data reload: false
query1	1378	1075	1062	1062
query2	6210	1889	1896	1889
query3	11120	4722	4697	4697
query4	25143	23890	22886	22886
query5	4786	617	454	454
query6	323	209	205	205
query7	3984	477	281	281
query8	300	257	233	233
query9	8500	2555	2560	2555
query10	501	309	264	264
query11	15101	15027	14811	14811
query12	161	111	110	110
query13	1566	516	393	393
query14	9567	6157	6239	6157
query15	212	187	165	165
query16	6975	622	448	448
query17	1305	709	586	586
query18	1876	403	326	326
query19	197	201	176	176
query20	134	124	120	120
query21	205	126	110	110
query22	4474	4579	4196	4196
query23	34299	33579	33426	33426
query24	8400	2370	2420	2370
query25	510	472	392	392
query26	1236	278	150	150
query27	2727	498	331	331
query28	4620	2179	2126	2126
query29	654	566	439	439
query30	270	224	191	191
query31	921	881	798	798
query32	74	60	64	60
query33	552	369	299	299
query34	802	869	553	553
query35	812	840	767	767
query36	966	1019	909	909
query37	114	103	83	83
query38	4281	4239	4134	4134
query39	1548	1432	1450	1432
query40	218	120	113	113
query41	53	53	52	52
query42	117	112	109	109
query43	524	534	487	487
query44	1299	809	822	809
query45	182	174	176	174
query46	862	1040	650	650
query47	1866	1868	1775	1775
query48	382	425	296	296
query49	761	505	433	433
query50	651	705	419	419
query51	4327	4212	4181	4181
query52	117	111	110	110
query53	252	264	183	183
query54	583	590	509	509
query55	82	78	85	78
query56	329	310	308	308
query57	1163	1167	1142	1142
query58	260	260	258	258
query59	2846	2922	2717	2717
query60	330	327	305	305
query61	128	123	126	123
query62	821	733	676	676
query63	217	195	187	187
query64	4278	1015	703	703
query65	4458	4332	4418	4332
query66	990	424	342	342
query67	16001	15646	15302	15302
query68	8242	866	508	508
query69	464	302	269	269
query70	1230	1133	1112	1112
query71	401	315	298	298
query72	5550	4794	4899	4794
query73	650	662	349	349
query74	9027	8876	8806	8806
query75	3173	3228	2649	2649
query76	3215	1192	746	746
query77	500	377	276	276
query78	9956	10289	9299	9299
query79	1950	846	564	564
query80	633	500	482	482
query81	497	254	223	223
query82	374	131	94	94
query83	252	248	278	248
query84	248	112	80	80
query85	750	347	309	309
query86	379	301	294	294
query87	4425	4409	4378	4378
query88	2888	2253	2234	2234
query89	393	311	284	284
query90	1850	213	221	213
query91	141	141	116	116
query92	67	59	58	58
query93	1654	952	583	583
query94	658	424	308	308
query95	386	290	290	290
query96	486	550	277	277
query97	3107	3266	3145	3145
query98	230	220	205	205
query99	1372	1473	1259	1259
Total cold run time: 276049 ms
Total hot run time: 192107 ms

doris-robot · 2025-04-26T20:27:20Z

ClickBench: Total hot run time: 29.2 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d025c528065775642501a38aa45e825ba7c02de9, data reload: false
query1	0.03	0.03	0.03
query2	0.12	0.10	0.11
query3	0.25	0.19	0.20
query4	1.59	0.18	0.19
query5	0.59	0.59	0.59
query6	1.17	0.71	0.72
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.56	0.50	0.53
query10	0.57	0.56	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.10
query13	0.62	0.60	0.59
query14	1.20	1.17	1.20
query15	0.86	0.84	0.85
query16	0.38	0.38	0.38
query17	1.04	1.05	1.03
query18	0.21	0.20	0.19
query19	1.97	1.75	1.82
query20	0.01	0.01	0.01
query21	15.40	0.89	0.55
query22	0.76	1.22	0.72
query23	14.85	1.38	0.61
query24	7.39	2.00	0.41
query25	0.49	0.12	0.15
query26	0.57	0.16	0.13
query27	0.05	0.05	0.05
query28	9.59	0.89	0.44
query29	12.56	3.99	3.33
query30	0.25	0.09	0.07
query31	2.84	0.59	0.39
query32	3.23	0.55	0.46
query33	3.05	3.03	2.99
query34	15.87	5.12	4.53
query35	4.60	4.53	4.55
query36	0.67	0.50	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.19 s
Total hot run time: 29.2 s

github-actions · 2025-05-06T03:20:26Z

PR approved by anyone and no changes requested.

gavinchou · 2025-05-06T04:15:18Z

run buildall

doris-robot · 2025-05-06T04:45:39Z

TPC-H: Total hot run time: 33852 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ee59de2e0587735a0f111af7a77bfb030651d54, data reload: false
------ Round 1 ----------------------------------
q1	26185	5050	5048	5048
q2	2070	265	171	171
q3	10419	1246	704	704
q4	10220	1012	523	523
q5	7511	2320	2358	2320
q6	180	160	131	131
q7	916	761	630	630
q8	9331	1294	1042	1042
q9	6931	5146	5240	5146
q10	6818	2283	1880	1880
q11	499	284	260	260
q12	353	375	213	213
q13	17769	3687	3092	3092
q14	232	230	206	206
q15	523	491	485	485
q16	440	447	376	376
q17	599	846	348	348
q18	7520	7140	7129	7129
q19	1352	940	553	553
q20	336	335	229	229
q21	4324	3430	2440	2440
q22	1046	1030	926	926
Total cold run time: 115574 ms
Total hot run time: 33852 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5180	5085	5110	5085
q2	236	325	239	239
q3	2173	2659	2281	2281
q4	1377	1792	1365	1365
q5	4445	4359	4432	4359
q6	211	175	132	132
q7	2086	1915	1760	1760
q8	2600	2582	2493	2493
q9	7196	7198	6973	6973
q10	3018	3201	2753	2753
q11	575	512	492	492
q12	685	773	618	618
q13	3541	3922	3346	3346
q14	278	302	302	302
q15	553	505	519	505
q16	461	464	426	426
q17	1155	1576	1390	1390
q18	7657	7563	7447	7447
q19	835	854	906	854
q20	2011	2010	1814	1814
q21	5049	4712	4807	4712
q22	1062	1033	982	982
Total cold run time: 52384 ms
Total hot run time: 50328 ms

doris-robot · 2025-05-06T04:57:51Z

TPC-DS: Total hot run time: 190995 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ee59de2e0587735a0f111af7a77bfb030651d54, data reload: false
query1	1396	1100	1071	1071
query2	6111	1822	1814	1814
query3	10979	4528	4455	4455
query4	53554	25560	23430	23430
query5	5205	523	467	467
query6	364	222	199	199
query7	4938	522	286	286
query8	327	251	254	251
query9	5764	2574	2588	2574
query10	415	307	259	259
query11	15123	14992	14912	14912
query12	163	115	131	115
query13	1057	512	402	402
query14	10185	6531	6557	6531
query15	210	186	181	181
query16	7039	662	472	472
query17	1068	747	597	597
query18	1536	439	300	300
query19	204	192	163	163
query20	130	122	130	122
query21	203	124	110	110
query22	4394	4387	4200	4200
query23	34232	33489	33456	33456
query24	6804	2386	2447	2386
query25	465	468	428	428
query26	668	279	157	157
query27	2264	504	352	352
query28	3119	2174	2118	2118
query29	573	587	460	460
query30	283	228	193	193
query31	866	852	797	797
query32	73	64	74	64
query33	480	375	322	322
query34	788	872	540	540
query35	785	829	763	763
query36	928	996	910	910
query37	126	105	79	79
query38	4258	4372	4177	4177
query39	1586	1462	1425	1425
query40	215	125	112	112
query41	58	60	54	54
query42	126	109	117	109
query43	507	516	471	471
query44	1381	813	816	813
query45	187	180	175	175
query46	885	1040	650	650
query47	1810	1866	1770	1770
query48	407	438	322	322
query49	706	505	451	451
query50	745	742	410	410
query51	4197	4219	4186	4186
query52	113	109	102	102
query53	232	267	188	188
query54	625	580	536	536
query55	84	83	84	83
query56	327	301	309	301
query57	1189	1208	1126	1126
query58	269	265	262	262
query59	2750	2808	2666	2666
query60	323	336	311	311
query61	133	129	147	129
query62	725	733	681	681
query63	233	195	195	195
query64	1564	1138	809	809
query65	4430	4335	4356	4335
query66	765	436	321	321
query67	16091	15686	15601	15601
query68	7215	954	501	501
query69	546	302	268	268
query70	1176	1106	1102	1102
query71	518	327	291	291
query72	5781	2339	5147	2339
query73	1486	771	349	349
query74	9004	8996	8878	8878
query75	3911	3218	2706	2706
query76	4229	1196	788	788
query77	640	376	291	291
query78	9987	10079	9330	9330
query79	3044	839	581	581
query80	638	514	459	459
query81	481	263	219	219
query82	452	131	95	95
query83	353	306	234	234
query84	291	100	90	90
query85	797	354	312	312
query86	352	289	281	281
query87	4351	4376	4276	4276
query88	3405	2214	2210	2210
query89	391	311	291	291
query90	1921	216	220	216
query91	145	145	113	113
query92	73	63	59	59
query93	1263	939	567	567
query94	670	404	307	307
query95	367	300	282	282
query96	491	571	274	274
query97	3166	3228	3121	3121
query98	230	218	198	198
query99	1455	1437	1260	1260
Total cold run time: 298569 ms
Total hot run time: 190995 ms

doris-robot · 2025-05-06T05:03:12Z

ClickBench: Total hot run time: 29.5 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ee59de2e0587735a0f111af7a77bfb030651d54, data reload: false
query1	0.04	0.03	0.03
query2	0.11	0.10	0.11
query3	0.25	0.19	0.20
query4	1.60	0.19	0.19
query5	0.60	0.58	0.57
query6	1.18	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.53	0.51
query10	0.59	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.59
query14	0.78	0.81	0.81
query15	0.87	0.86	0.86
query16	0.37	0.38	0.40
query17	1.06	1.08	1.04
query18	0.20	0.19	0.20
query19	1.90	1.81	1.79
query20	0.01	0.02	0.01
query21	15.40	0.89	0.56
query22	0.77	1.42	0.72
query23	15.31	1.36	0.64
query24	7.51	1.11	1.16
query25	0.55	0.30	0.07
query26	0.58	0.15	0.14
query27	0.05	0.05	0.05
query28	9.87	0.84	0.45
query29	12.65	3.96	3.30
query30	0.26	0.09	0.07
query31	2.82	0.59	0.38
query32	3.24	0.55	0.47
query33	3.02	3.07	3.03
query34	15.73	5.14	4.44
query35	4.53	4.49	4.49
query36	0.66	0.50	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 104.56 s
Total hot run time: 29.5 s

dataroaring

LGTM

github-actions · 2025-05-06T08:22:05Z

PR approved by at least one committer and no changes requested.

…y query (#50439) Support new session variable to get rid of frequent get_version request to meta-service. set `cloud_partition_version_cache_ttl_ms` with a suitable value as needed, it means a partition's version may be cached on FE at most X milliseconds. pro: reduce RPC/IO to meta-service and fdb, increase query throughput and reduce query latency (plan phase) con: visibility may be delayed when doing point query; data consistency (snapshot) may be broken when doing query wi involved multiple partitions and there is continuous data import;

…igh frequency query #50439 (#50619) Cherry-picked from #50439 Co-authored-by: Gavin Chou <gavin@selectdb.com>

…y query (apache#50439) Support new session variable to get rid of frequent get_version request to meta-service. set `cloud_partition_version_cache_ttl_ms` with a suitable value as needed, it means a partition's version may be cached on FE at most X milliseconds. pro: reduce RPC/IO to meta-service and fdb, increase query throughput and reduce query latency (plan phase) con: visibility may be delayed when doing point query; data consistency (snapshot) may be broken when doing query wi involved multiple partitions and there is continuous data import;

gavinchou requested review from dataroaring and w41ter as code owners April 26, 2025 09:17

gavinchou added the dev/3.0.x label Apr 26, 2025

[opt](cloud) Support cached cloud partition version for high frequenc…
eda01ce
…y query

gavinchou force-pushed the gavin-opt-cloud-partition-get-version branch from afed61d to eda01ceCompare April 26, 2025 14:15

gavinchou added the usercase Important user case type label label Apr 26, 2025

Add UT
d025c52

gavinchou force-pushed the gavin-opt-cloud-partition-get-version branch from ab071d3 to d025c52Compare April 26, 2025 19:32

Hastyshell approved these changes May 6, 2025
View reviewed changes

github-actionsBot added the reviewed label May 6, 2025

gavinchou added 4 commits May 6, 2025 12:02

Update SessionVariable.java
cd257cd

Update CloudPartition.java
d555ff3

Update CloudPartitionTest.java
3b5e8e9

Update test_fe_cached_partition_version.groovy
3ee59de

dataroaring approved these changes May 6, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label May 6, 2025

gavinchou merged commit f5d25ba into apache:masterMay 6, 2025

github-actionsBot mentioned this pull request May 6, 2025
branch-3.0: [opt](cloud) Support cached cloud partition version for high frequency query #50439 #50619
Merged

dataroaring pushed a commit that referenced this pull request May 14, 2025
branch-3.0: [opt](cloud) Support cached cloud partition version for h…
53e0ab6
…igh frequency query #50439 (#50619) Cherry-picked from #50439 Co-authored-by: Gavin Chou <gavin@selectdb.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 14, 2025

gavinchou mentioned this pull request Jun 11, 2025
Release Note 3.0.6 #51585
Closed

Uh oh!

Conversation

gavinchou commented Apr 26, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Apr 26, 2025

Uh oh!

gavinchou commented Apr 26, 2025

Uh oh!

gavinchou commented Apr 26, 2025

Uh oh!

gavinchou commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

gavinchou commented Apr 26, 2025

Uh oh!

gavinchou commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

doris-robot commented Apr 26, 2025

Uh oh!

github-actionsBot commented May 6, 2025

Uh oh!

gavinchou commented May 6, 2025

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

doris-robot commented May 6, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented May 6, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gavinchou commented Apr 26, 2025•
edited
Loading