[enhance-wip](multi-catalog) Speed up consume rate of hms events.

[dutyu]

Proposed changes

The current implement will persist all catalogs/databases of external catalogs, and only the master FE can handle hms events and make all slave nodes replay these events, this will bring some problems:

• The hms event processor ( MetastoreEventsProcessor ) can not consume the events in time. (Add journal log is a synchronized method, we can not speed up the consume rate by using concurrent processing, and each add-journal-log operation costs about tens of milliseconds) So the meta info of hive maybe out of date.

• Slave FE nodes maybe crashed if FE replays the journal logs of hms events failed. (In fact we have fixed some issues about this, but we can not make sure all the issues have been resolved)

• There are many journal logs which are produced by hms events, but in fact these logs are not used anymore after FE restart. It makes the start time of all FE nodes very long.

Now doris try to persis all databases/tables of external catalogs just to make sure that the dbId/tableId of databases/tables are the same through all FE nodes, it will be used by analysis jobs.

In this pr, we use a meta id manager called ExternalMetaIdMgr to manage these meta ids. On every loop when master fetches a batch of hms events, it handles the meta ids first and produce only one meta id mappings log, slave FE nodes will replay this log to sync the changes about these meta ids. MetastoreEventsProcessor will start on every FE nodes and try to consume these hms events as soon as possible.

Further comments

I've submitted two prs ( #22869#21589 ) to speed up the consume rate of hms events before, it works fine when there are many AlterTableEvent / DropTableEvent on hive cluster. But the improvement is not that significant when most of hms events are partition-events. Unfortunately, we performed a cluster upgrade (upgrade spark 2.x to spark 3.x), maybe this is the reason that resulting in the majority of Hive Metastore events became partition-events. This is also the reason for the existence of this pull request.

Based on our observation, after merging this pull request, Doris is now capable of processing thousands of Hive Metastore events per second, compared to the previous capability of handling only a few dozen events.

2023-12-0705:17:03,518INFO (replayer|105) [Env.replayJournal():2614] replayedjournalidis18287902, replaytojournalidis182879032023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEventFactory.mergeEvents():188] Eventsizeoncatalog [xxx] beforemergeis [1947], aftermergeis [1849]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955309EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[partitions=2022-05-27],partitionNameAfter:[partitions=2022-05-27]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955310EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[pday=20230318],partitionNameAfter:[pday=20230318]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955311EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[pday=20190826],partitionNameAfter:[pday=20190826]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955312EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[partitions=2021-09-16],partitionNameAfter:[partitions=2021-09-16]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955314EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[partitions=2020-04-26],partitionNameAfter:[partitions=2020-04-26]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955315EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[pday=20230702],partitionNameAfter:[pday=20230702]
2023-12-0705:17:03,735INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357955317EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[pday=20211019],partitionNameAfter:[pday=20211019]
...
2023-12-0705:17:03,989INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357957252EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[partitions=2021-08-27],partitionNameAfter:[partitions=2021-08-27]
2023-12-0705:17:03,989INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEvent.infoLog():193] EventId: 357957253EventType: ALTER_PARTITIONcatalogName:[xxx],dbName:[xxx],tableName:[xxx],partitionNameBefore:[partitions=2022-02-05],partitionNameAfter:[partitions=2022-02-05]
2023-12-0705:17:04,661INFO (replayer|105) [Env.replayJournal():2614] replayedjournalidis18287903, replaytojournalidis182879042023-12-0705:17:05,028INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEventsProcessor.realRun():116] Eventssizeare587oncatalog [xxx]
2023-12-0705:17:05,662INFO (org.apache.doris.datasource.hive.event.MetastoreEventsProcessor|37) [MetastoreEventFactory.mergeEvents():188] Eventsizeoncatalog [xxx] beforemergeis [587], aftermergeis [587]

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 22e3e717fa29be6e64fd883f3379de002f2693f3, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4914	4647	4650	4647
q2	359	156	158	156
q3	2052	1910	1918	1910
q4	1387	1268	1243	1243
q5	3921	3920	3989	3920
q6	255	130	129	129
q7	1363	853	840	840
q8	2794	2819	2785	2785
q9	9885	9667	9537	9537
q10	3474	3549	3497	3497
q11	388	254	243	243
q12	440	294	302	294
q13	4593	3845	3799	3799
q14	324	280	290	280
q15	591	533	521	521
q16	506	440	450	440
q17	1145	969	927	927
q18	7912	7520	7645	7520
q19	1664	1708	1688	1688
q20	592	331	280	280
q21	4497	4012	3997	3997
q22	479	367	380	367
Total cold run time: 53535 ms
Total hot run time: 49020 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4583	4563	4584	4563
q2	336	227	262	227
q3	4041	4028	4029	4028
q4	2718	2722	2741	2722
q5	9551	9646	9594	9594
q6	243	122	122	122
q7	2964	2501	2482	2482
q8	4468	4467	4472	4467
q9	13253	13079	13190	13079
q10	4050	4136	4187	4136
q11	811	645	698	645
q12	974	816	800	800
q13	4292	3589	3583	3583
q14	372	363	362	362
q15	583	520	513	513
q16	596	565	599	565
q17	3979	3929	3924	3924
q18	9637	9200	9168	9168
q19	1833	1786	1774	1774
q20	2406	2066	2048	2048
q21	8859	8484	8691	8484
q22	960	863	824	824
Total cold run time: 81509 ms
Total hot run time: 78110 ms

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.06 seconds
stream load tsv: 570 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098991697 Bytes

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 0f9bc4c5b55547f312c1c8f26820137f84f25789, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4947	4692	4688	4688
q2	365	151	157	151
q3	2049	1926	1914	1914
q4	1395	1267	1266	1266
q5	3958	3942	4005	3942
q6	257	130	130	130
q7	1378	830	881	830
q8	2822	2820	2777	2777
q9	9725	9496	9402	9402
q10	3459	3538	3523	3523
q11	386	241	254	241
q12	438	296	303	296
q13	4565	3826	3800	3800
q14	317	298	283	283
q15	571	530	515	515
q16	500	452	451	451
q17	1143	967	914	914
q18	8123	7496	7658	7496
q19	1673	1702	1664	1664
q20	563	298	319	298
q21	4433	4078	4073	4073
q22	468	371	380	371
Total cold run time: 53535 ms
Total hot run time: 49025 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4612	4592	4591	4591
q2	351	218	296	218
q3	4058	4022	4044	4022
q4	2732	2744	2720	2720
q5	9560	9617	9552	9552
q6	253	127	126	126
q7	3000	2541	2444	2444
q8	4479	4432	4478	4432
q9	12977	12936	12902	12902
q10	4062	4170	4173	4170
q11	764	643	650	643
q12	977	826	807	807
q13	4292	3564	3592	3564
q14	372	368	343	343
q15	567	516	513	513
q16	612	625	567	567
q17	3843	3811	3866	3811
q18	9700	9207	9146	9146
q19	1818	1804	1778	1778
q20	2396	2068	2028	2028
q21	8951	8427	8623	8427
q22	891	873	818	818
Total cold run time: 81267 ms
Total hot run time: 77622 ms

[dutyu]

run feut

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c265929721a8f1b8122f74273b17098362e8cccd, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4876	4632	4657	4632
q2	369	146	158	146
q3	1501	1364	1279	1279
q4	1156	999	961	961
q5	3249	3248	3217	3217
q6	254	130	128	128
q7	971	517	558	517
q8	2261	2257	2242	2242
q9	6948	7008	6976	6976
q10	3322	3382	3375	3375
q11	347	209	201	201
q12	351	216	217	216
q13	4624	4254	3874	3874
q14	246	210	212	210
q15	584	534	524	524
q16	433	366	374	366
q17	1020	692	640	640
q18	8332	7882	7430	7430
q19	1556	1540	1537	1537
q20	607	362	327	327
q21	3392	3026	2999	2999
q22	369	298	308	298
Total cold run time: 46768 ms
Total hot run time: 42095 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4585	4587	4601	4587
q2	320	204	218	204
q3	3758	3750	3766	3750
q4	2538	2522	2518	2518
q5	6206	6213	6221	6213
q6	246	122	123	122
q7	2639	1972	1964	1964
q8	3761	3707	3749	3707
q9	9578	9519	9501	9501
q10	4068	4143	4144	4143
q11	630	486	514	486
q12	817	621	637	621
q13	4393	3641	3615	3615
q14	284	245	250	245
q15	587	535	521	521
q16	515	463	515	463
q17	2103	2050	2085	2050
q18	9529	9028	9234	9028
q19	1837	1779	1771	1771
q20	2301	2012	1986	1986
q21	7378	7032	7080	7032
q22	664	570	566	566
Total cold run time: 68737 ms
Total hot run time: 65093 ms

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.59 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17164038888 Bytes

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.37 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17163918863 Bytes

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit b8721173c707137c9c3527f14c8feea324cf3dcb, data reload: true
run tpch-sf100 query with default conf and session variables
q1	4882	4592	4605	4592
q2	374	129	127	127
q3	1529	1297	1241	1241
q4	1540	945	921	921
q5	3327	3303	3275	3275
q6	250	120	119	119
q7	1003	513	490	490
q8	2239	2243	2216	2216
q9	7088	6991	7364	6991
q10	3304	3384	3391	3384
q11	350	197	201	197
q12	353	209	209	209
q13	4710	3864	3875	3864
q14	245	217	203	203
q15	587	530	512	512
q16	418	372	398	372
q17	1029	649	623	623
q18	7963	7440	7666	7440
q19	1584	1563	1550	1550
q20	519	306	288	288
q21	3398	2985	3009	2985
q22	366	294	292	292
Total cold run time: 47058 ms
Total hot run time: 41891 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4587	4603	4595	4595
q2	315	204	228	204
q3	3733	3751	3735	3735
q4	2535	2517	2512	2512
q5	6198	6172	6187	6172
q6	244	127	122	122
q7	2557	1997	1958	1958
q8	3762	3702	3773	3702
q9	9488	9421	9470	9421
q10	4101	4180	4197	4180
q11	644	547	524	524
q12	793	657	604	604
q13	4363	3642	3628	3628
q14	286	249	239	239
q15	583	529	522	522
q16	517	442	484	442
q17	2089	2092	2078	2078
q18	9515	9043	9126	9043
q19	1808	1768	1764	1764
q20	2311	1995	1983	1983
q21	7296	6904	7146	6904
q22	633	581	599	581
Total cold run time: 68358 ms
Total hot run time: 64913 ms

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.8 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17167276149 Bytes

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.11 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17163998614 Bytes

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.42 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17164002408 Bytes

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5447e66d44ffbc127d7e2bf87f155cc254bf035b, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4744	4477	4490	4477
q2	370	144	159	144
q3	1449	1231	1215	1215
q4	1107	921	897	897
q5	3173	3175	3181	3175
q6	248	131	130	130
q7	990	494	476	476
q8	2169	2222	2175	2175
q9	6700	6668	6643	6643
q10	3206	3270	3290	3270
q11	318	205	205	205
q12	353	209	214	209
q13	4559	3782	3778	3778
q14	240	211	221	211
q15	574	515	529	515
q16	436	384	380	380
q17	1018	607	557	557
q18	7098	6888	6920	6888
q19	1551	1449	1433	1433
q20	552	308	287	287
q21	3055	2637	2688	2637
q22	352	283	292	283
Total cold run time: 44262 ms
Total hot run time: 39985 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4394	4397	4407	4397
q2	266	165	171	165
q3	3495	3496	3480	3480
q4	2386	2369	2370	2369
q5	5696	5694	5675	5675
q6	238	123	121	121
q7	2356	1835	1871	1835
q8	3533	3513	3543	3513
q9	9013	9023	9002	9002
q10	3921	3980	3992	3980
q11	490	389	378	378
q12	762	617	617	617
q13	4280	3594	3555	3555
q14	296	254	241	241
q15	579	535	524	524
q16	510	454	467	454
q17	1896	1837	1851	1837
q18	8596	8055	8029	8029
q19	1721	1760	1786	1760
q20	2256	1948	1937	1937
q21	6506	6136	6116	6116
q22	494	411	418	411
Total cold run time: 63684 ms
Total hot run time: 60396 ms

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.39 seconds
stream load tsv: 604 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17221354068 Bytes

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.46 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17183830231 Bytes

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 5d3885608df7df71b95ec25aa5e712a05fec46be, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4696	4443	4459	4443
q2	359	150	156	150
q3	1462	1213	1217	1213
q4	1121	920	857	857
q5	3136	3136	3155	3136
q6	247	128	129	128
q7	976	478	489	478
q8	2231	2236	2182	2182
q9	6719	6666	6632	6632
q10	3218	3247	3280	3247
q11	308	184	188	184
q12	349	209	208	208
q13	4561	3825	3816	3816
q14	243	215	218	215
q15	579	528	538	528
q16	435	392	387	387
q17	1005	593	575	575
q18	7206	6969	6873	6873
q19	1508	1368	1439	1368
q20	496	290	296	290
q21	3050	2672	2709	2672
q22	346	278	279	278
Total cold run time: 44251 ms
Total hot run time: 39860 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4351	4338	4355	4338
q2	268	167	180	167
q3	3544	3527	3518	3518
q4	2414	2375	2362	2362
q5	5740	5753	5772	5753
q6	240	121	124	121
q7	2362	1919	1846	1846
q8	3507	3505	3509	3505
q9	9121	8996	8987	8987
q10	3885	4005	3991	3991
q11	478	374	362	362
q12	765	586	592	586
q13	4283	3570	3583	3570
q14	286	248	270	248
q15	576	529	524	524
q16	502	472	484	472
q17	1854	1852	1851	1851
q18	8623	8233	8315	8233
q19	1735	1740	1759	1740
q20	2276	1966	1921	1921
q21	6509	6172	6165	6165
q22	501	430	401	401
Total cold run time: 63820 ms
Total hot run time: 60661 ms

[dutyu]

run buildall

[doris-robot]

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit c0c0e137de47d6b3b2ac634161f9027cb4bcba0a, data reload: false
run tpch-sf100 query with default conf and session variables
q1	4718	4410	4415	4410
q2	372	143	159	143
q3	1449	1246	1250	1246
q4	1095	907	914	907
q5	3168	3165	3183	3165
q6	252	138	133	133
q7	1004	499	486	486
q8	2175	2207	2197	2197
q9	6676	6673	6649	6649
q10	3204	3287	3283	3283
q11	308	182	189	182
q12	350	210	208	208
q13	4525	3829	3765	3765
q14	243	218	215	215
q15	568	522	521	521
q16	437	394	391	391
q17	1000	653	602	602
q18	7125	6721	6805	6721
q19	1530	1442	1387	1387
q20	543	298	286	286
q21	3075	2641	2645	2641
q22	351	280	281	280
Total cold run time: 44168 ms
Total hot run time: 39818 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4347	4363	4343	4343
q2	277	166	174	166
q3	3521	3503	3496	3496
q4	2404	2371	2383	2371
q5	5702	5702	5694	5694
q6	241	121	121	121
q7	2365	1900	1835	1835
q8	3517	3522	3518	3518
q9	9025	9059	9018	9018
q10	3893	3997	4013	3997
q11	496	368	367	367
q12	765	589	586	586
q13	4282	3549	3570	3549
q14	290	255	251	251
q15	579	522	520	520
q16	515	464	462	462
q17	1883	1849	1853	1849
q18	8444	8132	8159	8132
q19	1731	1782	1740	1740
q20	2263	1956	1960	1956
q21	6482	6178	6130	6130
q22	519	412	420	412
Total cold run time: 63541 ms
Total hot run time: 60513 ms

[doris-robot]

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.32 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17183123141 Bytes

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c8be27d08e4568e18cf32b3e58cb85c0b2316cfc with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 12.5 seconds inserted 10000000 Rows, about 800K ops/s

[morningman]

LGTM

[morningman]

run buildall

[github-actions]

PR approved by at least one committer and no changes requested.

[doris-robot]

TPC-H: Total hot run time: 39080 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c8e2e531a1f352af034f4404f2abab31e67635c, data reload: false
------ Round 1 ----------------------------------
q1	18387	5424	5299	5299
q2	2193	147	143	143
q3	11407	1206	1210	1206
q4	10909	738	852	738
q5	7919	3282	3228	3228
q6	203	124	124	124
q7	895	519	493	493
q8	9309	1943	1986	1943
q9	7317	6367	6336	6336
q10	8224	3041	2998	2998
q11	425	225	213	213
q12	363	190	192	190
q13	17999	3336	3331	3331
q14	253	210	216	210
q15	548	504	503	503
q16	425	398	403	398
q17	944	541	503	503
q18	7653	6821	6969	6821
q19	1591	1347	1339	1339
q20	603	291	293	291
q21	2812	2457	2456	2456
q22	357	324	317	317
Total cold run time: 110736 ms
Total hot run time: 39080 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5440	5289	5192	5192
q2	323	216	216	216
q3	3339	3175	3272	3175
q4	2103	2034	2067	2034
q5	6052	5867	5857	5857
q6	198	119	116	116
q7	2326	1868	1927	1868
q8	3233	3395	3385	3385
q9	8862	8890	8780	8780
q10	3885	3834	3825	3825
q11	577	448	464	448
q12	813	647	648	647
q13	16931	3149	3113	3113
q14	282	261	263	261
q15	556	510	514	510
q16	532	474	469	469
q17	1868	1807	1841	1807
q18	9513	19441	9837	9837
q19	23409	1552	1535	1535
q20	4598	1947	1925	1925
q21	14538	5435	5518	5435
q22	1027	556	599	556
Total cold run time: 110405 ms
Total hot run time: 60991 ms

[doris-robot]

TPC-DS: Total hot run time: 176819 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c8e2e531a1f352af034f4404f2abab31e67635c, data reload: false
query1	942	343	326	326
query2	6547	1862	1860	1860
query3	6734	200	196	196
query4	30315	22430	22255	22255
query5	6898	531	556	531
query6	257	185	163	163
query7	4600	273	262	262
query8	233	179	174	174
query9	8332	2554	2538	2538
query10	424	246	225	225
query11	17078	15510	15440	15440
query12	125	74	70	70
query13	1682	379	399	379
query14	10574	6983	6943	6943
query15	212	178	178	178
query16	4599	234	233	233
query17	984	479	489	479
query18	1787	255	256	255
query19	178	134	140	134
query20	68	72	69	69
query21	187	142	120	120
query22	4929	4689	4550	4550
query23	31734	30876	30906	30876
query24	12669	2803	2810	2803
query25	577	307	315	307
query26	1808	145	143	143
query27	3230	268	268	268
query28	7073	1823	1825	1823
query29	2146	642	635	635
query30	286	135	135	135
query31	926	743	761	743
query32	81	52	49	49
query33	716	208	213	208
query34	1167	457	462	457
query35	884	787	813	787
query36	1319	1246	1213	1213
query37	176	61	58	58
query38	3380	3188	3209	3188
query39	1317	1269	1259	1259
query40	342	89	85	85
query41	40	36	36	36
query42	94	82	86	82
query43	518	497	508	497
query44	1066	689	705	689
query45	197	184	185	184
query46	1062	671	646	646
query47	1651	1589	1627	1589
query48	403	308	317	308
query49	1222	296	295	295
query50	693	312	310	310
query51	5382	5197	5278	5197
query52	95	75	79	75
query53	324	257	257	257
query54	897	438	447	438
query55	84	78	81	78
query56	189	168	169	168
query57	996	918	884	884
query58	192	162	163	162
query59	2904	2812	2768	2768
query60	217	181	191	181
query61	90	91	86	86
query62	656	386	410	386
query63	287	265	270	265
query64	6171	1793	1762	1762
query65	3343	3244	3238	3238
query66	1429	323	316	316
query67	15475	15100	14862	14862
query68	12117	541	535	535
query69	612	308	303	303
query70	1636	1435	1542	1435
query71	565	216	208	208
query72	4945	2847	2847	2847
query73	2771	317	317	317
query74	6975	6339	6462	6339
query75	5223	2296	2275	2275
query76	6194	1052	1035	1035
query77	799	241	245	241
query78	9154	8606	8649	8606
query79	1043	510	521	510
query80	600	325	319	319
query81	448	210	206	206
query82	201	78	79	78
query83	159	118	118	118
query84	273	67	69	67
query85	1061	352	315	315
query86	380	418	385	385
query87	3517	3353	3304	3304
query88	3087	2171	2158	2158
query89	453	357	360	357
query90	2020	186	185	185
query91	165	124	129	124
query92	57	46	45	45
query93	1457	448	449	448
query94	1186	159	158	158
query95	510	461	454	454
query96	621	316	310	310
query97	4274	4092	4144	4092
query98	213	189	181	181
query99	1090	736	688	688
Total cold run time: 296999 ms
Total hot run time: 176819 ms

[doris-robot]

ClickBench: Total hot run time: 31.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c8e2e531a1f352af034f4404f2abab31e67635c, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.04	0.05
query4	1.70	0.06	0.07
query5	0.54	0.53	0.52
query6	1.23	0.64	0.65
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.55	0.49	0.51
query10	0.57	0.56	0.56
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.60	0.61	0.61
query14	0.77	0.80	0.78
query15	0.80	0.78	0.78
query16	0.36	0.39	0.37
query17	1.02	1.00	1.02
query18	0.24	0.26	0.24
query19	1.84	1.77	1.78
query20	0.01	0.01	0.00
query21	15.41	0.59	0.58
query22	2.57	2.74	2.17
query23	17.59	0.77	0.70
query24	2.29	0.77	1.29
query25	0.38	0.12	0.13
query26	0.63	0.14	0.14
query27	0.05	0.06	0.06
query28	12.10	0.79	0.77
query29	12.61	3.25	3.25
query30	0.52	0.49	0.50
query31	2.78	0.34	0.35
query32	3.37	0.49	0.49
query33	3.21	3.24	3.21
query34	15.71	4.28	4.22
query35	4.30	4.27	4.30
query36	1.11	1.07	1.08
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.12	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.84 s
Total hot run time: 31.02 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8c8e2e531a1f352af034f4404f2abab31e67635c with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s

[morningman]

run buildall

[doris-robot]

TPC-H: Total hot run time: 38766 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7728d9d7642969635ac9009dfb46b8da48516a3a, data reload: false
------ Round 1 ----------------------------------
q1	17660	5200	5254	5200
q2	2031	153	141	141
q3	10618	1181	1154	1154
q4	10220	790	869	790
q5	7711	3097	3152	3097
q6	202	128	127	127
q7	849	512	486	486
q8	9233	1962	1959	1959
q9	7200	6377	6348	6348
q10	8177	3078	3060	3060
q11	423	198	199	198
q12	360	201	207	201
q13	17996	3381	3375	3375
q14	250	217	223	217
q15	552	520	507	507
q16	430	380	397	380
q17	955	518	516	516
q18	7387	6909	6707	6707
q19	2005	1329	1415	1329
q20	598	311	280	280
q21	2768	2385	2490	2385
q22	360	326	309	309
Total cold run time: 107985 ms
Total hot run time: 38766 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5463	5375	5234	5234
q2	337	219	209	209
q3	3320	3276	3241	3241
q4	2090	2070	2054	2054
q5	6019	5894	5877	5877
q6	203	123	122	122
q7	2278	1914	1835	1835
q8	3233	3401	3376	3376
q9	8997	8875	8794	8794
q10	5354	3748	3851	3748
q11	704	466	447	447
q12	784	623	633	623
q13	16986	3140	3125	3125
q14	299	255	266	255
q15	556	514	508	508
q16	531	473	487	473
q17	1900	1818	1905	1818
q18	9477	9299	9545	9299
q19	23513	1586	1561	1561
q20	4593	1927	1913	1913
q21	14215	5351	5483	5351
q22	981	516	548	516
Total cold run time: 111833 ms
Total hot run time: 60379 ms

[doris-robot]

TPC-DS: Total hot run time: 177060 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7728d9d7642969635ac9009dfb46b8da48516a3a, data reload: false
query1	927	332	338	332
query2	6564	1873	1912	1873
query3	6705	196	202	196
query4	31722	22364	22189	22189
query5	6869	530	627	530
query6	251	173	174	173
query7	4606	263	259	259
query8	252	175	198	175
query9	8349	2600	2585	2585
query10	422	234	239	234
query11	17062	15622	15598	15598
query12	122	71	67	67
query13	1697	385	388	385
query14	10492	6843	6940	6843
query15	212	179	188	179
query16	4598	243	229	229
query17	983	485	480	480
query18	1783	272	249	249
query19	175	135	133	133
query20	73	74	72	72
query21	183	137	121	121
query22	4885	4746	4712	4712
query23	31357	30843	31024	30843
query24	11782	2830	2819	2819
query25	563	324	314	314
query26	1629	143	150	143
query27	3226	273	276	273
query28	7109	1851	1845	1845
query29	1490	621	624	621
query30	284	135	140	135
query31	961	740	746	740
query32	78	49	50	49
query33	696	219	215	215
query34	1111	475	445	445
query35	890	777	781	777
query36	1339	1196	1256	1196
query37	92	58	59	58
query38	3369	3270	3235	3235
query39	1332	1263	1282	1263
query40	213	84	83	83
query41	39	35	36	35
query42	93	94	88	88
query43	521	484	445	445
query44	1062	693	703	693
query45	201	188	177	177
query46	1069	660	652	652
query47	1638	1607	1582	1582
query48	395	346	333	333
query49	1130	287	289	287
query50	668	310	306	306
query51	5417	5233	5228	5228
query52	88	79	75	75
query53	319	266	253	253
query54	888	436	458	436
query55	84	79	77	77
query56	177	173	173	173
query57	994	919	941	919
query58	189	169	172	169
query59	2957	2678	2667	2667
query60	211	187	189	187
query61	92	84	85	84
query62	636	358	346	346
query63	285	256	259	256
query64	5086	1793	1747	1747
query65	3343	3264	3279	3264
query66	1293	327	319	319
query67	15746	15498	14984	14984
query68	12359	545	548	545
query69	608	295	300	295
query70	1773	1522	1494	1494
query71	493	216	214	214
query72	4969	2832	2829	2829
query73	3105	316	313	313
query74	6951	6371	6507	6371
query75	4893	2336	2265	2265
query76	5742	1038	1034	1034
query77	678	235	234	234
query78	9376	8877	8500	8500
query79	1030	495	507	495
query80	547	326	323	323
query81	468	206	210	206
query82	199	80	79	79
query83	141	120	127	120
query84	276	68	69	68
query85	1050	321	324	321
query86	389	370	370	370
query87	3485	3416	3292	3292
query88	2860	2177	2160	2160
query89	421	347	346	346
query90	1955	188	185	185
query91	164	121	134	121
query92	51	47	44	44
query93	1422	434	426	426
query94	1216	169	158	158
query95	502	462	448	448
query96	618	317	309	309
query97	4267	4150	4176	4150
query98	208	193	193	193
query99	1002	705	702	702
Total cold run time: 294292 ms
Total hot run time: 177060 ms

[doris-robot]

ClickBench: Total hot run time: 30.82 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7728d9d7642969635ac9009dfb46b8da48516a3a, data reload: false
query1	0.03	0.03	0.02
query2	0.06	0.02	0.03
query3	0.23	0.04	0.05
query4	1.71	0.07	0.07
query5	0.53	0.53	0.53
query6	1.29	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.50	0.50
query10	0.55	0.56	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.08
query13	0.60	0.61	0.60
query14	0.77	0.79	0.80
query15	0.80	0.78	0.77
query16	0.37	0.39	0.36
query17	0.99	1.00	1.05
query18	0.24	0.25	0.26
query19	1.87	1.80	1.79
query20	0.02	0.01	0.01
query21	15.42	0.57	0.58
query22	2.30	2.78	1.70
query23	17.27	0.85	0.78
query24	2.44	1.15	1.76
query25	0.42	0.25	0.12
query26	0.57	0.16	0.15
query27	0.06	0.04	0.06
query28	11.30	0.79	0.77
query29	12.79	3.20	3.13
query30	0.53	0.49	0.48
query31	2.78	0.34	0.36
query32	3.37	0.48	0.49
query33	3.25	3.22	3.24
query34	15.81	4.33	4.18
query35	4.23	4.30	4.24
query36	1.12	1.07	1.07
query37	0.06	0.05	0.06
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104.95 s
Total hot run time: 30.82 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7728d9d7642969635ac9009dfb46b8da48516a3a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s

[morningman]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.