[feature](cloud-compaction) Support shadow tablet to do cumulative compaction during schema change in cloud mode by Hastyshell · Pull Request #39558 · apache/doris

Hastyshell · 2024-08-19T08:04:38Z

Proposed changes

In cloud mode, when do schema change, shadow tablet encounters -235 because it cant do cumulative compaction in the case of a large number of loads. And it will prevents the user from continuing to loads. Implementation details:

When start schema change, record the end convert rowset version alter_version into SchemaChangeJob.
For origin tablet, only can do base compaction in [0, alter_version] and do cumulative compaction in (alter_version, N]. can not do compaction across alter_verison such as compaction [a, alter_version + n].
For shadow tablet, cannot do base compaction and and do cumulative compaction in (alter_version, N].
When the schema change failed because FE or BE coredump, it will retry. When retry the schema change, it will get the alter_version from meta_serive, and continue to do it.
When finish the schema change job or cancel it, we need to clear the schema change job. Before this pr, it will cover by next schema change.

co-author(main author): @Lchangliang
original PR: #37293

doris-robot · 2024-08-19T08:04:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Hastyshell · 2024-08-19T08:06:28Z

run buildall

…oud mode (apache#37293) In cloud mode, when do schema change, shadow tablet encounters -235 because it cant do cumulative compaction in the case of a large number of loads. And it will prevents the user from continuing to loads. Implementation details: 1. When start schema change, record the end convert rowset version `alter_version` into SchemaChangeJob. 2. For origin tablet, only can do base compaction in [0, `alter_version`] and do cumulative compaction in (`alter_version`, N]. can not do compaction across `alter_verison` such as compaction [a, `alter_version` + n]. 3. For shadow tablet, cannot do base compaction and and do cumulative compaction in (`alter_version`, N]. 4. When the schema change failed because FE or BE coredump, it will retry. When retry the schema change, it will get the `alter_version` from meta_serive, and continue to do it. 5. When finish the schema change job or cancel it, we need to clear the schema change job. Before this pr, it will cover by next schema change.

Hastyshell · 2024-08-19T10:28:06Z

run buildall

doris-robot · 2024-08-19T12:01:19Z

TPC-H: Total hot run time: 38124 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9097027b56899334b77546c2e702468ee7d80eb, data reload: false
------ Round 1 ----------------------------------
q1	19318	5304	4302	4302
q2	2055	206	232	206
q3	11372	981	1139	981
q4	10518	766	807	766
q5	7793	2868	2769	2769
q6	264	158	157	157
q7	1015	648	648	648
q8	9410	2101	2112	2101
q9	7334	6585	6570	6570
q10	7044	2277	2188	2188
q11	479	276	276	276
q12	433	262	261	261
q13	17790	3037	3009	3009
q14	300	260	258	258
q15	552	519	541	519
q16	517	436	420	420
q17	993	669	774	669
q18	7423	6738	6643	6643
q19	6053	1082	1113	1082
q20	699	354	346	346
q21	3913	2974	2955	2955
q22	1153	998	1017	998
Total cold run time: 116428 ms
Total hot run time: 38124 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4585	4339	4273	4273
q2	415	318	307	307
q3	2905	2607	2690	2607
q4	1965	1770	1664	1664
q5	5697	5684	5643	5643
q6	254	156	151	151
q7	2210	1799	1753	1753
q8	3314	3507	3462	3462
q9	8803	8720	8764	8720
q10	3639	3345	3304	3304
q11	619	519	528	519
q12	835	660	634	634
q13	17090	3212	3100	3100
q14	319	285	302	285
q15	560	525	525	525
q16	514	456	458	456
q17	1850	1580	1532	1532
q18	8271	8069	7811	7811
q19	3779	1400	1472	1400
q20	2159	1884	1880	1880
q21	15757	5395	5322	5322
q22	1153	1099	1048	1048
Total cold run time: 86693 ms
Total hot run time: 56396 ms

doris-robot · 2024-08-19T12:14:06Z

TPC-DS: Total hot run time: 196619 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9097027b56899334b77546c2e702468ee7d80eb, data reload: false
query1	1332	916	900	900
query2	6724	2018	1979	1979
query3	10651	3960	3859	3859
query4	54877	24449	23257	23257
query5	5748	703	733	703
query6	491	214	213	213
query7	5763	333	348	333
query8	532	452	438	438
query9	8959	2542	2514	2514
query10	573	354	335	335
query11	18147	15009	15162	15009
query12	201	145	147	145
query13	1687	510	428	428
query14	11178	7400	7568	7400
query15	295	206	192	192
query16	7611	516	505	505
query17	1190	595	655	595
query18	2013	344	347	344
query19	311	182	176	176
query20	148	137	134	134
query21	247	143	140	140
query22	4702	4365	4382	4365
query23	34738	33766	33848	33766
query24	6223	2966	2998	2966
query25	571	440	430	430
query26	702	191	187	187
query27	1808	299	307	299
query28	4092	2198	2157	2157
query29	704	455	469	455
query30	231	193	190	190
query31	1014	823	824	823
query32	100	77	79	77
query33	497	345	333	333
query34	907	502	505	502
query35	868	755	790	755
query36	1123	991	943	943
query37	156	105	105	105
query38	4008	3874	3889	3874
query39	1517	1476	1469	1469
query40	231	156	154	154
query41	140	139	139	139
query42	135	119	120	119
query43	559	498	522	498
query44	1114	783	778	778
query45	235	195	193	193
query46	1123	770	740	740
query47	1933	1846	1851	1846
query48	439	366	336	336
query49	916	576	578	576
query50	868	462	475	462
query51	6944	6701	6721	6701
query52	122	108	116	108
query53	298	223	221	221
query54	623	505	514	505
query55	92	90	89	89
query56	346	307	309	307
query57	1172	1108	1158	1108
query58	298	301	304	301
query59	2918	2962	2809	2809
query60	349	323	330	323
query61	154	148	145	145
query62	808	704	702	702
query63	251	224	258	224
query64	3188	1828	1871	1828
query65	3225	3192	3162	3162
query66	1037	692	674	674
query67	15244	15140	14933	14933
query68	5722	582	593	582
query69	582	318	312	312
query70	1162	1197	1152	1152
query71	507	317	320	317
query72	7049	2392	1991	1991
query73	826	353	352	352
query74	9317	8752	8864	8752
query75	3521	2795	2731	2731
query76	2524	1036	1041	1036
query77	722	432	449	432
query78	11626	9670	10028	9670
query79	10800	562	550	550
query80	1170	598	616	598
query81	622	260	261	260
query82	748	161	156	156
query83	389	211	213	211
query84	300	97	95	95
query85	847	357	356	356
query86	465	325	322	322
query87	4464	4221	4220	4220
query88	5396	2469	2475	2469
query89	506	331	321	321
query90	2263	234	235	234
query91	157	129	126	126
query92	86	76	75	75
query93	4335	549	544	544
query94	918	330	335	330
query95	381	302	299	299
query96	630	285	280	280
query97	3189	3067	3047	3047
query98	256	227	226	226
query99	1647	1305	1297	1297
Total cold run time: 330528 ms
Total hot run time: 196619 ms

doris-robot · 2024-08-19T12:19:39Z

ClickBench: Total hot run time: 31.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9097027b56899334b77546c2e702468ee7d80eb, data reload: false
query1	0.05	0.05	0.03
query2	0.08	0.04	0.05
query3	0.23	0.06	0.05
query4	1.66	0.07	0.09
query5	0.50	0.50	0.48
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.56	0.49	0.50
query10	0.55	0.55	0.55
query11	0.17	0.13	0.12
query12	0.16	0.13	0.13
query13	0.61	0.60	0.59
query14	0.77	0.78	0.79
query15	0.86	0.82	0.81
query16	0.37	0.37	0.38
query17	1.01	1.00	1.00
query18	0.21	0.19	0.21
query19	1.88	1.66	1.73
query20	0.01	0.01	0.01
query21	15.42	0.67	0.68
query22	4.18	6.41	2.65
query23	18.27	1.42	1.32
query24	2.19	0.22	0.22
query25	0.16	0.09	0.08
query26	0.28	0.18	0.18
query27	0.08	0.09	0.09
query28	13.22	1.03	1.02
query29	12.68	3.31	3.32
query30	0.40	0.20	0.19
query31	2.81	0.42	0.40
query32	3.24	0.49	0.49
query33	2.99	2.94	2.92
query34	17.05	4.40	4.34
query35	4.44	4.40	4.41
query36	0.66	0.49	0.48
query37	0.20	0.18	0.19
query38	0.18	0.16	0.16
query39	0.06	0.05	0.07
query40	0.18	0.15	0.15
query41	0.11	0.07	0.07
query42	0.08	0.07	0.07
query43	0.06	0.06	0.07
Total cold run time: 109.82 s
Total hot run time: 31.64 s

Hastyshell · 2024-08-20T11:15:45Z

run cloud_p0

Hastyshell · 2024-08-20T11:36:47Z

run cloud_p0

Hastyshell · 2024-08-20T14:27:51Z

run cloud_p0

Hastyshell · 2024-08-20T14:51:29Z

+ << compaction.input_versions(0)
+ << " input_version_end=" << compaction.input_versions(1)


Here is supicious

Hastyshell · 2024-08-20T14:57:28Z

run buildall

github-actions

clang-tidy made some suggestions

Hastyshell · 2024-08-20T15:53:05Z

run buildall

doris-robot · 2024-08-20T18:26:26Z

TPC-H: Total hot run time: 39740 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65244ae2e6a11cc9d7cbb74754ab900e2be59565, data reload: false
------ Round 1 ----------------------------------
q1	18266	4662	4535	4535
q2	2080	218	220	218
q3	11539	1021	1195	1021
q4	10544	801	796	796
q5	7825	3034	2921	2921
q6	289	162	161	161
q7	1120	702	682	682
q8	9379	2204	2246	2204
q9	7247	6889	6860	6860
q10	7134	2365	2351	2351
q11	502	289	274	274
q12	472	283	258	258
q13	18782	3010	3018	3010
q14	317	275	257	257
q15	576	532	523	523
q16	534	429	421	421
q17	1046	684	841	684
q18	7539	6804	6890	6804
q19	6591	1245	1181	1181
q20	697	367	361	361
q21	4147	3159	3178	3159
q22	1172	1069	1059	1059
Total cold run time: 117798 ms
Total hot run time: 39740 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4809	4555	4512	4512
q2	437	317	302	302
q3	2999	2845	2850	2845
q4	2104	1781	1742	1742
q5	5855	5773	5752	5752
q6	259	153	147	147
q7	2282	1750	1843	1750
q8	3602	3742	3753	3742
q9	8976	8857	9004	8857
q10	3701	3390	3416	3390
q11	657	520	520	520
q12	870	672	651	651
q13	15944	3178	3066	3066
q14	324	304	291	291
q15	606	545	549	545
q16	536	466	471	466
q17	2018	1588	1614	1588
q18	8363	8006	7802	7802
q19	7568	1776	1802	1776
q20	2210	1935	1925	1925
q21	13951	5476	5358	5358
q22	1219	1078	1100	1078
Total cold run time: 89290 ms
Total hot run time: 58105 ms

doris-robot · 2024-08-20T19:33:34Z

TPC-H: Total hot run time: 38069 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65244ae2e6a11cc9d7cbb74754ab900e2be59565, data reload: false
------ Round 1 ----------------------------------
q1	17851	4476	4316	4316
q2	2065	208	211	208
q3	11358	987	1162	987
q4	10539	791	768	768
q5	7765	2866	2801	2801
q6	268	160	159	159
q7	1017	649	648	648
q8	9387	2073	2104	2073
q9	6918	6535	6557	6535
q10	7038	2218	2222	2218
q11	478	266	270	266
q12	445	268	273	268
q13	17779	2994	2972	2972
q14	299	249	250	249
q15	548	529	541	529
q16	545	409	409	409
q17	986	700	732	700
q18	7346	6963	6688	6688
q19	6251	1081	1138	1081
q20	729	350	350	350
q21	3806	2871	2828	2828
q22	1142	1039	1016	1016
Total cold run time: 114560 ms
Total hot run time: 38069 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4794	4347	4291	4291
q2	414	307	306	306
q3	2840	2616	2612	2612
q4	1931	1744	1749	1744
q5	5633	5650	5706	5650
q6	247	154	158	154
q7	2219	1818	1800	1800
q8	3333	3463	3437	3437
q9	8835	8771	8719	8719
q10	3524	3271	3391	3271
q11	641	543	537	537
q12	867	664	679	664
q13	16433	3176	3188	3176
q14	320	291	283	283
q15	556	518	512	512
q16	499	447	456	447
q17	1853	1575	1563	1563
q18	8273	7831	7726	7726
q19	6979	1594	1662	1594
q20	2204	1939	1890	1890
q21	13885	5458	5398	5398
q22	1173	1080	1075	1075
Total cold run time: 87453 ms
Total hot run time: 56849 ms

Hastyshell · 2024-08-21T08:59:18Z

run buildall

doris-robot · 2024-08-21T10:06:09Z

TPC-H: Total hot run time: 38659 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ea46edd5ef97dd6f0fe853a26c8fcd27db899b5, data reload: false
------ Round 1 ----------------------------------
q1	18417	4772	4451	4451
q2	2650	208	217	208
q3	11616	1166	1243	1166
q4	10576	803	791	791
q5	8373	2920	2918	2918
q6	269	159	157	157
q7	1002	653	648	648
q8	9384	2124	2127	2124
q9	7357	6625	6637	6625
q10	7104	2259	2254	2254
q11	512	283	275	275
q12	427	259	259	259
q13	17798	3033	3035	3033
q14	315	267	264	264
q15	578	497	520	497
q16	519	406	405	405
q17	983	736	690	690
q18	7553	6919	6715	6715
q19	5389	1147	1023	1023
q20	686	346	353	346
q21	4144	2838	2753	2753
q22	1158	1069	1057	1057
Total cold run time: 116810 ms
Total hot run time: 38659 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4527	4290	4283	4283
q2	402	314	286	286
q3	2882	2641	2635	2635
q4	1893	1600	1591	1591
q5	5417	5398	5442	5398
q6	227	144	145	144
q7	2096	1689	1662	1662
q8	3227	3375	3384	3375
q9	8498	8403	8417	8403
q10	3440	3197	3175	3175
q11	633	519	513	513
q12	807	641	649	641
q13	16653	3026	3020	3020
q14	332	295	295	295
q15	566	525	528	525
q16	499	447	465	447
q17	1805	1508	1509	1508
q18	7791	7576	7346	7346
q19	1725	1627	1649	1627
q20	2056	1886	1856	1856
q21	5337	5138	5228	5138
q22	1172	1071	1077	1071
Total cold run time: 71985 ms
Total hot run time: 54939 ms

doris-robot · 2024-08-21T10:18:32Z

TPC-DS: Total hot run time: 191408 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ea46edd5ef97dd6f0fe853a26c8fcd27db899b5, data reload: false
query1	956	414	388	388
query2	6760	2024	1985	1985
query3	6669	244	253	244
query4	34177	23242	23272	23242
query5	4421	710	701	701
query6	318	219	210	210
query7	4606	340	327	327
query8	483	437	432	432
query9	8588	2548	2523	2523
query10	505	352	335	335
query11	17727	14870	15063	14870
query12	193	140	137	137
query13	1717	456	448	448
query14	10200	7192	7256	7192
query15	346	194	196	194
query16	8094	517	497	497
query17	1741	635	592	592
query18	2124	338	338	338
query19	349	173	171	171
query20	142	157	138	138
query21	247	148	140	140
query22	4426	4091	4018	4018
query23	34248	33550	33462	33462
query24	11180	2908	2944	2908
query25	665	425	423	423
query26	1020	184	182	182
query27	2442	300	302	300
query28	7073	2170	2129	2129
query29	826	457	439	439
query30	348	191	192	191
query31	1054	853	843	843
query32	124	84	83	83
query33	816	345	350	345
query34	909	505	518	505
query35	920	766	786	766
query36	1128	976	983	976
query37	163	103	104	103
query38	4030	3854	3860	3854
query39	1548	1475	1474	1474
query40	238	159	163	159
query41	139	139	139	139
query42	141	121	121	121
query43	557	511	506	506
query44	1323	783	785	783
query45	227	201	197	197
query46	1138	778	776	776
query47	1958	1828	1863	1828
query48	411	340	330	330
query49	1206	585	576	576
query50	856	461	472	461
query51	7146	7198	7052	7052
query52	127	112	110	110
query53	302	233	228	228
query54	1056	534	496	496
query55	92	91	94	91
query56	350	313	304	304
query57	1243	1156	1176	1156
query58	326	302	323	302
query59	2967	2813	2875	2813
query60	350	334	326	326
query61	152	149	156	149
query62	880	688	717	688
query63	279	228	227	227
query64	5260	2365	1863	1863
query65	3261	3162	3167	3162
query66	1084	723	677	677
query67	15669	15086	15044	15044
query68	7098	586	595	586
query69	753	438	327	327
query70	1238	1075	1161	1075
query71	556	313	309	309
query72	7414	2357	2082	2082
query73	847	353	371	353
query74	9527	8749	8716	8716
query75	4819	2757	2681	2681
query76	4802	1115	993	993
query77	988	461	448	448
query78	10574	9320	9084	9084
query79	8748	563	570	563
query80	1129	608	612	608
query81	620	259	269	259
query82	726	161	163	161
query83	355	218	214	214
query84	299	114	101	101
query85	1284	362	350	350
query86	465	335	317	317
query87	4369	4234	4272	4234
query88	4936	2455	2448	2448
query89	552	342	328	328
query90	2118	229	229	229
query91	158	129	129	129
query92	93	74	77	74
query93	6437	557	554	554
query94	982	324	325	324
query95	389	298	295	295
query96	616	285	278	278
query97	3279	3093	3120	3093
query98	249	237	233	233
query99	1769	1293	1328	1293
Total cold run time: 320210 ms
Total hot run time: 191408 ms

doris-robot · 2024-08-21T10:24:05Z

ClickBench: Total hot run time: 30.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3ea46edd5ef97dd6f0fe853a26c8fcd27db899b5, data reload: false
query1	0.05	0.04	0.04
query2	0.09	0.05	0.04
query3	0.23	0.05	0.05
query4	1.65	0.08	0.08
query5	0.49	0.50	0.51
query6	1.12	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.06
query9	0.55	0.49	0.49
query10	0.55	0.55	0.54
query11	0.15	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.59	0.59
query14	0.78	0.80	0.79
query15	0.85	0.83	0.83
query16	0.36	0.38	0.38
query17	1.00	0.97	0.98
query18	0.23	0.21	0.21
query19	1.91	1.83	1.69
query20	0.02	0.01	0.02
query21	15.41	0.67	0.65
query22	4.36	8.48	1.38
query23	18.31	1.36	1.26
query24	2.07	0.24	0.22
query25	0.15	0.09	0.09
query26	0.28	0.19	0.19
query27	0.08	0.08	0.08
query28	13.28	1.03	1.02
query29	12.64	3.34	3.34
query30	0.43	0.24	0.24
query31	2.80	0.41	0.41
query32	3.23	0.50	0.48
query33	2.94	2.98	2.95
query34	17.12	4.38	4.39
query35	4.42	4.41	4.49
query36	0.68	0.52	0.49
query37	0.21	0.18	0.17
query38	0.18	0.17	0.17
query39	0.08	0.06	0.06
query40	0.19	0.15	0.15
query41	0.12	0.08	0.07
query42	0.07	0.06	0.06
query43	0.07	0.06	0.06
Total cold run time: 110 s
Total hot run time: 30.59 s

Hastyshell · 2024-08-22T09:12:09Z

run external

github-actions · 2024-08-22T09:12:26Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-22T09:12:28Z

PR approved by anyone and no changes requested.

Hastyshell · 2024-08-23T09:29:47Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-08-23T09:37:40Z

 }

-Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params) {
+Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params,


warning: function '_convert_historical_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params, ^
Additional context
be/src/cloud/cloud_schema_change_job.cpp:222: 177 lines including whitespace and comments (threshold 80)
Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params, ^

github-actions · 2024-08-23T09:46:24Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-08-23T10:39:22Z

TPC-H: Total hot run time: 38256 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee1f4ad41944065b8e614b038fa1aa04068dca05, data reload: false
------ Round 1 ----------------------------------
q1	18114	4698	4394	4394
q2	2633	186	189	186
q3	11131	1244	1146	1146
q4	10485	774	790	774
q5	8098	2951	2858	2858
q6	225	137	140	137
q7	988	623	619	619
q8	9534	2086	2098	2086
q9	7348	6551	6557	6551
q10	7013	2159	2222	2159
q11	516	241	253	241
q12	391	221	220	220
q13	17787	3060	3047	3047
q14	279	239	231	231
q15	524	491	485	485
q16	499	398	388	388
q17	1002	641	718	641
q18	7510	6800	6930	6800
q19	1401	1055	1083	1055
q20	699	350	342	342
q21	3940	3034	2894	2894
q22	1107	1017	1002	1002
Total cold run time: 111224 ms
Total hot run time: 38256 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4342	4306	4272	4272
q2	380	297	280	280
q3	2935	2720	2715	2715
q4	1889	1630	1662	1630
q5	5435	5446	5428	5428
q6	218	128	129	128
q7	2122	1770	1732	1732
q8	3180	3355	3395	3355
q9	8471	8420	8411	8411
q10	3456	3246	3170	3170
q11	602	511	507	507
q12	803	589	601	589
q13	12545	3047	3046	3046
q14	291	271	290	271
q15	519	492	514	492
q16	491	439	425	425
q17	1784	1499	1479	1479
q18	7834	7495	7416	7416
q19	1667	1587	1518	1518
q20	2086	1803	1824	1803
q21	5602	5297	5218	5218
q22	1118	1049	1066	1049
Total cold run time: 67770 ms
Total hot run time: 54934 ms

doris-robot · 2024-08-23T10:51:01Z

TPC-DS: Total hot run time: 186022 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee1f4ad41944065b8e614b038fa1aa04068dca05, data reload: false
query1	919	375	367	367
query2	6478	1943	1942	1942
query3	6647	211	219	211
query4	31544	23175	23125	23125
query5	4165	505	493	493
query6	266	164	182	164
query7	4582	304	292	292
query8	264	216	208	208
query9	8415	2500	2476	2476
query10	435	251	275	251
query11	17071	14978	15137	14978
query12	146	99	100	99
query13	1627	356	368	356
query14	9511	6589	6405	6405
query15	221	173	175	173
query16	7801	483	506	483
query17	1608	575	571	571
query18	1888	295	296	295
query19	200	152	149	149
query20	119	111	115	111
query21	213	103	104	103
query22	4501	4107	4042	4042
query23	34101	33516	33382	33382
query24	11230	2907	2864	2864
query25	637	372	393	372
query26	1413	178	157	157
query27	2936	273	270	270
query28	7517	2080	2069	2069
query29	898	403	390	390
query30	301	158	153	153
query31	984	783	764	764
query32	100	54	57	54
query33	748	277	272	272
query34	1006	462	468	462
query35	850	732	701	701
query36	1080	938	946	938
query37	155	90	90	90
query38	3957	3796	3880	3796
query39	1425	1388	1391	1388
query40	278	119	116	116
query41	49	45	46	45
query42	112	97	96	96
query43	492	464	462	462
query44	1216	745	758	745
query45	195	168	169	168
query46	1108	713	744	713
query47	1870	1781	1824	1781
query48	357	295	296	295
query49	1094	427	425	425
query50	804	418	407	407
query51	7249	7049	7163	7049
query52	99	90	86	86
query53	256	187	182	182
query54	981	469	460	460
query55	107	74	80	74
query56	289	254	248	248
query57	1187	1060	1054	1054
query58	234	217	236	217
query59	2956	2782	2908	2782
query60	299	274	264	264
query61	101	100	104	100
query62	832	654	648	648
query63	215	187	186	186
query64	6431	2279	1741	1741
query65	3227	3159	3124	3124
query66	1352	333	335	333
query67	15668	15260	15301	15260
query68	4517	557	565	557
query69	417	275	283	275
query70	1198	1144	1108	1108
query71	330	271	277	271
query72	6539	2233	2021	2021
query73	762	315	318	315
query74	9202	8853	8710	8710
query75	3392	2703	2684	2684
query76	2519	1090	1037	1037
query77	473	314	307	307
query78	9659	9065	9119	9065
query79	1032	540	539	539
query80	739	499	518	499
query81	440	230	225	225
query82	226	137	144	137
query83	178	148	174	148
query84	225	82	76	76
query85	716	288	292	288
query86	307	291	300	291
query87	4387	4387	4262	4262
query88	3075	2297	2294	2294
query89	382	292	288	288
query90	1925	198	194	194
query91	173	101	103	101
query92	60	49	52	49
query93	1024	545	545	545
query94	849	301	269	269
query95	352	265	258	258
query96	590	263	270	263
query97	3214	3127	3045	3045
query98	233	200	202	200
query99	1504	1276	1224	1224
Total cold run time: 287656 ms
Total hot run time: 186022 ms

doris-robot · 2024-08-23T10:56:28Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee1f4ad41944065b8e614b038fa1aa04068dca05, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.06
query5	0.49	0.48	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.06	0.04	0.05
query9	0.54	0.47	0.49
query10	0.55	0.53	0.53
query11	0.15	0.12	0.12
query12	0.15	0.13	0.12
query13	0.63	0.58	0.58
query14	0.76	0.79	0.78
query15	0.85	0.82	0.82
query16	0.36	0.38	0.38
query17	1.06	0.98	0.97
query18	0.22	0.20	0.20
query19	1.93	1.91	1.76
query20	0.02	0.01	0.01
query21	15.40	0.66	0.67
query22	4.30	7.34	1.75
query23	18.27	1.49	1.31
query24	2.09	0.22	0.23
query25	0.16	0.09	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.08
query28	13.23	1.01	1.01
query29	12.60	3.33	3.32
query30	0.24	0.05	0.05
query31	2.89	0.40	0.39
query32	3.25	0.48	0.48
query33	2.92	2.99	3.02
query34	16.94	4.38	4.43
query35	4.46	4.41	4.37
query36	0.67	0.48	0.49
query37	0.20	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.49 s
Total hot run time: 30.55 s

@Lchangliang

…mpaction during schema change in cloud mode (#39558) In cloud mode, when do schema change, shadow tablet encounters -235 because it cant do cumulative compaction in the case of a large number of loads. And it will prevents the user from continuing to loads. Implementation details: 1. When start schema change, record the end convert rowset version `alter_version` into SchemaChangeJob. 2. For origin tablet, only can do base compaction in [0, `alter_version`] and do cumulative compaction in (`alter_version`, N]. can not do compaction across `alter_verison` such as compaction [a, `alter_version` + n]. 3. For shadow tablet, cannot do base compaction and and do cumulative compaction in (`alter_version`, N]. 4. When the schema change failed because FE or BE coredump, it will retry. When retry the schema change, it will get the `alter_version` from meta_serive, and continue to do it. 5. When finish the schema change job or cancel it, we need to clear the schema change job. Before this pr, it will cover by next schema change. co-author(main author): @Lchangliang original PR: #37293 --------- Co-authored-by: Lightman <31928846+Lchangliang@users.noreply.github.com>

…n=true` (#48399) ### What problem does this PR solve? #39558 add a config to support shadow tablet to do cumulative compaction during schema change in cloud mode to avoid -235 error on new tablet in the case of a large number of loads. However, this introduces correctness problem on merge-on-write table because some rowsets' delete bitmaps are wrong if there are cumu compactions on new tablet when SC calculate delete bitmaps for incremental rowsets after converting historical data. This PR introduce a new type of compaction `STOP_TOKEN` to fail all existing compaction jobs and disallow doing any compaction on tablet and change the SC process as following: 1. converting historical data 2. register stop token on new tablet to fail all existing compaction jobs and disallow doing any compaction on new tablet 3. calculate delete bitmap for incremental rowsets without lock 4. calculate delete bitmap for incremental rowsets with lock 5. commit SC job and remove stop token ---- ref: #29386

…n=true` (apache#48399) apache#39558 add a config to support shadow tablet to do cumulative compaction during schema change in cloud mode to avoid -235 error on new tablet in the case of a large number of loads. However, this introduces correctness problem on merge-on-write table because some rowsets' delete bitmaps are wrong if there are cumu compactions on new tablet when SC calculate delete bitmaps for incremental rowsets after converting historical data. This PR introduce a new type of compaction `STOP_TOKEN` to fail all existing compaction jobs and disallow doing any compaction on tablet and change the SC process as following: 1. converting historical data 2. register stop token on new tablet to fail all existing compaction jobs and disallow doing any compaction on new tablet 3. calculate delete bitmap for incremental rowsets without lock 4. calculate delete bitmap for incremental rowsets with lock 5. commit SC job and remove stop token ---- ref: apache#29386

…n=true` (apache#48399) ### What problem does this PR solve? apache#39558 add a config to support shadow tablet to do cumulative compaction during schema change in cloud mode to avoid -235 error on new tablet in the case of a large number of loads. However, this introduces correctness problem on merge-on-write table because some rowsets' delete bitmaps are wrong if there are cumu compactions on new tablet when SC calculate delete bitmaps for incremental rowsets after converting historical data. This PR introduce a new type of compaction `STOP_TOKEN` to fail all existing compaction jobs and disallow doing any compaction on tablet and change the SC process as following: 1. converting historical data 2. register stop token on new tablet to fail all existing compaction jobs and disallow doing any compaction on new tablet 3. calculate delete bitmap for incremental rowsets without lock 4. calculate delete bitmap for incremental rowsets with lock 5. commit SC job and remove stop token ---- ref: apache#29386

@Lchangliang

…mpaction during schema change in cloud mode (apache#39558) ## Proposed changes In cloud mode, when do schema change, shadow tablet encounters -235 because it cant do cumulative compaction in the case of a large number of loads. And it will prevents the user from continuing to loads. Implementation details: 1. When start schema change, record the end convert rowset version `alter_version` into SchemaChangeJob. 2. For origin tablet, only can do base compaction in [0, `alter_version`] and do cumulative compaction in (`alter_version`, N]. can not do compaction across `alter_verison` such as compaction [a, `alter_version` + n]. 3. For shadow tablet, cannot do base compaction and and do cumulative compaction in (`alter_version`, N]. 4. When the schema change failed because FE or BE coredump, it will retry. When retry the schema change, it will get the `alter_version` from meta_serive, and continue to do it. 5. When finish the schema change job or cancel it, we need to clear the schema change job. Before this pr, it will cover by next schema change. co-author(main author): @Lchangliang original PR: apache#37293 --------- Co-authored-by: Lightman <31928846+Lchangliang@users.noreply.github.com>

Hastyshell marked this pull request as draft August 19, 2024 08:04

Hastyshell force-pushed the shadow-tablet-compaction branch from ff30024 to a909702Compare August 19, 2024 10:27

Hastyshell changed the title ~~[DNM](cloud-schema-change) Support shadow tablet to do cumulative compaction in cloud mode (#37293)~~[DNM](cloud-compaction) Support shadow tablet to do cumulative compaction in cloud mode (#37293)Aug 20, 2024

Hastyshell commented Aug 20, 2024
View reviewed changes

github-actionsBot reviewed Aug 20, 2024
View reviewed changes

Comment threadbe/src/cloud/cloud_schema_change_job.cpp
Comment threadcloud/src/meta-service/meta_service_job.cpp Outdated
Comment threadcloud/test/meta_service_job_test.cpp

Hastyshell changed the title ~~[DNM](cloud-compaction) Support shadow tablet to do cumulative compaction in cloud mode (#37293)~~[Feature](cloud-compaction) Support shadow tablet to do cumulative compaction in cloud mode (#37293)Aug 21, 2024

Hastyshell changed the title ~~[Feature](cloud-compaction) Support shadow tablet to do cumulative compaction in cloud mode (#37293)~~[feature](cloud-compaction) Support shadow tablet to do cumulative compaction during schema change in cloud mode (#37293)Aug 21, 2024

tmp fix select replace test
3ea46ed

Hastyshell force-pushed the shadow-tablet-compaction branch from 65244ae to 3ea46edCompare August 21, 2024 08:53

Hastyshell changed the title ~~[feature](cloud-compaction) Support shadow tablet to do cumulative compaction during schema change in cloud mode (#37293)~~[feature](cloud-compaction) Support shadow tablet to do cumulative compaction during schema change in cloud modeAug 21, 2024

Hastyshell marked this pull request as ready for review August 21, 2024 13:48

Hastyshell added the area/compact Issues or PRs related to the compact label Aug 21, 2024

gavinchou previously approved these changes Aug 22, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024

github-actionsBot added the reviewed label Aug 22, 2024

Merge branch 'master' into shadow-tablet-compaction
ee1f4ad

Hastyshell dismissed gavinchou’s stale review via ee1f4adAugust 23, 2024 09:29

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 23, 2024

github-actionsBot reviewed Aug 23, 2024
View reviewed changes

gavinchou approved these changes Aug 23, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 23, 2024

luwei16 approved these changes Aug 24, 2024
View reviewed changes

gavinchou merged commit ac5fdec into apache:masterAug 27, 2024

dataroaring added the dev/3.0.x-conflict label Sep 3, 2024

Hastyshell added dev/3.0.2-merged and removed dev/3.0.x dev/3.0.x-conflict labels Sep 3, 2024

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

bobhan1 mentioned this pull request Feb 28, 2025
[Fix](cloud) Fix dup key problem when enable_new_tablet_do_compaction=true#48399
Merged
16 tasks

		<< compaction.input_versions(0)
		<< " input_version_end=" << compaction.input_versions(1)

Uh oh!

Conversation

Hastyshell commented Aug 19, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Aug 19, 2024

Uh oh!

Hastyshell commented Aug 19, 2024

Uh oh!

Hastyshell commented Aug 19, 2024

Uh oh!

doris-robot commented Aug 19, 2024

Uh oh!

doris-robot commented Aug 19, 2024

Uh oh!

doris-robot commented Aug 19, 2024

Uh oh!

Hastyshell commented Aug 20, 2024

Uh oh!

Hastyshell commented Aug 20, 2024

Uh oh!

Hastyshell commented Aug 20, 2024

Uh oh!

HastyshellAug 20, 2024

Choose a reason for hiding this comment

Uh oh!

Hastyshell commented Aug 20, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Hastyshell commented Aug 20, 2024

Uh oh!

doris-robot commented Aug 20, 2024

Uh oh!

doris-robot commented Aug 20, 2024

Uh oh!

Hastyshell commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

doris-robot commented Aug 21, 2024

Uh oh!

Hastyshell commented Aug 22, 2024

Uh oh!

github-actionsBot commented Aug 22, 2024

Uh oh!

github-actionsBot commented Aug 22, 2024

Uh oh!

Hastyshell commented Aug 23, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotAug 23, 2024

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 23, 2024

Uh oh!

doris-robot commented Aug 23, 2024

Uh oh!

doris-robot commented Aug 23, 2024

Uh oh!

doris-robot commented Aug 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Hastyshell commented Aug 19, 2024•
edited
Loading