Skip to content

Cherry-pick fix about restore indexes and ids - #33772

Merged
dataroaring merged 2 commits into
apache:branch-2.0from
w41ter:pick_reset_ids_for_restore
Apr 17, 2024
Merged

Cherry-pick fix about restore indexes and ids #33772
dataroaring merged 2 commits into
apache:branch-2.0from
w41ter:pick_reset_ids_for_restore

Conversation

@w41ter

@w41terw41ter commented Apr 17, 2024

Copy link
Copy Markdown
Contributor

Cherry-pick #33648, #33650

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter

w41ter commented Apr 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@w41ter
w41terforce-pushed the pick_reset_ids_for_restore branch 2 times, most recently from 67d9a27 to 81aae40CompareApril 17, 2024 08:03
@w41ter

w41ter commented Apr 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49902 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 81aae40768504c1101fc8ef94f0523fcf258eb0c, data reload: false
------ Round 1 ----------------------------------
q1	17774	4372	4343	4343
q2	2041	151	137	137
q3	10447	1870	1868	1868
q4	10310	1223	1299	1223
q5	8413	3930	3945	3930
q6	229	121	122	121
q7	2022	1578	1574	1574
q8	9532	2693	2696	2693
q9	13287	10596	10507	10507
q10	8645	3498	3493	3493
q11	411	234	241	234
q12	454	300	307	300
q13	18339	3958	4030	3958
q14	355	328	329	328
q15	500	454	452	452
q16	686	603	601	601
q17	1127	967	958	958
q18	7269	6891	6909	6891
q19	1687	1541	1555	1541
q20	533	327	318	318
q21	4419	4128	4052	4052
q22	494	380	404	380
Total cold run time: 118974 ms
Total hot run time: 49902 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4304	4306	4319	4306
q2	318	217	222	217
q3	4176	4145	4170	4145
q4	2737	2748	2740	2740
q5	7241	7199	7187	7187
q6	232	124	119	119
q7	3230	2852	2884	2852
q8	4332	4485	4500	4485
q9	17060	16985	16970	16970
q10	4215	4276	4302	4276
q11	813	691	671	671
q12	1040	870	846	846
q13	7393	3768	3747	3747
q14	450	421	415	415
q15	499	453	449	449
q16	769	705	716	705
q17	3873	3855	3880	3855
q18	8873	8928	8818	8818
q19	1718	1700	1690	1690
q20	2395	2129	2144	2129
q21	8579	8553	8500	8500
q22	1066	981	1027	981
Total cold run time: 85313 ms
Total hot run time: 80103 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 201509 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 81aae40768504c1101fc8ef94f0523fcf258eb0c, data reload: false
query1	921	391	381	381
query2	6679	2252	2038	2038
query3	8448	211	207	207
query4	26173	18131	18049	18049
query5	19742	6565	6489	6489
query6	407	210	231	210
query7	5124	297	292	292
query8	262	265	221	221
query9	3139	2713	2647	2647
query10	419	271	294	271
query11	11331	10695	10646	10646
query12	129	75	66	66
query13	5578	652	631	631
query14	19087	13280	13659	13280
query15	355	224	229	224
query16	6388	275	258	258
query17	1362	1475	860	860
query18	2293	404	411	404
query19	199	140	143	140
query20	85	75	76	75
query21	179	90	90	90
query22	5184	4929	4950	4929
query23	32583	31879	32023	31879
query24	6754	6515	6537	6515
query25	506	424	413	413
query26	521	160	159	159
query27	1792	293	295	293
query28	6058	2300	2266	2266
query29	2893	2738	2798	2738
query30	242	165	162	162
query31	913	719	747	719
query32	65	62	57	57
query33	391	248	257	248
query34	845	471	483	471
query35	1125	926	890	890
query36	1212	1161	1082	1082
query37	100	64	59	59
query38	3051	2908	2933	2908
query39	1365	1323	1318	1318
query40	199	92	99	92
query41	37	33	32	32
query42	90	78	84	78
query43	631	520	522	520
query44	1117	718	731	718
query45	234	228	225	225
query46	1235	961	986	961
query47	1707	2022	1810	1810
query48	978	677	675	675
query49	618	361	380	361
query50	864	607	594	594
query51	4775	4679	4654	4654
query52	80	90	84	84
query53	448	329	325	325
query54	2644	2463	2455	2455
query55	94	80	80	80
query56	210	198	205	198
query57	1246	1188	1103	1103
query58	213	196	190	190
query59	3705	3321	3498	3321
query60	196	192	187	187
query61	82	78	86	78
query62	844	505	462	462
query63	475	331	339	331
query64	2484	1493	1451	1451
query65	3621	3579	3540	3540
query66	769	367	357	357
query67	15939	15368	15865	15368
query68	8373	644	671	644
query69	582	348	351	348
query70	1663	1544	1533	1533
query71	410	299	300	299
query72	6388	3425	3404	3404
query73	733	321	328	321
query74	6298	5929	5818	5818
query75	4570	3621	3656	3621
query76	4668	1175	1219	1175
query77	619	248	250	248
query78	12483	11636	11444	11444
query79	7798	649	661	649
query80	1104	399	385	385
query81	497	230	232	230
query82	1631	108	99	99
query83	178	133	133	133
query84	260	68	71	68
query85	858	286	280	280
query86	325	297	304	297
query87	3240	3037	3008	3008
query88	5028	2355	2349	2349
query89	349	315	296	296
query90	1918	211	213	211
query91	163	115	123	115
query92	55	51	50	50
query93	4716	640	600	600
query94	773	200	197	197
query95	1108	1111	1061	1061
query96	644	330	326	326
query97	6430	6326	6347	6326
query98	190	179	170	170
query99	3022	1023	969	969
Total cold run time: 316509 ms
Total hot run time: 201509 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81aae40768504c1101fc8ef94f0523fcf258eb0c, data reload: false
query1	0.03	0.02	0.02
query2	0.08	0.03	0.02
query3	0.24	0.05	0.06
query4	1.79	0.08	0.06
query5	0.52	0.53	0.52
query6	1.27	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.53	0.48	0.46
query10	0.54	0.52	0.53
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.62	0.60	0.62
query14	0.77	0.78	0.78
query15	0.78	0.76	0.76
query16	0.36	0.39	0.36
query17	1.03	1.03	1.00
query18	0.22	0.25	0.25
query19	1.90	1.85	1.79
query20	0.01	0.00	0.01
query21	15.45	0.56	0.56
query22	2.04	2.28	1.43
query23	17.27	1.13	0.87
query24	6.48	0.62	1.10
query25	0.39	0.09	0.05
query26	0.69	0.16	0.15
query27	0.05	0.04	0.04
query28	6.95	0.73	0.76
query29	12.73	2.37	2.34
query30	0.57	0.52	0.51
query31	2.82	0.39	0.39
query32	3.37	0.51	0.49
query33	3.06	3.08	3.08
query34	15.29	4.79	4.79
query35	4.86	4.86	4.86
query36	1.09	1.03	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 104.47 s
Total hot run time: 30.18 s

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 81aae40768504c1101fc8ef94f0523fcf258eb0c with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 20.1 seconds inserted 10000000 Rows, about 497K ops/s

@w41ter
w41terforce-pushed the pick_reset_ids_for_restore branch from 81aae40 to 6f54105CompareApril 17, 2024 08:58
@w41ter

w41ter commented Apr 17, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 50322 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f54105a68029e94a52b16aba86f70101c14b7f2, data reload: false
------ Round 1 ----------------------------------
q1	17634	4389	4439	4389
q2	2041	149	139	139
q3	10358	1937	1946	1937
q4	10312	1285	1333	1285
q5	8466	3965	4014	3965
q6	228	123	124	123
q7	2058	1598	1620	1598
q8	9270	2748	2747	2747
q9	10849	10489	10669	10489
q10	8697	3557	3507	3507
q11	433	256	238	238
q12	480	301	305	301
q13	18346	4004	4037	4004
q14	357	340	346	340
q15	504	449	463	449
q16	720	596	588	588
q17	1135	963	960	960
q18	7308	6956	6924	6924
q19	1693	1593	1521	1521
q20	508	303	317	303
q21	4486	4154	4122	4122
q22	504	393	399	393
Total cold run time: 116387 ms
Total hot run time: 50322 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4387	4274	4281	4274
q2	326	230	224	224
q3	4143	4177	4133	4133
q4	2772	2759	2767	2759
q5	7276	7188	7159	7159
q6	235	120	117	117
q7	3264	2793	2859	2793
q8	4405	4496	4495	4495
q9	17127	16877	16989	16877
q10	4236	4281	4289	4281
q11	742	724	703	703
q12	1023	839	857	839
q13	7215	3707	3750	3707
q14	444	424	426	424
q15	499	453	449	449
q16	752	719	698	698
q17	3813	3904	3864	3864
q18	8786	8708	8880	8708
q19	1714	1729	1644	1644
q20	2400	2199	2098	2098
q21	8500	8574	8503	8503
q22	1022	941	928	928
Total cold run time: 85081 ms
Total hot run time: 79677 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 201081 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f54105a68029e94a52b16aba86f70101c14b7f2, data reload: false
query1	929	393	382	382
query2	6538	2240	2004	2004
query3	6919	203	199	199
query4	21567	18334	18078	18078
query5	19711	6521	6476	6476
query6	289	208	231	208
query7	4177	297	301	297
query8	259	227	263	227
query9	3123	2689	2633	2633
query10	417	282	300	282
query11	11518	10680	10745	10680
query12	116	74	69	69
query13	5582	629	632	629
query14	17401	13417	13650	13417
query15	392	231	235	231
query16	6438	273	261	261
query17	1583	1432	854	854
query18	2300	415	403	403
query19	210	144	147	144
query20	76	78	75	75
query21	193	95	93	93
query22	5251	5034	4775	4775
query23	32372	31894	32086	31894
query24	6989	6501	6501	6501
query25	537	426	414	414
query26	524	162	159	159
query27	1894	288	289	288
query28	6108	2253	2247	2247
query29	2895	2638	2703	2638
query30	245	158	161	158
query31	908	754	730	730
query32	68	61	58	58
query33	398	242	251	242
query34	854	463	474	463
query35	1101	926	901	901
query36	1251	1246	1463	1246
query37	87	58	59	58
query38	3081	2942	2927	2927
query39	1371	1310	1322	1310
query40	199	92	95	92
query41	35	33	32	32
query42	76	86	89	86
query43	643	619	557	557
query44	1143	728	712	712
query45	235	232	226	226
query46	1227	978	980	978
query47	1703	1861	1674	1674
query48	983	661	668	661
query49	619	371	359	359
query50	867	613	610	610
query51	4707	4650	4588	4588
query52	93	84	96	84
query53	446	315	317	315
query54	2664	2492	2472	2472
query55	83	75	80	75
query56	209	206	221	206
query57	1346	1102	1085	1085
query58	218	193	186	186
query59	3553	3365	3094	3094
query60	205	196	184	184
query61	84	80	85	80
query62	879	421	480	421
query63	466	341	338	338
query64	2526	1453	1362	1362
query65	3644	3571	3547	3547
query66	797	363	363	363
query67	15747	15543	17410	15543
query68	8350	631	643	631
query69	567	368	340	340
query70	1581	1505	1307	1307
query71	395	295	305	295
query72	6376	3449	3422	3422
query73	732	326	327	326
query74	6310	5824	5854	5824
query75	4707	3799	3766	3766
query76	4864	1192	1197	1192
query77	693	246	241	241
query78	12738	11586	11718	11586
query79	8120	631	635	631
query80	1689	405	390	390
query81	511	232	232	232
query82	1497	98	91	91
query83	168	130	132	130
query84	254	65	71	65
query85	1254	278	269	269
query86	345	296	289	289
query87	3219	2967	3000	2967
query88	5113	2359	2371	2359
query89	388	278	285	278
query90	1794	209	203	203
query91	156	129	117	117
query92	58	54	50	50
query93	4974	571	589	571
query94	816	205	201	201
query95	1113	1041	1052	1041
query96	645	336	325	325
query97	6353	6311	6432	6311
query98	182	167	163	163
query99	2857	907	887	887
Total cold run time: 309888 ms
Total hot run time: 201081 ms

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.75 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f54105a68029e94a52b16aba86f70101c14b7f2, data reload: false
query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.04
query4	1.80	0.05	0.06
query5	0.53	0.52	0.52
query6	1.24	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.56	0.48	0.49
query10	0.54	0.55	0.54
query11	0.13	0.09	0.08
query12	0.11	0.10	0.08
query13	0.62	0.62	0.61
query14	0.80	0.79	0.80
query15	0.78	0.75	0.76
query16	0.38	0.36	0.37
query17	1.04	1.00	1.01
query18	0.19	0.30	0.25
query19	1.93	1.87	1.79
query20	0.02	0.02	0.01
query21	15.45	0.55	0.56
query22	1.84	1.95	1.96
query23	17.32	1.00	0.95
query24	5.90	0.62	1.99
query25	0.36	0.13	0.05
query26	0.66	0.16	0.17
query27	0.04	0.04	0.04
query28	6.99	0.74	0.73
query29	12.77	2.37	2.35
query30	0.59	0.52	0.52
query31	2.81	0.41	0.38
query32	3.35	0.49	0.49
query33	3.05	3.06	3.05
query34	15.27	4.80	4.78
query35	4.83	4.84	4.86
query36	1.06	1.01	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.01	0.01	0.01
query40	0.17	0.14	0.13
query41	0.06	0.02	0.02
query42	0.03	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 103.72 s
Total hot run time: 30.75 s

@doris-robot

doris-robot commented Apr 17, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6f54105a68029e94a52b16aba86f70101c14b7f2 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 20.9 seconds inserted 10000000 Rows, about 478K ops/s

@dataroaring
dataroaring merged commit 4adb907 into apache:branch-2.0Apr 17, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41terweixingyu12
Cherry-pick fix about restore indexes and ids (apache#33772)
9c8e821
* [fix](restore) Reset index id for restore (apache#33648)
* [fix](restore) add indexes as part of table signature (apache#33650)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@w41ter@doris-robot@dataroaring