[chore](be) Acquire and check MD5 digest of the file to download by w41ter · Pull Request #37420 · apache/doris

w41ter · 2024-07-08T02:48:37Z

Cherry-pick #35807, #36621, #36726

Required to upgrade curl 7.79.0 -> 8.2.1

doris-robot · 2024-07-08T02:48:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

w41ter · 2024-07-08T02:49:22Z

run buildall

w41ter · 2024-07-08T03:19:50Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-08T03:26:30Z

@@ -17,8 +17,11 @@

 #include "http/http_client.h"


warning: 'http/http_client.h' file not found [clang-diagnostic-error]
#include"http/http_client.h" ^

ref #23414. It was acquired by #37420.

dataroaring · 2024-07-17T14:32:25Z

run buildall

w41ter · 2024-07-26T01:57:22Z

run buildall

doris-robot · 2024-07-26T02:30:52Z

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21468)
Line Coverage: 29.48% (66571/225841)
Region Coverage: 28.96% (34319/118501)
Branch Coverage: 24.84% (17632/70996)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a10a374f3359e45a3651783cb8ceb3ec40658ee1_a10a374f3359e45a3651783cb8ceb3ec40658ee1/report/index.html

…load Cherry-pick apache#35807

Cherry-pick apache#36621

Cherry-pick apache#36726

w41ter · 2024-07-26T02:36:54Z

run buildall

doris-robot · 2024-07-26T03:07:40Z

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21468)
Line Coverage: 29.48% (66570/225841)
Region Coverage: 28.96% (34316/118501)
Branch Coverage: 24.83% (17630/70996)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6a39e8d3409995b02e34d80a164faf2bfad9d53b_6a39e8d3409995b02e34d80a164faf2bfad9d53b/report/index.html

doris-robot · 2024-07-26T03:08:49Z

TPC-H: Total hot run time: 50435 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false
------ Round 1 ----------------------------------
q1	18959	4465	4366	4366
q2	2036	156	151	151
q3	10445	1917	1953	1917
q4	11669	1276	1340	1276
q5	8473	4006	4036	4006
q6	242	132	130	130
q7	2149	1669	1694	1669
q8	9554	2781	2765	2765
q9	11299	10864	10513	10513
q10	10178	3561	3558	3558
q11	434	254	252	252
q12	471	313	312	312
q13	20469	3971	4025	3971
q14	359	331	330	330
q15	512	456	469	456
q16	669	564	560	560
q17	1129	967	957	957
q18	7150	6837	6952	6837
q19	1771	1672	1610	1610
q20	537	324	298	298
q21	4392	4097	4073	4073
q22	529	438	428	428
Total cold run time: 123426 ms
Total hot run time: 50435 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4352	4388	4309	4309
q2	321	226	223	223
q3	4160	4125	4139	4125
q4	2751	2726	2723	2723
q5	7253	7112	7095	7095
q6	237	122	117	117
q7	3263	2855	2802	2802
q8	4341	4462	4487	4462
q9	17367	17135	17091	17091
q10	4205	4265	4224	4224
q11	756	711	659	659
q12	1021	877	861	861
q13	6825	3702	3725	3702
q14	459	426	417	417
q15	501	457	443	443
q16	724	668	687	668
q17	3775	3847	3836	3836
q18	8928	8758	8803	8758
q19	1730	1727	1676	1676
q20	2348	2118	2078	2078
q21	8496	8456	8431	8431
q22	1033	975	994	975
Total cold run time: 84846 ms
Total hot run time: 79675 ms

doris-robot · 2024-07-26T03:21:33Z

TPC-DS: Total hot run time: 203589 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false
query1	916	419	378	378
query2	6525	2719	2478	2478
query3	6922	214	202	202
query4	21012	18003	18031	18003
query5	19747	6508	6598	6508
query6	286	227	239	227
query7	4174	302	309	302
query8	438	400	476	400
query9	3123	2708	2642	2642
query10	407	297	295	295
query11	11325	10750	10813	10750
query12	133	75	75	75
query13	5607	718	697	697
query14	17462	13273	13770	13273
query15	365	234	268	234
query16	6490	284	262	262
query17	1715	1449	870	870
query18	2318	404	417	404
query19	214	152	148	148
query20	81	77	80	77
query21	191	94	94	94
query22	5257	5052	5068	5052
query23	32570	32047	31808	31808
query24	6937	6502	6485	6485
query25	519	430	419	419
query26	520	164	160	160
query27	1884	291	294	291
query28	6152	2406	2358	2358
query29	2878	2767	2899	2767
query30	250	168	165	165
query31	896	721	726	721
query32	68	63	63	63
query33	418	268	260	260
query34	856	468	487	468
query35	1103	945	935	935
query36	1191	1157	1077	1077
query37	87	62	59	59
query38	3109	2943	2896	2896
query39	1376	1330	1327	1327
query40	201	95	94	94
query41	47	43	45	43
query42	87	90	77	77
query43	735	693	587	587
query44	1117	724	721	721
query45	242	236	237	236
query46	1244	970	972	970
query47	1786	1717	1929	1717
query48	1022	702	722	702
query49	631	374	373	373
query50	865	601	612	601
query51	4910	4672	4635	4635
query52	89	88	85	85
query53	456	328	323	323
query54	2661	2427	2463	2427
query55	87	75	84	75
query56	226	215	198	198
query57	1237	1130	1108	1108
query58	211	208	196	196
query59	4095	4108	3976	3976
query60	219	205	214	205
query61	102	96	98	96
query62	810	477	453	453
query63	483	341	342	341
query64	2606	1530	1400	1400
query65	3625	3508	3613	3508
query66	778	380	387	380
query67	15835	15649	15368	15368
query68	10042	659	679	659
query69	582	334	351	334
query70	1812	1669	1372	1372
query71	415	323	319	319
query72	6563	3465	3513	3465
query73	736	323	326	323
query74	6367	5904	5901	5901
query75	5389	3737	3700	3700
query76	6445	1172	1115	1115
query77	1099	251	254	251
query78	12661	11928	11897	11897
query79	7754	647	669	647
query80	1185	398	407	398
query81	495	233	246	233
query82	1545	101	97	97
query83	167	134	132	132
query84	264	71	71	71
query85	895	335	341	335
query86	324	309	292	292
query87	3262	3052	3030	3030
query88	4543	2322	2323	2322
query89	404	287	294	287
query90	1987	210	195	195
query91	185	140	143	140
query92	59	52	54	52
query93	6096	625	566	566
query94	688	211	213	211
query95	1126	1050	1040	1040
query96	645	315	318	315
query97	6550	6402	6354	6354
query98	187	176	179	176
query99	3032	941	835	835
Total cold run time: 315433 ms
Total hot run time: 203589 ms

doris-robot · 2024-07-26T03:26:59Z

ClickBench: Total hot run time: 31.78 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false
query1	0.02	0.03	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.83	0.06	0.06
query5	0.56	0.53	0.53
query6	1.27	0.61	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.47	0.49
query10	0.54	0.54	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.08
query13	0.63	0.62	0.61
query14	0.78	0.78	0.78
query15	0.77	0.76	0.75
query16	0.37	0.37	0.38
query17	1.05	1.02	1.00
query18	0.24	0.20	0.29
query19	1.95	1.81	1.83
query20	0.01	0.02	0.02
query21	15.46	0.55	0.56
query22	2.17	2.24	1.62
query23	16.87	1.04	0.97
query24	3.43	2.29	1.98
query25	0.39	0.09	0.04
query26	0.54	0.15	0.15
query27	0.05	0.04	0.04
query28	7.44	0.71	0.75
query29	12.72	2.36	2.39
query30	0.54	0.51	0.54
query31	2.80	0.38	0.38
query32	3.37	0.49	0.49
query33	3.06	3.07	3.08
query34	15.27	4.84	4.78
query35	4.88	4.87	4.84
query36	1.06	1.02	1.02
query37	0.05	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.01
Total cold run time: 101.59 s
Total hot run time: 31.78 s

doris-robot · 2024-07-26T03:30:15Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

ref apache#23414. It was acquired by apache#37420.

…che#37420) Cherry-pick apache#35807, apache#36621, apache#36726

w41ter requested review from dataroaring, gavinchou and platoneko as code owners July 8, 2024 02:48

w41ter changed the base branch from master to branch-2.0July 8, 2024 02:49

w41ter force-pushed the pick/md5_check_to_2.0 branch from af18208 to 439d792Compare July 8, 2024 03:19

github-actionsBot reviewed Jul 8, 2024
View reviewed changes

w41ter mentioned this pull request Jul 16, 2024
[chore](third) Upgrade curl 7.79.0 -> 8.2.1 #37893
Merged

dataroaring pushed a commit that referenced this pull request Jul 17, 2024
[chore](third) Upgrade curl 7.79.0 -> 8.2.1 (#37893)
aca7ce9
ref #23414. It was acquired by #37420.

w41ter added 3 commits July 26, 2024 02:35

[improvement](http) Support to acquire md5 digest of the file to down…
f89f45a
…load Cherry-pick apache#35807

[fix](be) Check MD5 of the downloaded files before ingesting binlog
c18d347
Cherry-pick apache#36621

[fix](be) Check MD5 when downloading snapshot in http
6a39e8d
Cherry-pick apache#36726

w41ter force-pushed the pick/md5_check_to_2.0 branch from a10a374 to 6a39e8dCompare July 26, 2024 02:36

w41ter merged commit 7fc4ddc into apache:branch-2.0Jul 26, 2024

w41ter deleted the pick/md5_check_to_2.0 branch July 26, 2024 06:03

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[chore](third) Upgrade curl 7.79.0 -> 8.2.1 (apache#37893)
3a6f6a7
ref apache#23414. It was acquired by apache#37420.

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[chore](be) Acquire and check MD5 digest of the file to download (apa…
8939fe7
…che#37420) Cherry-pick apache#35807, apache#36621, apache#36726

Uh oh!

Conversation

w41ter commented Jul 8, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jul 8, 2024

Uh oh!

w41ter commented Jul 8, 2024

Uh oh!

w41ter commented Jul 8, 2024

Uh oh!

github-actionsBot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBotJul 8, 2024

Choose a reason for hiding this comment

Uh oh!

dataroaring commented Jul 17, 2024

Uh oh!

w41ter commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

w41ter commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

doris-robot commented Jul 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

w41ter commented Jul 8, 2024•
edited
Loading