Skip to content

[enhancement](exception) catch exception for streamload and validate column - #40092

Merged
gavinchou merged 8 commits into
apache:masterfrom
yiguolei:fixs
Aug 29, 2024
Merged

[enhancement](exception) catch exception for streamload and validate column#40092
gavinchou merged 8 commits into
apache:masterfrom
yiguolei:fixs

Conversation

@yiguolei

@yiguoleiyiguolei commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yiguolei

yiguolei commented Aug 29, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Aug 29, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/vec/sink/vtablet_block_convertor.cpp
Comment threadbe/src/vec/sink/vtablet_block_convertor.cpp
@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 38158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false
------ Round 1 ----------------------------------
q1	17620	4967	4304	4304
q2	2027	186	199	186
q3	11976	991	1094	991
q4	10509	753	717	717
q5	7771	2866	2840	2840
q6	232	140	142	140
q7	973	612	617	612
q8	9339	2064	2083	2064
q9	7306	6540	6592	6540
q10	7013	2242	2187	2187
q11	474	247	239	239
q12	398	226	236	226
q13	18728	3042	3029	3029
q14	270	251	236	236
q15	521	482	501	482
q16	590	492	521	492
q17	997	714	675	675
q18	7461	6929	6908	6908
q19	1413	1066	1067	1066
q20	673	339	331	331
q21	3987	3095	2877	2877
q22	1141	1021	1016	1016
Total cold run time: 111419 ms
Total hot run time: 38158 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4546	4308	4333	4308
q2	376	284	277	277
q3	2925	2612	2666	2612
q4	1985	1657	1754	1657
q5	5669	5699	5776	5699
q6	238	137	136	136
q7	2231	1828	1803	1803
q8	3303	3413	3519	3413
q9	8919	8859	8854	8854
q10	3632	3456	3350	3350
q11	606	510	502	502
q12	826	663	651	651
q13	14812	3201	3262	3201
q14	331	297	293	293
q15	542	494	514	494
q16	623	601	579	579
q17	1832	1536	1546	1536
q18	8031	7828	7894	7828
q19	1782	1537	1502	1502
q20	2179	1911	1910	1910
q21	5733	5513	5414	5414
q22	1138	1088	1095	1088
Total cold run time: 72259 ms
Total hot run time: 57107 ms

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192661 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false
query1	1274	878	881	878
query2	6415	1972	1889	1889
query3	10729	4173	4085	4085
query4	60127	24057	23358	23358
query5	5573	523	492	492
query6	435	164	168	164
query7	5864	305	288	288
query8	298	202	201	201
query9	8748	2500	2497	2497
query10	467	270	259	259
query11	17792	15070	15323	15070
query12	158	106	107	106
query13	1496	406	373	373
query14	11070	7377	6613	6613
query15	236	183	175	175
query16	7169	480	522	480
query17	1121	617	575	575
query18	2003	337	294	294
query19	299	155	148	148
query20	121	109	127	109
query21	203	103	101	101
query22	4762	4402	4465	4402
query23	34426	33594	33420	33420
query24	5934	2893	2853	2853
query25	527	383	380	380
query26	679	157	157	157
query27	1785	300	277	277
query28	3741	2124	2112	2112
query29	644	404	407	404
query30	237	158	159	158
query31	917	767	773	767
query32	81	54	59	54
query33	453	298	285	285
query34	859	479	498	479
query35	844	772	724	724
query36	1068	919	989	919
query37	153	96	94	94
query38	4009	3799	3898	3799
query39	1431	1398	1557	1398
query40	199	115	118	115
query41	46	48	44	44
query42	123	97	94	94
query43	526	455	474	455
query44	1098	753	756	753
query45	199	171	171	171
query46	1091	730	730	730
query47	1961	1782	1833	1782
query48	364	299	295	295
query49	747	423	441	423
query50	815	413	421	413
query51	7161	7111	7061	7061
query52	100	84	90	84
query53	251	176	181	176
query54	555	449	463	449
query55	77	75	76	75
query56	274	256	258	256
query57	1181	1112	1120	1112
query58	221	264	234	234
query59	3026	2925	2778	2778
query60	291	264	267	264
query61	101	98	94	94
query62	746	660	672	660
query63	216	188	186	186
query64	2855	679	721	679
query65	3232	3139	3171	3139
query66	679	336	329	329
query67	15409	15215	15443	15215
query68	2917	577	588	577
query69	394	287	274	274
query70	1168	1130	1062	1062
query71	332	275	272	272
query72	5067	4038	3972	3972
query73	754	334	334	334
query74	9251	8815	8867	8815
query75	3358	2659	2710	2659
query76	1373	1000	938	938
query77	547	333	313	313
query78	11235	9803	9103	9103
query79	1055	540	527	527
query80	683	516	502	502
query81	463	235	234	234
query82	242	145	145	145
query83	169	148	152	148
query84	251	79	83	79
query85	747	354	344	344
query86	305	306	281	281
query87	4437	4302	4380	4302
query88	3289	2337	2336	2336
query89	382	289	292	289
query90	1918	198	194	194
query91	138	114	112	112
query92	66	54	54	54
query93	1031	543	539	539
query94	697	321	302	302
query95	317	269	265	265
query96	596	269	270	269
query97	3205	3074	3109	3074
query98	224	217	206	206
query99	1510	1260	1237	1237
Total cold run time: 305813 ms
Total hot run time: 192661 ms

@doris-robot

doris-robot commented Aug 29, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b275b77fd707e912c0cbb7d6ff7b0d9ced36d8d7, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.05	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.49	0.50	0.49
query6	1.13	0.75	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.63	0.58	0.58
query14	2.01	2.05	2.13
query15	0.86	0.83	0.83
query16	0.37	0.37	0.37
query17	1.06	1.05	1.07
query18	0.22	0.21	0.21
query19	1.84	1.76	1.82
query20	0.01	0.01	0.01
query21	15.41	0.68	0.67
query22	3.98	7.69	1.83
query23	18.25	1.42	1.26
query24	2.06	0.24	0.21
query25	0.16	0.08	0.08
query26	0.27	0.17	0.18
query27	0.09	0.07	0.08
query28	13.30	1.02	1.00
query29	12.62	3.30	3.27
query30	0.24	0.05	0.06
query31	2.90	0.39	0.40
query32	3.25	0.47	0.47
query33	2.97	3.03	3.10
query34	17.61	4.54	4.55
query35	4.60	4.55	4.62
query36	0.66	0.48	0.48
query37	0.20	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.06	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 111.22 s
Total hot run time: 32.26 s

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 29, 2024
@github-actions

github-actionsBot commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Aug 29, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

xinyiZzz
xinyiZzz approved these changes Aug 29, 2024
View reviewed changes

@xinyiZzzxinyiZzz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou
gavinchou merged commit 6fae472 into apache:masterAug 29, 2024
yiguolei added a commit to yiguolei/incubator-doris that referenced this pull request Sep 9, 2024
@yiguolei@Doris-Extras
[enhancement](exception) catch exception for streamload and validate …
dc7a90c
…column (apache#40092)
@yiguoleiyiguolei mentioned this pull request Sep 9, 2024
Closed
@xinyiZzzxinyiZzz mentioned this pull request Sep 9, 2024
Closed
xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Sep 14, 2024
@yiguolei@xinyiZzz
[enhancement](exception) catch exception for streamload and validate …
fbb7c85
…column (apache#40092)
@xinyiZzzxinyiZzz mentioned this pull request Sep 14, 2024
Merged
yiguolei added a commit that referenced this pull request Sep 15, 2024
@xinyiZzz@sollhui@yiguolei
[branch-2.1](memory) When Load ends, check memory tracker value retur…
b52b572
…ns is equal to 0 (#40850)
pick
#38960#39908#40043#40092#40016#40439
---------
Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@yiguolei@dataroaring
[enhancement](exception) catch exception for streamload and validate …
4406309
…column (#40092)
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@yiguolei@HappenLee
[enhancement](exception) catch exception for streamload and validate …
c260517
…column (apache#40092)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@gavinchougavinchougavinchou approved these changes
@xinyiZzzxinyiZzzxinyiZzz approved these changes
@platonekoplatonekoAwaiting requested review from platoneko
@dataroaringdataroaringAwaiting requested review from dataroaring

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@yiguolei@doris-robot@gavinchou@xinyiZzz@dataroaring@Doris-Extras