Skip to content

[fix](stream-load) catch exception when parsing CONTENT_LENGTH - #40796

Merged
dataroaring merged 2 commits into
apache:masterfrom
kaijchen:DORIS-15037
Sep 15, 2024
Merged

[fix](stream-load) catch exception when parsing CONTENT_LENGTH#40796
dataroaring merged 2 commits into
apache:masterfrom
kaijchen:DORIS-15037

Conversation

@kaijchen

@kaijchenkaijchen commented Sep 13, 2024

Copy link
Copy Markdown
Member

Proposed changes

Catch exception when parsing numbers from HTTP header CONTENT_LENGTH.

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen

kaijchen commented Sep 13, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 43106 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6af3b268af73aeec4a7b642eb03b28173d7fdd0b, data reload: false
------ Round 1 ----------------------------------
q1	17618	7514	7236	7236
q2	2041	187	179	179
q3	10555	1263	1408	1263
q4	10221	1019	1016	1016
q5	7728	3205	3181	3181
q6	240	156	155	155
q7	1057	635	624	624
q8	9465	2018	2056	2018
q9	6818	6333	6297	6297
q10	7017	2529	2513	2513
q11	434	247	255	247
q12	412	228	228	228
q13	17765	3060	3046	3046
q14	279	259	250	250
q15	587	529	534	529
q16	520	418	426	418
q17	992	954	974	954
q18	7570	6901	6845	6845
q19	1378	1246	1236	1236
q20	630	347	339	339
q21	3967	3568	3555	3555
q22	1112	977	1036	977
Total cold run time: 108406 ms
Total hot run time: 43106 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7242	7259	7206	7206
q2	363	237	240	237
q3	3107	3131	3053	3053
q4	2117	2100	2079	2079
q5	5708	5599	5786	5599
q6	237	153	155	153
q7	2168	1785	1791	1785
q8	3372	3472	3426	3426
q9	8854	8927	8880	8880
q10	3486	3587	3577	3577
q11	608	487	513	487
q12	816	576	631	576
q13	8747	3226	3198	3198
q14	323	281	272	272
q15	584	538	544	538
q16	514	479	470	470
q17	1812	1783	1797	1783
q18	8663	8119	8175	8119
q19	1780	1757	1773	1757
q20	2150	1905	1906	1905
q21	6130	5606	5665	5606
q22	1126	1059	1040	1040
Total cold run time: 69907 ms
Total hot run time: 61746 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 199639 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6af3b268af73aeec4a7b642eb03b28173d7fdd0b, data reload: false
query1	1257	894	849	849
query2	6394	1907	1841	1841
query3	10670	3943	4048	3943
query4	55387	26039	23941	23941
query5	4991	555	539	539
query6	315	183	169	169
query7	5585	311	301	301
query8	312	227	226	226
query9	6039	2583	2585	2583
query10	413	293	283	283
query11	16050	15495	15511	15495
query12	168	103	98	98
query13	1457	394	380	380
query14	9730	7225	6925	6925
query15	210	183	187	183
query16	7047	509	480	480
query17	1203	632	598	598
query18	1936	321	319	319
query19	220	158	154	154
query20	124	123	117	117
query21	207	106	107	106
query22	4836	4681	4434	4434
query23	35021	33858	33558	33558
query24	6145	3126	3121	3121
query25	501	381	388	381
query26	614	154	154	154
query27	1575	275	289	275
query28	2841	2104	2082	2082
query29	671	400	404	400
query30	223	152	148	148
query31	965	776	789	776
query32	73	51	53	51
query33	424	295	287	287
query34	895	476	472	472
query35	856	746	732	732
query36	1013	900	920	900
query37	140	83	79	79
query38	4002	3928	3918	3918
query39	1517	1440	1420	1420
query40	211	111	112	111
query41	48	47	44	44
query42	118	98	95	95
query43	527	451	466	451
query44	1191	779	787	779
query45	203	171	170	170
query46	1098	820	818	818
query47	1909	1765	1796	1765
query48	378	313	285	285
query49	729	458	431	431
query50	930	428	444	428
query51	6955	7025	6864	6864
query52	103	87	87	87
query53	256	175	179	175
query54	569	457	472	457
query55	77	75	73	73
query56	275	256	262	256
query57	1202	1075	1065	1065
query58	245	259	279	259
query59	2785	2791	2748	2748
query60	292	274	277	274
query61	102	99	99	99
query62	815	684	670	670
query63	223	186	187	186
query64	1616	674	721	674
query65	3266	3156	3171	3156
query66	673	292	315	292
query67	15879	15458	15399	15399
query68	2006	595	575	575
query69	399	279	277	277
query70	1157	1134	1071	1071
query71	338	286	276	276
query72	4721	4129	4008	4008
query73	764	324	330	324
query74	9407	9053	9024	9024
query75	3358	2729	2764	2729
query76	1339	1327	1346	1327
query77	500	324	321	321
query78	9835	9383	9335	9335
query79	936	904	874	874
query80	888	861	819	819
query81	465	271	263	263
query82	271	269	263	263
query83	193	189	190	189
query84	257	110	108	108
query85	707	408	392	392
query86	339	304	302	302
query87	4355	4380	4441	4380
query88	4203	4121	4131	4121
query89	381	379	371	371
query90	1991	320	311	311
query91	120	120	122	120
query92	78	77	76	76
query93	1053	1057	1050	1050
query94	784	384	410	384
query95	493	436	431	431
query96	493	479	481	479
query97	3160	3128	3104	3104
query98	236	242	231	231
query99	1693	1330	1275	1275
Total cold run time: 291622 ms
Total hot run time: 199639 ms

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6af3b268af73aeec4a7b642eb03b28173d7fdd0b, data reload: false
query1	0.05	0.05	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.50	0.49
query6	1.12	0.74	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.59	0.51	0.52
query10	0.57	0.61	0.56
query11	0.16	0.11	0.11
query12	0.16	0.13	0.12
query13	0.62	0.61	0.60
query14	1.48	1.47	1.49
query15	0.91	0.87	0.87
query16	0.38	0.37	0.37
query17	1.04	1.06	1.05
query18	0.16	0.16	0.16
query19	1.91	1.87	1.84
query20	0.01	0.01	0.01
query21	15.40	0.66	0.66
query22	3.83	8.15	1.10
query23	17.79	1.38	1.31
query24	2.24	0.23	0.22
query25	0.19	0.08	0.08
query26	0.29	0.17	0.17
query27	0.08	0.09	0.08
query28	13.18	1.14	1.10
query29	12.58	3.36	3.34
query30	0.24	0.06	0.06
query31	2.87	0.42	0.41
query32	3.23	0.49	0.49
query33	3.05	3.18	3.06
query34	15.46	4.31	4.32
query35	4.35	4.35	4.33
query36	0.69	0.50	0.50
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.94 s
Total hot run time: 30.97 s

@doris-robot

doris-robot commented Sep 13, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.91% (9465/25645)
Line Coverage: 28.26% (77836/275389)
Region Coverage: 27.66% (40186/145270)
Branch Coverage: 24.27% (20417/84124)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6af3b268af73aeec4a7b642eb03b28173d7fdd0b_6af3b268af73aeec4a7b642eb03b28173d7fdd0b/report/index.html

@liaoxin01

liaoxin01 commented Sep 13, 2024

Copy link
Copy Markdown
Contributor

std::stoll(req->header(HttpHeaders::CONTENT_LENGTH) in _handle_group_commit function also need to be processed.

dataroaring
dataroaring approved these changes Sep 14, 2024
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Sep 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 14, 2024
@github-actions

github-actionsBot commented Sep 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@dataroaring
dataroaring merged commit a39d2ff into apache:masterSep 15, 2024
kaijchen added a commit to kaijchen/doris that referenced this pull request Sep 19, 2024
@kaijchen
[fix](stream-load) catch exception when parsing CONTENT_LENGTH (apach…
968b932
…e#40796)
Catch exception when parsing numbers from HTTP header `CONTENT_LENGTH`.
@kaijchenkaijchen mentioned this pull request Sep 19, 2024
Merged
yiguolei pushed a commit that referenced this pull request Sep 19, 2024
@kaijchen
[fix](stream-load) catch exception when parsing CONTENT_LENGTH (#40796)…
b7ded71
… (#40988)
backport #40796
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@kaijchen@dataroaring
[fix](stream-load) catch exception when parsing CONTENT_LENGTH (#40796)
1b11a35
Catch exception when parsing numbers from HTTP header `CONTENT_LENGTH`.
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@kaijchen@HappenLee
[fix](stream-load) catch exception when parsing CONTENT_LENGTH (apach…
b9424e5
…e#40796)
Catch exception when parsing numbers from HTTP header `CONTENT_LENGTH`.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Yukang-LianYukang-LianYukang-Lian approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaijchen@doris-robot@liaoxin01@Yukang-Lian@dataroaring@yiguolei