[fix](rpc) fix transfer large data and enable transfer_large_data_by_brpc by default by liaoxin01 · Pull Request #35770 · apache/doris

liaoxin01 · 2024-06-02T15:02:51Z

Proposed changes

Issue Number: close #xxx

when send large data by rpc，load will report the following error:

add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000.

The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure.

The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

doris-robot · 2024-06-02T15:02:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

…brpc by default

github-actions · 2024-06-02T15:09:06Z

clang-tidy review says "All clean, LGTM! 👍"

liaoxin01 · 2024-06-02T15:09:15Z

run buildall

github-actions · 2024-06-02T15:09:38Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-06-02T15:11:29Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-02T15:38:55Z

TeamCity be ut coverage result:
Function Coverage: 36.27% (9235/25460)
Line Coverage: 27.61% (75717/274205)
Region Coverage: 26.84% (39221/146151)
Branch Coverage: 23.55% (19866/84346)
Coverage Report: http://coverage.selectdb-in.cc/coverage/04496888e6dcc4fd26bb15954439312c2c14ec93_04496888e6dcc4fd26bb15954439312c2c14ec93/report/index.html

dataroaring

LGTM

dataroaring · 2024-06-03T01:31:29Z

run performance

github-actions · 2024-06-03T01:31:42Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-03T01:31:44Z

PR approved by anyone and no changes requested.

dataroaring · 2024-06-03T02:03:19Z

run performance

doris-robot · 2024-06-03T02:35:54Z

TPC-H: Total hot run time: 42062 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 04496888e6dcc4fd26bb15954439312c2c14ec93, data reload: false
------ Round 1 ----------------------------------
q1	17600	4365	4273	4273
q2	2029	195	195	195
q3	10460	1212	1177	1177
q4	10204	911	764	764
q5	7475	2674	2766	2674
q6	233	138	141	138
q7	985	643	628	628
q8	9373	2145	2124	2124
q9	9399	6694	6809	6694
q10	9245	3965	3952	3952
q11	473	249	247	247
q12	488	226	245	226
q13	17825	3272	3199	3199
q14	276	222	225	222
q15	530	481	487	481
q16	528	421	400	400
q17	1211	591	657	591
q18	8466	7901	8161	7901
q19	8485	1389	1302	1302
q20	693	344	371	344
q21	5267	4179	4491	4179
q22	417	352	351	351
Total cold run time: 121662 ms
Total hot run time: 42062 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4696	4576	4647	4576
q2	393	290	281	281
q3	3238	3045	2925	2925
q4	2029	1639	1683	1639
q5	5403	5597	5545	5545
q6	219	129	131	129
q7	2176	1844	1770	1770
q8	3252	3373	3379	3373
q9	8704	8649	8611	8611
q10	4086	3842	3906	3842
q11	588	512	496	496
q12	792	601	590	590
q13	15846	3063	3147	3063
q14	312	271	271	271
q15	523	505	492	492
q16	494	432	435	432
q17	1792	1494	1535	1494
q18	8129	7904	7366	7366
q19	1752	1543	1615	1543
q20	3029	1785	1807	1785
q21	4851	4839	4667	4667
q22	626	512	537	512
Total cold run time: 72930 ms
Total hot run time: 55402 ms

doris-robot · 2024-06-03T02:46:46Z

TPC-DS: Total hot run time: 170839 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 04496888e6dcc4fd26bb15954439312c2c14ec93, data reload: false
query1	927	386	381	381
query2	6463	2405	2183	2183
query3	6643	205	203	203
query4	19356	17559	17514	17514
query5	4174	460	466	460
query6	249	162	151	151
query7	4545	305	293	293
query8	305	287	279	279
query9	8462	2361	2361	2361
query10	432	300	281	281
query11	10510	10053	10027	10027
query12	145	92	90	90
query13	1643	378	377	377
query14	10257	7414	7660	7414
query15	231	193	184	184
query16	7816	272	271	271
query17	1723	551	546	546
query18	1925	284	284	284
query19	208	166	164	164
query20	97	91	87	87
query21	215	132	136	132
query22	4408	4000	4017	4000
query23	33902	33218	33048	33048
query24	10957	2868	2839	2839
query25	610	360	361	360
query26	1162	154	157	154
query27	2299	326	325	325
query28	7122	2067	2072	2067
query29	880	644	605	605
query30	268	148	151	148
query31	937	723	747	723
query32	99	52	54	52
query33	761	286	273	273
query34	966	490	472	472
query35	737	616	619	616
query36	1106	927	936	927
query37	144	67	76	67
query38	2931	2767	2742	2742
query39	907	802	799	799
query40	215	122	130	122
query41	52	50	54	50
query42	116	100	96	96
query43	576	553	518	518
query44	1220	748	755	748
query45	193	163	166	163
query46	1095	712	737	712
query47	1879	1773	1770	1770
query48	385	308	301	301
query49	997	421	428	421
query50	791	398	396	396
query51	6914	6854	6740	6740
query52	106	90	93	90
query53	357	293	304	293
query54	884	456	445	445
query55	74	76	74	74
query56	281	276	262	262
query57	1121	1060	1065	1060
query58	254	243	256	243
query59	3348	3188	3116	3116
query60	302	279	279	279
query61	93	87	88	87
query62	656	443	464	443
query63	327	305	302	302
query64	8834	2243	1780	1780
query65	3166	3113	3124	3113
query66	803	333	327	327
query67	15527	14945	15048	14945
query68	4575	568	549	549
query69	529	447	356	356
query70	1185	1110	1169	1110
query71	427	284	284	284
query72	7200	2726	2597	2597
query73	773	322	327	322
query74	5953	5529	5582	5529
query75	3470	2652	2756	2652
query76	2885	902	939	902
query77	484	301	302	301
query78	10272	10109	9772	9772
query79	2348	518	516	516
query80	2347	473	461	461
query81	586	227	220	220
query82	1347	106	102	102
query83	311	175	167	167
query84	278	93	85	85
query85	1585	270	257	257
query86	518	314	304	304
query87	3314	3078	3114	3078
query88	3643	2444	2447	2444
query89	484	391	388	388
query90	1811	254	188	188
query91	126	97	96	96
query92	64	50	48	48
query93	2470	521	512	512
query94	1272	190	190	190
query95	400	313	305	305
query96	608	270	283	270
query97	3206	2996	3028	2996
query98	256	227	218	218
query99	1368	843	825	825
Total cold run time: 273183 ms
Total hot run time: 170839 ms

doris-robot · 2024-06-03T02:52:18Z

ClickBench: Total hot run time: 30.61 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 04496888e6dcc4fd26bb15954439312c2c14ec93, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.06	0.07
query5	0.50	0.49	0.48
query6	1.13	0.74	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.04
query9	0.52	0.47	0.48
query10	0.54	0.56	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.78	0.78
query15	0.83	0.82	0.80
query16	0.35	0.37	0.34
query17	0.95	0.97	0.96
query18	0.22	0.27	0.22
query19	1.76	1.75	1.68
query20	0.01	0.02	0.02
query21	15.49	0.69	0.67
query22	4.44	7.41	2.01
query23	18.33	1.44	1.20
query24	1.92	0.22	0.20
query25	0.15	0.10	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.34	1.01	1.00
query29	13.24	3.34	3.29
query30	0.24	0.05	0.05
query31	2.89	0.38	0.38
query32	3.29	0.46	0.46
query33	2.84	2.94	2.91
query34	17.07	4.44	4.47
query35	4.52	4.55	4.68
query36	0.64	0.47	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.13	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 110.06 s
Total hot run time: 30.61 s

…brpc by default (#35770) ## Proposed changes Issue Number: close #xxx when send large data by rpc，load will report the following error: ``` add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000. ``` The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure. The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

…brpc by default (apache#35770) ## Proposed changes Issue Number: close #xxx when send large data by rpc，load will report the following error: ``` add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000. ``` The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure. The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

…brpc by default (apache#35770) Issue Number: close #xxx when send large data by rpc，load will report the following error: ``` add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000. ``` The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure. The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

…brpc by default (apache#35770) ## Proposed changes Issue Number: close #xxx when send large data by rpc，load will report the following error: ``` add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000. ``` The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure. The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

…brpc by default #35770 (#36169) cherry pick from #35770

…brpc by default #35770 (#36167) cherry pick from #35770

…brpc by default apache#35770 (apache#36167) cherry pick from apache#35770

…brpc by default (apache#35770) ## Proposed changes Issue Number: close #xxx when send large data by rpc，load will report the following error: ``` add batch req success but status isn't ok, err: [INTERNAL_ERROR]PStatus: (172.200.0.1)[INTERNAL_ERROR]fail to add batch in load channel. unknown load_id=0000000000000000-0000000000000000. ``` The maximum length of protobuf is 2GB. When it exceeds 2GB, we will use the brpc HTTP method. However, the block data was not removed during encoding, resulting in protobuf exceeding the length and encoding failure. The config of transfer_large_data_by_brpc was enabled by default on branch-2.0, so we also enable it on master.

liaoxin01 force-pushed the fix_brpc branch from 14e1e01 to 8ff5e6cCompare June 2, 2024 15:04

[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
0449688
…brpc by default

liaoxin01 force-pushed the fix_brpc branch from 8ff5e6c to 0449688Compare June 2, 2024 15:05

dataroaring approved these changes Jun 3, 2024
View reviewed changes

dataroaring added the dev/2.1.x label Jun 3, 2024

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jun 3, 2024

github-actionsBot added the reviewed label Jun 3, 2024

dataroaring added the dev/2.0.x label Jun 3, 2024

yiguolei approved these changes Jun 3, 2024
View reviewed changes

dataroaring merged commit c965161 into apache:masterJun 3, 2024

liaoxin01 mentioned this pull request Jun 12, 2024
[fix](rpc) fix transfer large data and enable transfer_large_data_by_brpc by default #35770 #36167
Merged

liaoxin01 mentioned this pull request Jun 12, 2024
[fix](rpc) fix transfer large data and enable transfer_large_data_by_brpc by default #35770 #36169
Merged

dataroaring pushed a commit that referenced this pull request Jun 12, 2024
[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
d1eb917
…brpc by default #35770 (#36169) cherry pick from #35770

dataroaring added dev/2.1.4-merged dev/2.0.12-merged and removed dev/2.0.x labels Jun 12, 2024

dataroaring pushed a commit that referenced this pull request Jun 12, 2024
[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
4228320
…brpc by default #35770 (#36167) cherry pick from #35770

liaoxin01 removed the dev/2.1.x label Jun 14, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[fix](rpc) fix transfer large data and enable transfer_large_data_by_…
9c388e0
…brpc by default apache#35770 (apache#36167) cherry pick from apache#35770

Uh oh!

Conversation

liaoxin01 commented Jun 2, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Jun 2, 2024

Uh oh!

github-actionsBot commented Jun 2, 2024

Uh oh!

liaoxin01 commented Jun 2, 2024

Uh oh!

github-actionsBot commented Jun 2, 2024

Uh oh!

github-actionsBot commented Jun 2, 2024

Uh oh!

doris-robot commented Jun 2, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

dataroaring commented Jun 3, 2024

Uh oh!

github-actionsBot commented Jun 3, 2024

Uh oh!

github-actionsBot commented Jun 3, 2024

Uh oh!

dataroaring commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

doris-robot commented Jun 3, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

liaoxin01 commented Jun 2, 2024•
edited
Loading