[opt](arrow-flight-sql) Add config `arrow_flight_result_sink_buffer_size_rows` by xinyiZzz · Pull Request #38221 · apache/doris

xinyiZzz · 2024-07-22T13:16:02Z

support modifying arrow flight result sink buffer rows size with parameters, default 4096 * 8.

we want to return a larger batch at a time, when large amounts of data.

doris-robot · 2024-07-22T13:16:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-07-22T13:16:09Z

run buildall

github-actions · 2024-07-22T13:22:16Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-07-23T01:53:55Z

run buildall

xinyiZzz · 2024-07-23T01:55:46Z

run buildall

github-actions · 2024-07-23T01:59:58Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-23T02:21:52Z

TPC-H: Total hot run time: 39815 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 313401fec41091f9e90fe8804a08b99c5a6b0016, data reload: false
------ Round 1 ----------------------------------
q1	17623	4451	4257	4257
q2	2029	190	208	190
q3	10430	1143	1110	1110
q4	10192	801	834	801
q5	7549	2699	2660	2660
q6	224	141	135	135
q7	960	593	601	593
q8	9223	2047	2055	2047
q9	8679	6530	6543	6530
q10	8721	3769	3817	3769
q11	445	232	241	232
q12	396	225	225	225
q13	17756	3014	3004	3004
q14	290	245	240	240
q15	530	479	499	479
q16	483	379	375	375
q17	958	685	746	685
q18	8192	7524	7352	7352
q19	8781	1425	1362	1362
q20	716	324	328	324
q21	4897	3830	3167	3167
q22	344	278	279	278
Total cold run time: 119418 ms
Total hot run time: 39815 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4333	4251	4281	4251
q2	367	269	271	269
q3	3039	2798	2725	2725
q4	1864	1614	1632	1614
q5	5262	5320	5346	5320
q6	220	129	128	128
q7	2107	1763	1723	1723
q8	3166	3342	3289	3289
q9	8456	8411	8368	8368
q10	3885	3685	3676	3676
q11	583	489	492	489
q12	759	595	581	581
q13	17598	2986	2958	2958
q14	302	271	276	271
q15	543	471	478	471
q16	477	416	410	410
q17	1809	1494	1460	1460
q18	7669	7581	7406	7406
q19	1646	1557	1612	1557
q20	2010	1785	1798	1785
q21	4921	4704	4785	4704
q22	583	507	467	467
Total cold run time: 71599 ms
Total hot run time: 53922 ms

doris-robot · 2024-07-23T02:33:00Z

TPC-DS: Total hot run time: 171271 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 313401fec41091f9e90fe8804a08b99c5a6b0016, data reload: false
query1	901	371	369	369
query2	6448	1947	1781	1781
query3	6674	215	227	215
query4	27717	17373	17187	17187
query5	4170	485	490	485
query6	289	164	180	164
query7	4585	280	280	280
query8	243	191	189	189
query9	8547	2410	2379	2379
query10	428	277	270	270
query11	10517	9968	10127	9968
query12	133	83	82	82
query13	1623	361	351	351
query14	10157	7573	6872	6872
query15	218	166	162	162
query16	7772	482	496	482
query17	1393	547	510	510
query18	1930	270	271	270
query19	192	141	146	141
query20	86	81	79	79
query21	202	129	122	122
query22	4174	3973	4138	3973
query23	33685	33022	33056	33022
query24	11992	2838	2936	2838
query25	676	358	365	358
query26	1805	144	145	144
query27	2980	263	261	261
query28	7547	1967	1967	1967
query29	1171	658	593	593
query30	283	146	143	143
query31	951	739	721	721
query32	94	52	52	52
query33	775	331	326	326
query34	908	466	476	466
query35	892	731	717	717
query36	1126	938	950	938
query37	196	80	77	77
query38	2856	2737	2731	2731
query39	858	826	806	806
query40	284	121	121	121
query41	45	45	48	45
query42	119	99	97	97
query43	504	481	474	474
query44	1224	712	714	712
query45	195	160	167	160
query46	1107	711	770	711
query47	1835	1774	1778	1774
query48	368	282	333	282
query49	1195	409	414	409
query50	786	384	380	380
query51	6659	6661	6599	6599
query52	97	94	93	93
query53	354	290	278	278
query54	1007	436	432	432
query55	75	70	72	70
query56	284	262	269	262
query57	1156	1042	1030	1030
query58	273	247	257	247
query59	2803	2652	2594	2594
query60	309	270	276	270
query61	128	94	94	94
query62	849	659	655	655
query63	325	287	288	287
query64	10450	2211	1655	1655
query65	3146	3103	3128	3103
query66	1362	343	346	343
query67	15436	14968	14962	14962
query68	4532	533	533	533
query69	507	335	337	335
query70	1200	1151	1118	1118
query71	387	274	280	274
query72	7222	5797	5678	5678
query73	756	321	321	321
query74	6087	5691	5615	5615
query75	3364	2680	2681	2680
query76	2525	829	921	829
query77	491	324	292	292
query78	9456	8991	8907	8907
query79	2398	523	507	507
query80	2156	466	460	460
query81	586	230	223	223
query82	835	132	135	132
query83	300	161	165	161
query84	270	91	84	84
query85	2208	317	327	317
query86	489	316	308	308
query87	3303	3070	3116	3070
query88	4033	2359	2346	2346
query89	467	377	454	377
query90	1842	188	183	183
query91	126	100	97	97
query92	62	49	50	49
query93	2741	492	479	479
query94	1273	285	270	270
query95	393	316	312	312
query96	596	271	264	264
query97	3161	3036	2992	2992
query98	203	203	198	198
query99	1560	1277	1230	1230
Total cold run time: 285098 ms
Total hot run time: 171271 ms

doris-robot · 2024-07-23T02:38:32Z

ClickBench: Total hot run time: 30.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 313401fec41091f9e90fe8804a08b99c5a6b0016, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.09	0.09
query5	0.50	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.56	0.49	0.48
query10	0.54	0.55	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.58
query14	0.76	0.80	0.77
query15	0.85	0.80	0.83
query16	0.36	0.36	0.36
query17	1.04	1.02	0.95
query18	0.23	0.21	0.22
query19	1.76	1.75	1.73
query20	0.01	0.01	0.01
query21	15.39	0.76	0.65
query22	4.43	7.18	1.67
query23	18.22	1.47	1.28
query24	2.23	0.24	0.22
query25	0.16	0.08	0.08
query26	0.29	0.22	0.20
query27	0.46	0.23	0.23
query28	13.19	1.02	0.99
query29	12.69	3.27	3.29
query30	0.26	0.05	0.05
query31	2.86	0.38	0.38
query32	3.29	0.48	0.47
query33	2.86	2.96	2.96
query34	17.13	4.37	4.35
query35	4.42	4.43	4.42
query36	0.66	0.47	0.47
query37	0.18	0.16	0.15
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.05	0.05
Total cold run time: 109.99 s
Total hot run time: 30.44 s

github-actions · 2024-07-23T05:45:17Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-23T05:45:19Z

PR approved by anyone and no changes requested.

…ffer_size_rows (#38223) pick #38221

wangbo

LGTM

…ize_rows` (#38221) support modifying arrow flight result sink buffer rows size with parameters, default 4096 * 8. we want to return a larger batch at a time, when large amounts of data.

…ize_rows` (apache#38221) support modifying arrow flight result sink buffer rows size with parameters, default 4096 * 8. we want to return a larger batch at a time, when large amounts of data.

1
f5cc58e

xinyiZzz mentioned this pull request Jul 22, 2024
[branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_buffer_size_rows #38223
Merged

2
313401f

yiguolei approved these changes Jul 23, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024

github-actionsBot added the reviewed label Jul 23, 2024

yiguolei pushed a commit that referenced this pull request Jul 24, 2024
[branch-2.1](arrow-flight-sql) Add config arrow_flight_result_sink_bu…
10c5c33
…ffer_size_rows (#38223) pick #38221

yiguolei added the dev/2.1.6-merged label Jul 24, 2024

wangbo approved these changes Jul 30, 2024
View reviewed changes

xinyiZzz merged commit 3119a08 into apache:masterJul 30, 2024

dataroaring added the dev/3.0.2-merged label Aug 11, 2024

Navigation Menu

Uh oh!

[opt](arrow-flight-sql) Add config `arrow_flight_result_sink_buffer_size_rows` - #38221

[opt](arrow-flight-sql) Add config `arrow_flight_result_sink_buffer_size_rows`#38221
xinyiZzz merged 2 commits into
apache:masterfrom
xinyiZzz:20240722_fix_arrow_flight_result_sink

xinyiZzz commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

xinyiZzz commented Jul 22, 2024

Uh oh!

github-actionsBot commented Jul 22, 2024

Uh oh!

xinyiZzz commented Jul 23, 2024

Uh oh!

xinyiZzz commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

wangbo left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

xinyiZzz commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

xinyiZzz commented Jul 22, 2024

Uh oh!

github-actionsBot commented Jul 22, 2024

Uh oh!

xinyiZzz commented Jul 23, 2024

Uh oh!

xinyiZzz commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

wangbo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants