Skip to content

[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader - #41683

Merged
morningman merged 12 commits into
apache:masterfrom
suxiaogang223:support_byte_stream_split
Oct 14, 2024
Merged

[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader#41683
morningman merged 12 commits into
apache:masterfrom
suxiaogang223:support_byte_stream_split

Conversation

@suxiaogang223

@suxiaogang223suxiaogang223 commented Oct 11, 2024

Copy link
Copy Markdown
Member

Proposed changes

Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding parquet.
relate pr: apache/arrow#42372

Apache Parquet does not have any encodings suitable for FP data and the available text compressors (zstd, gzip, etc) do not handle FP data very well.
It is possible to apply a simple data transformation named "stream splitting". Such could be "byte stream splitting" which creates K streams of length N where K is the number of bytes in the data type (4 for floats, 8 for doubles) and N is the number of elements in the sequence.

@doris-robot

doris-robot commented Oct 11, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions

github-actionsBot commented Oct 11, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

github-actionsBot commented Oct 11, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

morningman
morningman reviewed Oct 11, 2024
View reviewed changes
Comment threadbe/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp Outdated
Comment threadbe/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp Outdated
Comment threadbe/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp Outdated
github-actions[bot]
github-actionsBot reviewed Oct 11, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/util/byte_stream_split.cpp
Comment threadbe/src/util/byte_stream_split.cpp Outdated
Comment threadbe/src/util/byte_stream_split.cpp
Comment threadbe/src/util/byte_stream_split.cpp
Comment threadbe/src/util/byte_stream_split.cpp
Comment threadbe/src/util/byte_stream_split.cpp
github-actions[bot]
github-actionsBot reviewed Oct 11, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/util/byte_stream_split.cpp
github-actions[bot]
github-actionsBot reviewed Oct 11, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/util/byte_stream_split.cpp
github-actions[bot]
github-actionsBot reviewed Oct 11, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/test/util/byte_stream_split_test.cpp
@suxiaogang223

suxiaogang223 commented Oct 11, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Oct 11, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
------ Round 1 ----------------------------------
q1	17573	7349	7228	7228
q2	2030	275	270	270
q3	12084	1050	1163	1050
q4	10569	727	815	727
q5	7727	2858	2825	2825
q6	237	150	149	149
q7	984	620	603	603
q8	9340	1952	1945	1945
q9	7059	6412	6391	6391
q10	6950	2268	2293	2268
q11	449	244	246	244
q12	404	226	220	220
q13	17761	2990	2998	2990
q14	248	210	211	210
q15	564	525	519	519
q16	642	589	585	585
q17	965	543	563	543
q18	7141	6792	6691	6691
q19	1338	892	902	892
q20	488	199	195	195
q21	4011	3098	3229	3098
q22	1113	1016	1020	1016
Total cold run time: 109677 ms
Total hot run time: 40659 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7276	7186	7190	7186
q2	317	223	226	223
q3	2944	2898	2935	2898
q4	2112	1875	1800	1800
q5	5735	5692	5714	5692
q6	236	144	150	144
q7	2220	1851	1830	1830
q8	3320	3602	3406	3406
q9	8870	8863	8820	8820
q10	3590	3512	3485	3485
q11	585	494	503	494
q12	846	659	637	637
q13	10979	3203	3206	3203
q14	293	272	266	266
q15	566	512	518	512
q16	706	638	661	638
q17	1819	1605	1593	1593
q18	8198	7619	7650	7619
q19	1707	1429	1539	1429
q20	2131	1878	1889	1878
q21	5620	5378	5200	5200
q22	1156	1080	1023	1023
Total cold run time: 71226 ms
Total hot run time: 59976 ms

@doris-robot

doris-robot commented Oct 11, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191612 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
query1	834	380	396	380
query2	6226	2040	1992	1992
query3	8682	198	205	198
query4	33939	23598	23400	23400
query5	3734	476	493	476
query6	280	163	161	161
query7	4190	304	298	298
query8	286	221	226	221
query9	9523	2719	2680	2680
query10	463	279	271	271
query11	17679	15140	15197	15140
query12	153	101	95	95
query13	1594	464	458	458
query14	9631	7568	7605	7568
query15	247	174	179	174
query16	7418	454	435	435
query17	1617	603	617	603
query18	1870	331	328	328
query19	381	154	174	154
query20	121	111	111	111
query21	210	109	106	106
query22	4916	4403	4432	4403
query23	34880	34422	33813	33813
query24	11044	2766	2786	2766
query25	622	412	386	386
query26	967	157	162	157
query27	2240	301	293	293
query28	7482	2406	2391	2391
query29	739	440	433	433
query30	256	149	155	149
query31	1037	777	813	777
query32	99	55	59	55
query33	764	292	285	285
query34	923	497	517	497
query35	890	730	717	717
query36	1092	929	931	929
query37	152	85	87	85
query38	4132	3843	3969	3843
query39	1492	1419	1445	1419
query40	205	96	97	96
query41	48	44	46	44
query42	113	97	94	94
query43	524	485	473	473
query44	1274	803	800	800
query45	192	164	167	164
query46	1149	723	714	714
query47	1919	1839	1840	1839
query48	441	355	350	350
query49	904	412	411	411
query50	817	433	442	433
query51	7368	6889	6943	6889
query52	97	85	87	85
query53	272	182	180	180
query54	1172	491	472	472
query55	81	80	76	76
query56	280	258	290	258
query57	1205	1167	1145	1145
query58	226	225	234	225
query59	3178	3013	3038	3013
query60	293	269	277	269
query61	101	107	100	100
query62	872	659	668	659
query63	219	194	184	184
query64	3967	633	613	613
query65	3314	3248	3198	3198
query66	1321	296	300	296
query67	15936	15705	15716	15705
query68	4394	573	546	546
query69	461	290	319	290
query70	1178	1115	1150	1115
query71	342	268	274	268
query72	7120	3976	3960	3960
query73	758	351	353	351
query74	9502	8982	8914	8914
query75	3394	2684	2642	2642
query76	2903	909	896	896
query77	577	298	286	286
query78	10298	9539	9500	9500
query79	1910	581	602	581
query80	1487	447	447	447
query81	587	242	238	238
query82	699	137	142	137
query83	276	130	137	130
query84	283	79	75	75
query85	1480	302	286	286
query86	415	291	297	291
query87	4513	4273	4343	4273
query88	3766	2452	2394	2394
query89	396	287	286	286
query90	1984	188	185	185
query91	137	108	104	104
query92	65	52	45	45
query93	1270	545	544	544
query94	1060	290	283	283
query95	347	244	245	244
query96	610	280	285	280
query97	3229	3098	3106	3098
query98	225	206	191	191
query99	1528	1300	1313	1300
Total cold run time: 297487 ms
Total hot run time: 191612 ms

@doris-robot

doris-robot commented Oct 11, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
query1	0.03	0.02	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.64	0.10	0.10
query5	0.51	0.50	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.56	0.51	0.50
query10	0.56	0.56	0.55
query11	0.14	0.11	0.11
query12	0.14	0.10	0.10
query13	0.60	0.60	0.60
query14	2.71	2.74	2.73
query15	0.89	0.81	0.82
query16	0.39	0.38	0.38
query17	0.99	0.96	1.00
query18	0.20	0.19	0.21
query19	1.91	1.90	1.90
query20	0.02	0.01	0.01
query21	15.36	0.61	0.58
query22	2.19	1.79	1.50
query23	16.81	1.23	0.72
query24	2.72	1.83	1.68
query25	0.35	0.12	0.08
query26	0.50	0.15	0.14
query27	0.04	0.05	0.04
query28	9.70	1.08	1.07
query29	12.59	3.25	3.23
query30	0.25	0.05	0.06
query31	2.86	0.38	0.38
query32	3.28	0.46	0.45
query33	3.02	3.00	2.99
query34	16.87	4.44	4.46
query35	4.47	4.48	4.44
query36	0.67	0.49	0.49
query37	0.08	0.06	0.07
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.03	0.02
Total cold run time: 104.87 s
Total hot run time: 32.87 s

@doris-robot

doris-robot commented Oct 11, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.39% (9660/25837)
Line Coverage: 28.65% (80149/279726)
Region Coverage: 28.09% (41451/147573)
Branch Coverage: 24.70% (21120/85518)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1645ea725a237529dfe5f0647c6c9ba4eea116f0_1645ea725a237529dfe5f0647c6c9ba4eea116f0/report/index.html

@suxiaogang223

suxiaogang223 commented Oct 12, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Oct 12, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40984 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
------ Round 1 ----------------------------------
q1	18183	7500	7362	7362
q2	2447	185	179	179
q3	11775	1157	1182	1157
q4	11326	768	750	750
q5	7789	2901	2801	2801
q6	247	151	155	151
q7	996	645	619	619
q8	9357	1932	1912	1912
q9	6610	6436	6428	6428
q10	6947	2274	2295	2274
q11	447	254	241	241
q12	403	231	218	218
q13	17787	2969	2971	2969
q14	236	207	209	207
q15	588	531	510	510
q16	637	580	574	574
q17	971	558	612	558
q18	7076	6663	6746	6663
q19	1343	1019	969	969
q20	469	203	201	201
q21	3952	3245	3242	3242
q22	1110	1013	999	999
Total cold run time: 110696 ms
Total hot run time: 40984 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7288	7275	7280	7275
q2	330	229	220	220
q3	2943	2732	2783	2732
q4	1964	1759	1711	1711
q5	5452	5489	5494	5489
q6	226	144	140	140
q7	2128	1716	1722	1716
q8	3236	3391	3416	3391
q9	8526	8491	8508	8491
q10	3468	3452	3421	3421
q11	576	473	483	473
q12	800	615	585	585
q13	5593	2976	3019	2976
q14	287	254	275	254
q15	576	518	516	516
q16	663	627	632	627
q17	1796	1565	1558	1558
q18	7819	7469	7330	7330
q19	1658	1511	1463	1463
q20	2057	1829	1785	1785
q21	5242	5084	5178	5084
q22	1121	1054	987	987
Total cold run time: 63749 ms
Total hot run time: 58224 ms

@doris-robot

doris-robot commented Oct 12, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.40% (9662/25837)
Line Coverage: 28.67% (80205/279726)
Region Coverage: 28.09% (41459/147573)
Branch Coverage: 24.71% (21129/85518)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4c47c3e413b11965eadaf04df35fb5ffaf3f451a_4c47c3e413b11965eadaf04df35fb5ffaf3f451a/report/index.html

@doris-robot

doris-robot commented Oct 12, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191651 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
query1	974	369	378	369
query2	6510	2087	2080	2080
query3	6708	217	225	217
query4	33605	23361	23631	23361
query5	4317	470	455	455
query6	263	169	168	168
query7	4629	303	301	301
query8	284	224	231	224
query9	9728	2682	2697	2682
query10	462	284	285	284
query11	18098	15301	15123	15123
query12	151	97	93	93
query13	1670	459	435	435
query14	10008	7456	7416	7416
query15	254	166	174	166
query16	8158	456	456	456
query17	1676	576	556	556
query18	2100	305	317	305
query19	368	151	159	151
query20	119	105	108	105
query21	213	103	103	103
query22	4874	4539	4473	4473
query23	34783	34075	34116	34075
query24	11076	2768	2772	2768
query25	662	399	413	399
query26	1180	162	160	160
query27	2830	301	293	293
query28	8056	2443	2422	2422
query29	850	443	440	440
query30	319	156	156	156
query31	1037	774	820	774
query32	91	57	58	57
query33	779	305	288	288
query34	979	500	520	500
query35	872	722	732	722
query36	1116	908	968	908
query37	154	92	86	86
query38	3942	3948	3881	3881
query39	1505	1438	1417	1417
query40	218	98	97	97
query41	50	46	47	46
query42	119	98	99	98
query43	534	485	471	471
query44	1234	807	810	807
query45	196	168	171	168
query46	1145	722	724	722
query47	1947	1853	1859	1853
query48	433	345	340	340
query49	1137	427	426	426
query50	815	402	423	402
query51	7138	6903	6884	6884
query52	98	89	88	88
query53	260	198	186	186
query54	1327	472	486	472
query55	82	77	80	77
query56	297	266	272	266
query57	1300	1174	1141	1141
query58	256	244	256	244
query59	3207	3000	2860	2860
query60	324	286	275	275
query61	126	121	135	121
query62	873	682	684	682
query63	224	191	184	184
query64	5378	612	606	606
query65	3249	3172	3252	3172
query66	1422	316	310	310
query67	15948	15610	15503	15503
query68	4909	559	547	547
query69	453	295	295	295
query70	1111	1126	1105	1105
query71	317	270	268	268
query72	6355	3963	3976	3963
query73	768	344	343	343
query74	9727	8864	8997	8864
query75	3358	2660	2664	2660
query76	2972	932	870	870
query77	431	301	304	301
query78	10536	9742	9674	9674
query79	1835	604	611	604
query80	1702	440	430	430
query81	575	238	245	238
query82	978	139	140	139
query83	246	137	134	134
query84	258	77	76	76
query85	1290	284	279	279
query86	379	280	290	280
query87	4459	4252	4421	4252
query88	3850	2452	2424	2424
query89	405	285	289	285
query90	1859	186	186	186
query91	157	121	104	104
query92	61	46	47	46
query93	1106	529	543	529
query94	964	278	281	278
query95	352	250	251	250
query96	610	288	289	288
query97	3252	3086	3156	3086
query98	211	202	203	202
query99	1568	1317	1281	1281
Total cold run time: 301791 ms
Total hot run time: 191651 ms

@doris-robot

doris-robot commented Oct 12, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.66	0.10	0.11
query5	0.50	0.52	0.51
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.57	0.50	0.50
query10	0.56	0.55	0.55
query11	0.14	0.11	0.12
query12	0.13	0.10	0.11
query13	0.61	0.60	0.59
query14	2.73	2.75	2.76
query15	0.89	0.84	0.82
query16	0.38	0.39	0.38
query17	1.07	1.10	1.03
query18	0.24	0.22	0.22
query19	1.96	1.76	1.99
query20	0.02	0.01	0.01
query21	15.36	0.58	0.56
query22	3.26	1.63	2.36
query23	17.15	0.86	0.76
query24	3.46	0.79	1.14
query25	0.11	0.14	0.06
query26	0.58	0.15	0.15
query27	0.06	0.05	0.04
query28	10.64	1.10	1.07
query29	12.53	3.25	3.19
query30	0.25	0.06	0.06
query31	3.14	0.39	0.37
query32	3.76	0.45	0.46
query33	3.01	3.02	3.03
query34	17.10	4.47	4.50
query35	4.51	4.50	4.59
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 109.02 s
Total hot run time: 32.18 s

@morningman

morningman commented Oct 13, 2024

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Oct 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 40815 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
------ Round 1 ----------------------------------
q1	17610	7737	7258	7258
q2	2024	297	288	288
q3	12171	1078	1172	1078
q4	10589	739	746	739
q5	7767	2903	2856	2856
q6	237	152	150	150
q7	998	634	636	634
q8	9364	2010	1872	1872
q9	6554	6483	6431	6431
q10	6983	2256	2290	2256
q11	433	245	250	245
q12	401	216	221	216
q13	17776	3002	2969	2969
q14	250	207	210	207
q15	582	525	525	525
q16	647	578	593	578
q17	979	554	554	554
q18	7233	6678	6821	6678
q19	1350	966	914	914
q20	484	210	197	197
q21	4007	3217	3171	3171
q22	1142	999	1026	999
Total cold run time: 109581 ms
Total hot run time: 40815 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7265	7183	7204	7183
q2	332	230	232	230
q3	2989	2940	2943	2940
q4	2069	1866	1791	1791
q5	5806	5695	5805	5695
q6	236	140	140	140
q7	2267	1830	1826	1826
q8	3372	3635	3442	3442
q9	8935	8919	8904	8904
q10	3583	3575	3555	3555
q11	586	480	500	480
q12	806	674	666	666
q13	9695	3175	3129	3129
q14	310	287	267	267
q15	573	511	514	511
q16	678	634	631	631
q17	1872	1615	1608	1608
q18	8328	7840	7685	7685
q19	1707	1537	1404	1404
q20	2132	1848	1926	1848
q21	5576	5396	5488	5396
q22	1150	1099	1064	1064
Total cold run time: 70267 ms
Total hot run time: 60395 ms

@doris-robot

doris-robot commented Oct 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191118 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
query1	905	410	428	410
query2	6281	2068	1943	1943
query3	8682	198	202	198
query4	34321	23427	23709	23427
query5	3587	484	464	464
query6	276	164	161	161
query7	4203	299	301	299
query8	273	209	215	209
query9	9508	2682	2691	2682
query10	474	282	277	277
query11	17834	15073	15210	15073
query12	148	100	95	95
query13	1586	481	441	441
query14	9253	7418	7386	7386
query15	276	171	171	171
query16	7981	483	446	446
query17	1693	630	588	588
query18	2160	322	312	312
query19	381	160	164	160
query20	116	118	115	115
query21	214	103	104	103
query22	4611	4728	4453	4453
query23	34889	33588	34080	33588
query24	11202	2798	2821	2798
query25	613	409	403	403
query26	1100	159	161	159
query27	2442	298	300	298
query28	7570	2395	2396	2395
query29	828	448	429	429
query30	261	160	152	152
query31	1021	795	792	792
query32	103	54	52	52
query33	749	315	310	310
query34	899	509	504	504
query35	876	715	726	715
query36	1101	910	960	910
query37	150	83	88	83
query38	4051	3862	3884	3862
query39	1497	1443	1410	1410
query40	203	97	98	97
query41	47	45	46	45
query42	117	99	96	96
query43	530	498	479	479
query44	1308	822	813	813
query45	199	166	169	166
query46	1143	716	691	691
query47	1977	1892	1837	1837
query48	430	339	368	339
query49	912	420	405	405
query50	879	416	428	416
query51	6954	6982	7019	6982
query52	97	89	90	89
query53	258	194	183	183
query54	1152	465	484	465
query55	82	77	102	77
query56	266	257	265	257
query57	1259	1113	1154	1113
query58	222	230	240	230
query59	3202	3207	3078	3078
query60	296	275	261	261
query61	103	100	106	100
query62	875	658	657	657
query63	210	185	187	185
query64	4133	633	603	603
query65	3457	3182	3190	3182
query66	1289	297	316	297
query67	15807	15517	15488	15488
query68	4654	612	570	570
query69	535	301	305	301
query70	1104	1082	1096	1082
query71	360	270	275	270
query72	7030	3928	4082	3928
query73	779	348	361	348
query74	10275	8838	8948	8838
query75	3331	2686	2643	2643
query76	2880	895	884	884
query77	432	323	314	314
query78	10764	9742	9639	9639
query79	1345	590	592	590
query80	1492	436	452	436
query81	583	240	239	239
query82	753	138	133	133
query83	294	135	133	133
query84	283	72	75	72
query85	1342	291	282	282
query86	389	305	309	305
query87	4462	4362	4308	4308
query88	3118	2426	2356	2356
query89	404	284	282	282
query90	2263	184	179	179
query91	140	104	103	103
query92	60	48	49	48
query93	1059	540	541	540
query94	1178	285	278	278
query95	357	255	257	255
query96	613	277	274	274
query97	3253	3096	3143	3096
query98	211	197	193	193
query99	1683	1296	1295	1295
Total cold run time: 298948 ms
Total hot run time: 191118 ms

@doris-robot

doris-robot commented Oct 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.65	0.10	0.10
query5	0.52	0.52	0.53
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.02
query9	0.56	0.51	0.49
query10	0.54	0.55	0.54
query11	0.14	0.11	0.11
query12	0.13	0.10	0.11
query13	0.61	0.60	0.59
query14	2.85	2.71	2.76
query15	0.90	0.82	0.82
query16	0.39	0.38	0.35
query17	1.00	1.01	1.00
query18	0.20	0.19	0.21
query19	1.86	1.87	2.01
query20	0.01	0.01	0.01
query21	15.35	0.58	0.56
query22	2.61	2.39	1.50
query23	17.13	0.86	0.78
query24	2.63	1.40	1.96
query25	0.33	0.16	0.15
query26	0.40	0.13	0.14
query27	0.04	0.04	0.06
query28	9.90	1.10	1.06
query29	12.62	3.30	3.26
query30	0.24	0.06	0.05
query31	2.90	0.39	0.38
query32	3.25	0.45	0.46
query33	3.01	3.03	3.01
query34	17.20	4.55	4.47
query35	4.58	4.61	4.52
query36	0.64	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.03
query42	0.03	0.02	0.01
query43	0.04	0.03	0.02
Total cold run time: 106.15 s
Total hot run time: 32.81 s

@doris-robot

doris-robot commented Oct 13, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.42% (9682/25871)
Line Coverage: 28.68% (80309/280063)
Region Coverage: 28.11% (41537/147740)
Branch Coverage: 24.70% (21122/85520)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f67ba7392f1470d524153da38c7f285f2583d12f_f67ba7392f1470d524153da38c7f285f2583d12f/report/index.html

@github-actions

github-actionsBot commented Oct 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 14, 2024
@github-actions

github-actionsBot commented Oct 14, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit 4216b03 into apache:masterOct 14, 2024
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Oct 16, 2024
@suxiaogang223@morningman
[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet rea…
f285248
…der (apache#41683)
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding
parquet.
relate pr: apache/arrow#42372
> Apache Parquet does not have any encodings suitable for FP data and
the available text compressors (zstd, gzip, etc) do not handle FP data
very well.
It is possible to apply a simple data transformation named "stream
splitting". Such could be "byte stream splitting" which creates K
streams of length N where K is the number of bytes in the data type (4
for floats, 8 for doubles) and N is the number of elements in the
sequence.
---------
Co-authored-by: morningman <morningman@163.com>
@suxiaogang223suxiaogang223 mentioned this pull request Oct 16, 2024
Merged
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
@suxiaogang223@morningmangarenshi
[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet rea…
1667e6f
…der (apache#41683)
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding
parquet.
relate pr: apache/arrow#42372
> Apache Parquet does not have any encodings suitable for FP data and
the available text compressors (zstd, gzip, etc) do not handle FP data
very well.
It is possible to apply a simple data transformation named "stream
splitting". Such could be "byte stream splitting" which creates K
streams of length N where K is the number of bytes in the data type (4
for floats, 8 for doubles) and N is the number of elements in the
sequence.
---------
Co-authored-by: morningman <morningman@163.com>
suxiaogang223 added a commit to suxiaogang223/doris that referenced this pull request Oct 16, 2024
@suxiaogang223@morningman
[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet rea…
c476529
…der (apache#41683)
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding
parquet.
relate pr: apache/arrow#42372
> Apache Parquet does not have any encodings suitable for FP data and
the available text compressors (zstd, gzip, etc) do not handle FP data
very well.
It is possible to apply a simple data transformation named "stream
splitting". Such could be "byte stream splitting" which creates K
streams of length N where K is the number of bytes in the data type (4
for floats, 8 for doubles) and N is the number of elements in the
sequence.
---------
Co-authored-by: morningman <morningman@163.com>
@suxiaogang223suxiaogang223 mentioned this pull request Oct 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Oct 17, 2024
@suxiaogang223@morningman
[cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet …
1b901f6
…bug (#41931)
## Proposed changes
pick pr:
#41683#41506#41338#39326
---------
Co-authored-by: morningman <morningman@163.com>
morningman added a commit that referenced this pull request Oct 17, 2024
@suxiaogang223@morningman
[cherry-pick](branch-3.0) fix parquet cases (#41506#41526#41683#41816
e634f0d
) (#41923)
## Proposed changes
pick prs:
#41506#41526#41683#41816
---------
Co-authored-by: morningman <morningman@163.com>
@suxiaogang223
suxiaogang223 deleted the support_byte_stream_split branch October 21, 2024 11:23
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
@suxiaogang223@morningman@HappenLee
[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet rea…
dc3f53f
…der (apache#41683)
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding
parquet.
relate pr: apache/arrow#42372
> Apache Parquet does not have any encodings suitable for FP data and
the available text compressors (zstd, gzip, etc) do not handle FP data
very well.
It is possible to apply a simple data transformation named "stream
splitting". Such could be "byte stream splitting" which creates K
streams of length N where K is the number of bytes in the data type (4
for floats, 8 for doubles) and N is the number of elements in the
sequence.
---------
Co-authored-by: morningman <morningman@163.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments
@morningmanmorningmanmorningman approved these changes
@CalvinKirsCalvinKirsCalvinKirs approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.7-mergeddev/3.0.3-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@suxiaogang223@doris-robot@morningman@CalvinKirs@yiguolei