[Optimize](Row store) allow to set row_store_page_size for tables, change default value to 16KB by lxr599 · Pull Request #37145 · apache/doris

lxr599 · 2024-07-02T09:14:58Z

Proposed changes

In current row storage format, under different page sizes, the compression effect of zstd on the hits data set (70 GB) is as follows.

no_compression	zstd (default, page size=4KB)	zstd ( page size=16KB)	zstd ( page size=32KB)	zstd ( page size=64KB)
105.76 GB	33.38 GB	22.07 GB	21.83 GB	18.29 GB

When the page size is 16KB, the optimization is the greatest (33.89%). When the page size is increased, the effect is not obvious.

doris-robot · 2024-07-02T09:15:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

eldenmoon

LGTM

eldenmoon · 2024-07-02T09:16:22Z

run buildall

eldenmoon

LGTM

github-actions · 2024-07-02T09:18:00Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-02T09:18:02Z

PR approved by anyone and no changes requested.

github-actions · 2024-07-02T09:21:20Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-02T09:55:30Z

TPC-H: Total hot run time: 39399 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false
------ Round 1 ----------------------------------
q1	17633	4345	4214	4214
q2	2007	184	180	180
q3	10472	1201	1076	1076
q4	10185	875	832	832
q5	7461	2628	2635	2628
q6	215	138	134	134
q7	944	573	588	573
q8	9221	2052	2031	2031
q9	9109	6477	6411	6411
q10	8979	3691	3715	3691
q11	468	226	231	226
q12	403	226	229	226
q13	17782	2939	2970	2939
q14	256	218	241	218
q15	526	502	481	481
q16	534	398	373	373
q17	958	609	713	609
q18	8031	7397	7338	7338
q19	7089	1472	1433	1433
q20	664	322	313	313
q21	4833	3146	3915	3146
q22	389	336	327	327
Total cold run time: 118159 ms
Total hot run time: 39399 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4381	4241	4233	4233
q2	364	267	267	267
q3	2997	2837	2891	2837
q4	1979	1693	1704	1693
q5	5618	5482	5451	5451
q6	222	137	136	136
q7	2232	1888	1880	1880
q8	3225	3399	3402	3399
q9	8677	8718	8769	8718
q10	4151	3731	3736	3731
q11	595	493	499	493
q12	818	649	660	649
q13	17065	3107	3230	3107
q14	292	278	262	262
q15	529	477	485	477
q16	492	419	440	419
q17	1812	1532	1502	1502
q18	8181	7822	7880	7822
q19	1823	1703	1624	1624
q20	2186	1889	1840	1840
q21	8592	4760	4684	4684
q22	626	546	541	541
Total cold run time: 76857 ms
Total hot run time: 55765 ms

doris-robot · 2024-07-02T10:06:50Z

TPC-DS: Total hot run time: 173949 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false
query1	904	378	373	373
query2	6438	2468	2411	2411
query3	6627	205	213	205
query4	21485	17512	17326	17326
query5	3730	466	467	466
query6	256	170	166	166
query7	4599	287	293	287
query8	315	289	285	285
query9	8377	2356	2334	2334
query10	585	326	276	276
query11	10644	9972	10178	9972
query12	117	90	82	82
query13	1646	358	352	352
query14	9663	7778	7801	7778
query15	243	180	187	180
query16	8022	268	277	268
query17	1891	543	529	529
query18	2091	281	275	275
query19	197	158	153	153
query20	90	82	83	82
query21	215	132	124	124
query22	4714	4189	4160	4160
query23	33735	33535	33442	33442
query24	11069	2849	2904	2849
query25	599	403	386	386
query26	703	157	156	156
query27	2224	322	331	322
query28	6124	2105	2088	2088
query29	912	649	644	644
query30	250	159	159	159
query31	985	787	757	757
query32	101	57	55	55
query33	772	303	311	303
query34	1022	478	481	478
query35	758	660	638	638
query36	1165	993	968	968
query37	140	82	80	80
query38	2984	2852	2807	2807
query39	887	811	804	804
query40	211	128	124	124
query41	58	52	55	52
query42	113	100	97	97
query43	599	552	570	552
query44	1192	731	711	711
query45	198	168	166	166
query46	1084	737	706	706
query47	1852	1758	1776	1758
query48	377	299	297	297
query49	851	505	404	404
query50	763	378	374	374
query51	6843	6754	6697	6697
query52	102	93	87	87
query53	356	288	289	288
query54	930	421	425	421
query55	71	82	70	70
query56	268	258	261	258
query57	1107	1088	1062	1062
query58	234	240	236	236
query59	3463	3154	3058	3058
query60	295	265	266	265
query61	111	87	87	87
query62	593	447	447	447
query63	320	292	291	291
query64	8491	2259	1708	1708
query65	3125	3193	3074	3074
query66	754	321	343	321
query67	15724	14941	14906	14906
query68	9013	540	535	535
query69	716	388	342	342
query70	1361	1154	1083	1083
query71	528	315	281	281
query72	9503	5390	5570	5390
query73	2224	320	317	317
query74	5823	5445	5447	5445
query75	6039	2616	2651	2616
query76	5261	901	870	870
query77	785	302	300	300
query78	10548	10005	9788	9788
query79	7824	508	504	504
query80	928	475	500	475
query81	549	221	228	221
query82	262	104	103	103
query83	323	169	162	162
query84	280	84	84	84
query85	888	267	268	267
query86	349	299	300	299
query87	3302	3080	3072	3072
query88	4386	2361	2344	2344
query89	492	411	374	374
query90	2074	187	178	178
query91	124	98	98	98
query92	67	48	47	47
query93	5332	501	493	493
query94	1332	183	183	183
query95	400	311	308	308
query96	616	262	257	257
query97	3163	3003	2966	2966
query98	207	223	197	197
query99	1244	846	813	813
Total cold run time: 293307 ms
Total hot run time: 173949 ms

doris-robot · 2024-07-02T10:12:20Z

ClickBench: Total hot run time: 30.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec59d28e898d50455ab8fb919aead8d4cff9c869, data reload: false
query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.06
query4	1.69	0.08	0.08
query5	0.50	0.49	0.47
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.53	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.58	0.58	0.58
query14	0.78	0.76	0.79
query15	0.84	0.81	0.80
query16	0.35	0.35	0.36
query17	0.95	0.98	0.97
query18	0.20	0.27	0.26
query19	1.79	1.74	1.63
query20	0.01	0.01	0.01
query21	15.43	0.77	0.67
query22	4.61	6.37	2.08
query23	18.31	1.38	1.20
query24	2.17	0.23	0.23
query25	0.14	0.08	0.08
query26	0.28	0.18	0.18
query27	0.08	0.08	0.08
query28	13.22	1.01	0.99
query29	12.67	3.30	3.27
query30	0.25	0.06	0.06
query31	2.88	0.38	0.39
query32	3.29	0.48	0.46
query33	2.86	2.89	2.84
query34	16.78	4.41	4.42
query35	4.50	4.48	4.45
query36	0.63	0.45	0.46
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.16
query41	0.08	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 30.49 s

eldenmoon · 2024-07-02T10:13:08Z

change row_store_page_size to table property

github-actions · 2024-07-04T11:21:05Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-04T11:38:50Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-04T11:47:11Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-07-16T02:40:46Z

PR approved by at least one committer and no changes requested.

eldenmoon · 2024-07-16T02:56:16Z

run buildall

github-actions · 2024-07-16T03:01:45Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-16T03:38:53Z

TPC-H: Total hot run time: 39905 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false
------ Round 1 ----------------------------------
q1	18780	8571	4392	4392
q2	2718	197	191	191
q3	11207	1216	1020	1020
q4	11057	859	804	804
q5	7574	2747	2700	2700
q6	234	143	138	138
q7	969	612	607	607
q8	9578	2062	2075	2062
q9	8780	6595	6573	6573
q10	8787	3776	3720	3720
q11	445	243	245	243
q12	396	223	224	223
q13	18984	2931	2981	2931
q14	270	236	228	228
q15	535	478	488	478
q16	479	394	384	384
q17	970	636	683	636
q18	8185	7554	7533	7533
q19	6891	1429	1273	1273
q20	676	338	332	332
q21	4903	3175	3154	3154
q22	356	286	283	283
Total cold run time: 122774 ms
Total hot run time: 39905 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4363	4253	4237	4237
q2	366	264	272	264
q3	2991	2724	2791	2724
q4	1869	1626	1586	1586
q5	5332	5296	5323	5296
q6	220	129	129	129
q7	2125	1669	1758	1669
q8	3200	3328	3310	3310
q9	8413	8398	8413	8398
q10	3919	3713	3679	3679
q11	584	495	482	482
q12	766	598	608	598
q13	16653	3018	2952	2952
q14	327	271	274	271
q15	517	473	483	473
q16	465	411	437	411
q17	1767	1494	1498	1494
q18	7656	7469	7318	7318
q19	1693	1475	1379	1379
q20	2041	1785	1787	1785
q21	4910	4614	4525	4525
q22	600	472	489	472
Total cold run time: 70777 ms
Total hot run time: 53452 ms

doris-robot · 2024-07-16T03:50:03Z

TPC-DS: Total hot run time: 172990 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false
query1	921	362	367	362
query2	6493	1968	1856	1856
query3	6664	203	215	203
query4	28357	17578	17450	17450
query5	4197	500	481	481
query6	273	161	161	161
query7	4594	290	286	286
query8	237	190	184	184
query9	8550	2469	2445	2445
query10	431	290	267	267
query11	10924	10238	10227	10227
query12	139	79	80	79
query13	1627	378	365	365
query14	10123	7813	7669	7669
query15	216	168	167	167
query16	7548	324	319	319
query17	1787	548	540	540
query18	1781	271	266	266
query19	195	148	144	144
query20	92	81	87	81
query21	204	126	130	126
query22	4403	4074	4124	4074
query23	33786	33082	32888	32888
query24	12007	2883	2801	2801
query25	650	367	373	367
query26	1771	148	146	146
query27	2803	275	274	274
query28	7394	2023	2013	2013
query29	1046	658	612	612
query30	292	152	146	146
query31	954	733	765	733
query32	96	54	57	54
query33	782	311	319	311
query34	900	484	487	484
query35	656	596	585	585
query36	1067	926	976	926
query37	173	81	79	79
query38	2884	2743	2760	2743
query39	858	794	818	794
query40	281	120	116	116
query41	50	45	45	45
query42	114	95	99	95
query43	507	478	488	478
query44	1210	739	723	723
query45	189	163	162	162
query46	1079	710	725	710
query47	1870	1813	1757	1757
query48	377	296	297	296
query49	1192	412	409	409
query50	793	396	393	393
query51	6988	6795	6733	6733
query52	105	90	95	90
query53	353	287	284	284
query54	1008	445	492	445
query55	79	76	73	73
query56	280	253	260	253
query57	1158	1017	1059	1017
query58	251	251	243	243
query59	2955	2805	2719	2719
query60	309	275	282	275
query61	93	88	88	88
query62	837	653	668	653
query63	313	296	292	292
query64	10567	2222	1676	1676
query65	3175	3126	3106	3106
query66	1379	332	327	327
query67	15548	15040	15265	15040
query68	4599	537	540	537
query69	472	329	323	323
query70	1174	1095	1090	1090
query71	400	272	262	262
query72	6906	5408	5409	5408
query73	746	327	323	323
query74	6147	5680	5715	5680
query75	3414	2703	2657	2657
query76	2652	954	929	929
query77	444	309	300	300
query78	9575	9002	8962	8962
query79	2520	533	534	533
query80	1914	465	472	465
query81	597	219	231	219
query82	955	136	137	136
query83	287	161	176	161
query84	268	89	86	86
query85	1284	313	299	299
query86	405	332	302	302
query87	3287	3160	3168	3160
query88	3968	2483	2511	2483
query89	478	385	381	381
query90	1785	261	188	188
query91	127	102	100	100
query92	61	48	51	48
query93	1767	497	495	495
query94	1182	214	215	214
query95	397	315	313	313
query96	596	273	278	273
query97	3245	3036	3009	3009
query98	213	197	189	189
query99	1602	1259	1285	1259
Total cold run time: 284402 ms
Total hot run time: 172990 ms

doris-robot · 2024-07-16T03:55:35Z

ClickBench: Total hot run time: 31.26 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 617dafd5ec4a9b3ce1bdd6dffc65ad2c54189725, data reload: false
query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.10
query5	0.50	0.49	0.49
query6	1.22	0.72	0.74
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.48	0.49
query10	0.54	0.55	0.55
query11	0.17	0.12	0.12
query12	0.14	0.13	0.13
query13	0.60	0.58	0.58
query14	0.76	0.78	0.77
query15	0.84	0.80	0.81
query16	0.38	0.36	0.36
query17	0.96	1.02	1.06
query18	0.22	0.21	0.21
query19	1.90	1.83	1.79
query20	0.01	0.01	0.02
query21	15.39	0.75	0.67
query22	3.96	6.53	2.60
query23	18.30	1.39	1.25
query24	2.15	0.22	0.21
query25	0.17	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.24	0.22
query28	13.33	1.01	1.01
query29	12.62	3.28	3.25
query30	0.25	0.06	0.06
query31	2.90	0.38	0.39
query32	3.23	0.48	0.47
query33	2.87	2.94	2.84
query34	17.08	4.30	4.38
query35	4.43	4.45	4.39
query36	0.65	0.46	0.47
query37	0.19	0.15	0.16
query38	0.15	0.15	0.17
query39	0.04	0.03	0.03
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.67 s
Total hot run time: 31.26 s

xiaokang

LGTM

eldenmoon

LGTM

…ange default value to 16KB (apache#37145) In current row storage format, under different page sizes, the compression effect of zstd on the hits data set (70 GB) is as follows. | no_compression| zstd (default, page size=4KB) |zstd ( page size=16KB) |zstd ( page size=32KB) |zstd ( page size=64KB) | |:--------:| :---------:|:--------:|:--------:|:--------:| | 105.76 GB |33.38 GB |22.07 GB|21.83 GB|18.29 GB| When the page size is 16KB, the optimization is the greatest (33.89%). When the page size is increased, the effect is not obvious.

… default value to 16KB pick #37145

…ange default value to 16KB (apache#37145) In current row storage format, under different page sizes, the compression effect of zstd on the hits data set (70 GB) is as follows. | no_compression| zstd (default, page size=4KB) |zstd ( page size=16KB) |zstd ( page size=32KB) |zstd ( page size=64KB) | |:--------:| :---------:|:--------:|:--------:|:--------:| | 105.76 GB |33.38 GB |22.07 GB|21.83 GB|18.29 GB| When the page size is 16KB, the optimization is the greatest (33.89%). When the page size is increased, the effect is not obvious.

change default value of row_column_page_size to 16KB
ec59d28

eldenmoon reviewed Jul 2, 2024
View reviewed changes

eldenmoon changed the title ~~change default value of row_column_page_size to 16KB~~[Optimize](Row store) change default value of row_column_page_size to 16KBJul 2, 2024

eldenmoon previously approved these changes Jul 2, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024

github-actionsBot added the reviewed label Jul 2, 2024

eldenmoon added the dev/2.1.x label Jul 2, 2024

eldenmoon added the dev/3.0.x label Jul 2, 2024

allow to set row_column_page_size for tables
76a78da

lxr599 dismissed eldenmoon’s stale review via 76a78daJuly 4, 2024 11:14

lxr599 requested review from CalvinKirs, dataroaring and morningman as code owners July 4, 2024 11:14

github-actionsBot added the meta-change label Jul 4, 2024

[feature](Row store)allow to set row_column_page_size for tables
288bf5a

format codes
b544145

lxr599 added 2 commits July 5, 2024 12:54

change default value of row_column_page_size to 16KB
2339bd1

allow to set row_column_page_size for tables
9a211b1

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024

Merge branch 'master' into page_size
617dafd

eldenmoon added the doing label Jul 16, 2024

xiaokang approved these changes Jul 16, 2024
View reviewed changes

eldenmoon approved these changes Jul 16, 2024
View reviewed changes

xiaokang requested review from xiaokang and removed request for xiaokang July 16, 2024 06:16

morningman approved these changes Jul 16, 2024
View reviewed changes

eldenmoon merged commit 5c34742 into apache:masterJul 16, 2024

eldenmoon removed the dev/2.1.x label Jul 16, 2024

lxr599 mentioned this pull request Jul 16, 2024
[Pick](Row store)pick #37145 #37944
Merged

eldenmoon pushed a commit that referenced this pull request Jul 17, 2024
[Pick](Row store) allow to set row_store_page_size for tables, change…
5a44f57
… default value to 16KB pick #37145

xiaokang added dev/2.1.x dev/3.0.1-merged and removed dev/3.0.x labels Jul 19, 2024

yiguolei mentioned this pull request Jul 19, 2024
2.1.5 Release Notes #38111
Closed
1 task

lxr599 mentioned this pull request Aug 6, 2024
[Optimize](Row store) pick #37145, #38236 #38932
Merged

xiaokang pushed a commit that referenced this pull request Aug 7, 2024
[Optimize](Row store) pick#37145,#38236(#38932)
2543b56

xiaokang added dev/2.1.6-merged and removed dev/2.1.x doing labels Aug 7, 2024

yiguolei mentioned this pull request Sep 5, 2024
Release Note 2.1.6 #40443
Closed
3 tasks

Uh oh!

Conversation

lxr599 commented Jul 2, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon commented Jul 2, 2024

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 2, 2024

Uh oh!

github-actionsBot commented Jul 2, 2024

Uh oh!

github-actionsBot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

doris-robot commented Jul 2, 2024

Uh oh!

eldenmoon commented Jul 2, 2024

Uh oh!

github-actionsBot commented Jul 4, 2024

Uh oh!

github-actionsBot commented Jul 4, 2024

Uh oh!

github-actionsBot commented Jul 4, 2024

Uh oh!

github-actionsBot commented Jul 16, 2024

Uh oh!

eldenmoon commented Jul 16, 2024

Uh oh!

github-actionsBot commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

doris-robot commented Jul 16, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

eldenmoon left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants