Skip to content

[fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) (#38251) (#38356) - #39263

Merged
dataroaring merged 4 commits into
apache:branch-2.0from
luwei16:luwei/fix-segment-row-mismatch-2.0
Aug 14, 2024
Merged

[fix](compaction) fix mismatch between segment key and value column rows during compaction (#37960) (#38251) (#38356) #39263
dataroaring merged 4 commits into
apache:branch-2.0from
luwei16:luwei/fix-segment-row-mismatch-2.0

Conversation

@luwei16

@luwei16luwei16 commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

pick master #37960#38251#38356

@luwei16
[fix](compaction) fix mismatch between segment key and value column r…
ac0841b
…ows during compaction (apache#37960)
When a block is splitted to 3 segments, old code just handles 2 and the
last is overlowed.
@luwei16
[fix](compaction) fix the issue of writing segment with 0 rows during…
91c30d5
… compaction (apache#38251)
apache#37960 introduced this issue
@luwei16
[fix](compaction) fix missing flush column during compaction (apache#…
8cce17f
…38356)
apache#37960 introduces this problem
@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions

github-actionsBot commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring

dataroaring commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

run buildall

@github-actions

github-actionsBot commented Aug 13, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@luwei16

luwei16 commented Aug 13, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.74% (8107/21484)
Line Coverage: 29.37% (66437/226202)
Region Coverage: 28.87% (34269/118708)
Branch Coverage: 24.75% (17615/71172)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aaa5b10de5a546910512d351e13c9c287f29809b_aaa5b10de5a546910512d351e13c9c287f29809b/report/index.html

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 50029 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aaa5b10de5a546910512d351e13c9c287f29809b, data reload: false
------ Round 1 ----------------------------------
q1	17635	4376	4392	4376
q2	2067	158	151	151
q3	10416	1878	1937	1878
q4	10318	1265	1371	1265
q5	8572	3923	3899	3899
q6	234	129	125	125
q7	2040	1613	1565	1565
q8	9261	2753	2707	2707
q9	10883	10504	10497	10497
q10	8661	3532	3555	3532
q11	426	252	255	252
q12	458	310	299	299
q13	18342	3999	4017	3999
q14	353	336	327	327
q15	506	463	459	459
q16	682	586	569	569
q17	1130	975	971	971
q18	7196	6942	6797	6797
q19	1759	1663	1609	1609
q20	560	325	302	302
q21	4470	4080	4011	4011
q22	536	452	439	439
Total cold run time: 116505 ms
Total hot run time: 50029 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4363	4275	4321	4275
q2	328	223	220	220
q3	4201	4153	4119	4119
q4	2763	2773	2767	2767
q5	7234	7204	7115	7115
q6	240	124	125	124
q7	3275	2842	2901	2842
q8	4419	4508	4543	4508
q9	17426	16967	17303	16967
q10	4239	4262	4280	4262
q11	763	678	666	666
q12	1029	863	857	857
q13	6927	3706	3752	3706
q14	448	425	443	425
q15	496	472	460	460
q16	740	691	670	670
q17	3840	3853	3827	3827
q18	8755	8653	8915	8653
q19	1737	1705	1679	1679
q20	2385	2133	2113	2113
q21	8542	8448	8532	8448
q22	1083	953	979	953
Total cold run time: 85233 ms
Total hot run time: 79656 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 204998 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aaa5b10de5a546910512d351e13c9c287f29809b, data reload: false
query1	932	392	408	392
query2	6529	2949	2819	2819
query3	6915	202	202	202
query4	19974	18045	17907	17907
query5	19734	6539	6581	6539
query6	300	220	226	220
query7	4160	308	320	308
query8	433	403	404	403
query9	3210	2748	2682	2682
query10	420	297	307	297
query11	11374	10696	10830	10696
query12	122	80	77	77
query13	5624	731	736	731
query14	17777	13480	13213	13213
query15	396	256	241	241
query16	6476	291	272	272
query17	1730	1453	880	880
query18	2318	425	420	420
query19	209	154	156	154
query20	76	82	88	82
query21	191	108	99	99
query22	5243	5133	5103	5103
query23	32709	32137	32050	32050
query24	7000	6615	6552	6552
query25	547	429	429	429
query26	532	167	166	166
query27	1862	300	296	296
query28	6186	2404	2379	2379
query29	2945	2700	2816	2700
query30	245	165	165	165
query31	883	756	724	724
query32	69	66	61	61
query33	406	270	270	270
query34	852	481	491	481
query35	1155	920	970	920
query36	1458	1181	1115	1115
query37	90	61	59	59
query38	3094	2968	2934	2934
query39	1362	1346	1330	1330
query40	202	97	104	97
query41	46	42	44	42
query42	91	84	79	79
query43	718	689	678	678
query44	1125	727	730	727
query45	248	238	232	232
query46	1247	962	975	962
query47	1780	1817	1707	1707
query48	1043	728	755	728
query49	633	388	381	381
query50	887	622	594	594
query51	4735	4741	4683	4683
query52	93	86	86	86
query53	452	332	328	328
query54	2658	2508	2505	2505
query55	91	83	79	79
query56	239	217	227	217
query57	1206	1147	1229	1147
query58	222	198	206	198
query59	4253	4090	3846	3846
query60	212	209	224	209
query61	105	101	103	101
query62	881	503	470	470
query63	489	343	348	343
query64	2544	1577	1451	1451
query65	3612	3586	3567	3567
query66	808	393	399	393
query67	15488	16873	15464	15464
query68	8269	660	646	646
query69	577	365	352	352
query70	1825	1397	1398	1397
query71	405	319	325	319
query72	6539	3566	3562	3562
query73	742	318	322	318
query74	6265	5907	5852	5852
query75	4693	3703	3734	3703
query76	4552	1163	1233	1163
query77	549	262	270	262
query78	12879	11656	11886	11656
query79	8860	644	648	644
query80	2092	413	412	412
query81	513	240	238	238
query82	1685	102	97	97
query83	175	144	137	137
query84	261	74	73	73
query85	1315	356	342	342
query86	357	295	314	295
query87	3288	3071	3038	3038
query88	5215	2314	2336	2314
query89	389	307	286	286
query90	1814	213	215	213
query91	187	146	156	146
query92	59	53	53	53
query93	5258	605	583	583
query94	848	209	213	209
query95	1104	1096	1075	1075
query96	659	337	328	328
query97	6579	6482	6496	6482
query98	195	177	174	174
query99	2942	986	947	947
Total cold run time: 312736 ms
Total hot run time: 204998 ms

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 31.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aaa5b10de5a546910512d351e13c9c287f29809b, data reload: false
query1	0.02	0.02	0.03
query2	0.07	0.03	0.03
query3	0.24	0.04	0.05
query4	1.80	0.10	0.07
query5	0.54	0.52	0.52
query6	1.27	0.60	0.61
query7	0.01	0.02	0.01
query8	0.03	0.02	0.03
query9	0.54	0.51	0.47
query10	0.53	0.53	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.08
query13	0.63	0.61	0.61
query14	0.77	0.78	0.80
query15	0.79	0.75	0.76
query16	0.36	0.39	0.38
query17	1.00	1.00	1.01
query18	0.22	0.26	0.23
query19	1.92	1.85	1.85
query20	0.02	0.01	0.01
query21	15.43	0.56	0.56
query22	1.90	1.72	2.00
query23	17.02	1.01	1.06
query24	5.05	1.27	3.84
query25	0.38	0.12	0.05
query26	0.86	0.17	0.15
query27	0.05	0.04	0.04
query28	5.02	0.76	0.71
query29	12.58	2.25	2.35
query30	0.58	0.53	0.52
query31	2.81	0.39	0.37
query32	3.36	0.49	0.48
query33	3.09	3.09	3.09
query34	15.27	4.82	4.81
query35	4.88	4.90	4.88
query36	1.05	1.03	1.02
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.15	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 100.72 s
Total hot run time: 31.27 s

@doris-robot

doris-robot commented Aug 13, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit aaa5b10de5a546910512d351e13c9c287f29809b with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.4 seconds inserted 10000000 Rows, about 467K ops/s

@dataroaring
dataroaring merged commit e217aab into apache:branch-2.0Aug 14, 2024
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@luwei16weixingyu12
[fix](compaction) fix mismatch between segment key and value column r…
2a08d5a
…ows during compaction (apache#37960) (apache#38251) (apache#38356) (apache#39263)
pick master apache#37960apache#38251apache#38356
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@luwei16@doris-robot@dataroaring