[Fix](partial update) Fix incorrect result when partial update include delete sign columns by bobhan1 · Pull Request #46194 · apache/doris

bobhan1 · 2024-12-31T06:49:36Z

What problem does this PR solve?

Problem Summary:

Currently, when a partial update load include delete sign columns, it will not read delete sign columns from historical data. This may result in incorrect result because it may read data from rows whose delete sign is 1.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-31T06:49:41Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2024-12-31T06:52:12Z

run buildall

doris-robot · 2024-12-31T07:48:35Z

TPC-H: Total hot run time: 33579 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 502ee573a1efc61c383465f7cf93462aef4b0629, data reload: false
------ Round 1 ----------------------------------
q1	17579	6341	6211	6211
q2	2039	322	168	168
q3	10406	1384	784	784
q4	10228	944	440	440
q5	7927	2395	2210	2210
q6	234	190	149	149
q7	982	823	610	610
q8	9243	1544	1378	1378
q9	5512	5105	5210	5105
q10	6794	2387	1885	1885
q11	492	275	258	258
q12	361	394	227	227
q13	17776	3677	2927	2927
q14	237	231	211	211
q15	593	517	516	516
q16	637	619	588	588
q17	598	912	330	330
q18	7109	6513	6455	6455
q19	2493	1126	598	598
q20	311	340	188	188
q21	3106	2327	2028	2028
q22	373	356	313	313
Total cold run time: 105030 ms
Total hot run time: 33579 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6700	6521	6585	6521
q2	257	347	229	229
q3	2356	2773	2328	2328
q4	1467	1900	1374	1374
q5	4453	5022	5057	5022
q6	215	197	146	146
q7	2171	1997	1833	1833
q8	2851	3021	2902	2902
q9	7487	7449	7442	7442
q10	3105	3360	2935	2935
q11	656	526	502	502
q12	686	827	668	668
q13	3488	3850	3132	3132
q14	301	312	279	279
q15	607	524	530	524
q16	707	717	627	627
q17	1311	1925	1268	1268
q18	7759	7475	7376	7376
q19	912	1269	1368	1269
q20	2032	2079	1868	1868
q21	5949	5247	5144	5144
q22	669	633	585	585
Total cold run time: 56139 ms
Total hot run time: 53974 ms

doris-robot · 2024-12-31T08:00:59Z

TPC-DS: Total hot run time: 196779 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 502ee573a1efc61c383465f7cf93462aef4b0629, data reload: false
query1	1317	989	930	930
query2	6419	2292	2250	2250
query3	11018	4604	4813	4604
query4	33040	23692	23459	23459
query5	3692	613	452	452
query6	263	196	195	195
query7	3975	492	306	306
query8	298	231	226	226
query9	9367	2643	2645	2643
query10	463	319	259	259
query11	18172	15709	15400	15400
query12	162	109	103	103
query13	1580	532	402	402
query14	10918	7764	7278	7278
query15	238	213	188	188
query16	7943	661	498	498
query17	1571	831	615	615
query18	2050	398	318	318
query19	229	194	167	167
query20	123	116	116	116
query21	206	126	112	112
query22	4742	4559	4703	4559
query23	34996	34187	33955	33955
query24	6466	2366	2268	2268
query25	467	455	398	398
query26	785	281	156	156
query27	2120	461	331	331
query28	5471	2466	2423	2423
query29	570	566	423	423
query30	215	186	159	159
query31	992	979	875	875
query32	68	62	57	57
query33	477	353	301	301
query34	759	856	504	504
query35	848	839	749	749
query36	1038	1033	993	993
query37	119	107	78	78
query38	4252	4355	4509	4355
query39	1515	1472	1558	1472
query40	207	114	99	99
query41	48	43	46	43
query42	117	102	107	102
query43	518	538	505	505
query44	1308	821	852	821
query45	186	181	170	170
query46	884	1071	657	657
query47	2026	2080	1961	1961
query48	379	403	339	339
query49	731	477	401	401
query50	624	716	401	401
query51	7313	7106	7053	7053
query52	107	104	96	96
query53	232	264	213	213
query54	506	521	408	408
query55	81	81	80	80
query56	284	302	260	260
query57	1290	1248	1198	1198
query58	236	225	235	225
query59	3290	3278	3334	3278
query60	285	262	265	262
query61	107	113	119	113
query62	897	819	762	762
query63	225	190	192	190
query64	3064	1066	732	732
query65	3325	3254	3271	3254
query66	787	424	299	299
query67	16585	15805	15653	15653
query68	9099	759	521	521
query69	486	311	267	267
query70	1265	1148	1152	1148
query71	421	285	247	247
query72	6270	2770	3954	2770
query73	654	740	353	353
query74	10748	9035	8860	8860
query75	4694	3165	2685	2685
query76	5319	1188	805	805
query77	961	365	285	285
query78	10288	10173	9771	9771
query79	4073	902	585	585
query80	709	520	430	430
query81	474	273	230	230
query82	228	152	122	122
query83	195	164	141	141
query84	280	86	70	70
query85	750	383	304	304
query86	356	307	306	306
query87	4719	4467	4281	4281
query88	3589	2148	2156	2148
query89	425	316	285	285
query90	2141	220	185	185
query91	133	136	107	107
query92	64	56	54	54
query93	1980	827	528	528
query94	660	394	283	283
query95	346	267	258	258
query96	477	605	280	280
query97	2733	2852	2664	2664
query98	231	201	192	192
query99	1648	1590	1462	1462
Total cold run time: 301002 ms
Total hot run time: 196779 ms

doris-robot · 2024-12-31T08:06:21Z

ClickBench: Total hot run time: 32.03 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 502ee573a1efc61c383465f7cf93462aef4b0629, data reload: false
query1	0.03	0.04	0.04
query2	0.07	0.03	0.04
query3	0.24	0.06	0.07
query4	1.61	0.10	0.11
query5	0.41	0.40	0.41
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.50
query10	0.56	0.55	0.55
query11	0.15	0.12	0.11
query12	0.13	0.11	0.10
query13	0.60	0.62	0.60
query14	2.83	2.85	2.76
query15	0.89	0.85	0.82
query16	0.39	0.38	0.39
query17	1.00	1.05	1.05
query18	0.23	0.20	0.22
query19	1.98	1.80	1.98
query20	0.01	0.01	0.01
query21	15.37	0.92	0.57
query22	0.77	0.77	0.73
query23	15.25	1.49	0.59
query24	3.05	1.67	1.92
query25	0.23	0.14	0.05
query26	0.22	0.14	0.13
query27	0.07	0.05	0.05
query28	14.90	1.46	1.05
query29	12.58	3.91	3.21
query30	0.25	0.08	0.06
query31	2.83	0.62	0.39
query32	3.23	0.53	0.46
query33	3.08	3.08	3.11
query34	16.75	5.21	4.52
query35	4.46	4.50	4.46
query36	0.63	0.49	0.48
query37	0.10	0.07	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.1 s
Total hot run time: 32.03 s

doris-robot · 2024-12-31T08:09:56Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10121/26024)
Line Coverage: 29.89% (85532/286164)
Region Coverage: 29.02% (43717/150667)
Branch Coverage: 25.54% (22296/87296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/502ee573a1efc61c383465f7cf93462aef4b0629_502ee573a1efc61c383465f7cf93462aef4b0629/report/index.html

bobhan1 · 2024-12-31T11:00:02Z

run buildall

bobhan1 · 2024-12-31T11:01:24Z

run buildall

doris-robot · 2024-12-31T11:20:39Z

TPC-H: Total hot run time: 32492 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f35cad9e25aa9f29f5326faa81764c90cbd7bae8, data reload: false
------ Round 1 ----------------------------------
q1	17562	6211	6105	6105
q2	2051	296	177	177
q3	10417	1270	711	711
q4	10209	855	427	427
q5	7546	2204	1926	1926
q6	205	178	145	145
q7	898	756	613	613
q8	9245	1371	1191	1191
q9	5131	4878	4887	4878
q10	6758	2300	1848	1848
q11	475	273	258	258
q12	345	364	216	216
q13	17764	3631	2942	2942
q14	249	230	219	219
q15	560	508	491	491
q16	634	625	577	577
q17	576	847	335	335
q18	7102	6512	6386	6386
q19	4135	956	552	552
q20	295	314	183	183
q21	2732	2152	2009	2009
q22	359	323	303	303
Total cold run time: 105248 ms
Total hot run time: 32492 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6388	6269	6609	6269
q2	242	339	232	232
q3	2292	2640	2335	2335
q4	1427	1810	1385	1385
q5	4346	4735	4989	4735
q6	193	178	146	146
q7	2105	2003	1775	1775
q8	2593	2773	2689	2689
q9	7327	7270	7220	7220
q10	3068	3338	2881	2881
q11	592	527	500	500
q12	662	778	627	627
q13	3399	3700	3093	3093
q14	283	301	283	283
q15	561	512	490	490
q16	668	701	632	632
q17	1213	1741	1250	1250
q18	7701	7345	7141	7141
q19	755	1055	1033	1033
q20	1914	1996	1852	1852
q21	5500	5100	4931	4931
q22	589	645	554	554
Total cold run time: 53818 ms
Total hot run time: 52053 ms

doris-robot · 2024-12-31T11:32:47Z

TPC-DS: Total hot run time: 190883 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f35cad9e25aa9f29f5326faa81764c90cbd7bae8, data reload: false
query1	979	386	396	386
query2	6515	2444	2344	2344
query3	6713	215	211	211
query4	34226	23868	24020	23868
query5	4407	630	459	459
query6	282	206	192	192
query7	4642	503	303	303
query8	305	266	243	243
query9	9332	2642	2620	2620
query10	474	309	267	267
query11	18458	15668	15401	15401
query12	161	111	108	108
query13	1666	547	410	410
query14	10788	6680	6852	6680
query15	243	201	200	200
query16	8198	623	409	409
query17	1567	749	570	570
query18	2124	420	351	351
query19	214	171	158	158
query20	129	116	121	116
query21	202	127	107	107
query22	4182	4159	4359	4159
query23	34725	33382	33035	33035
query24	6512	2284	2336	2284
query25	493	446	383	383
query26	1224	276	154	154
query27	2443	461	344	344
query28	5522	2415	2396	2396
query29	748	552	421	421
query30	227	184	163	163
query31	966	925	824	824
query32	79	60	60	60
query33	552	347	305	305
query34	746	856	519	519
query35	812	794	732	732
query36	1019	1050	942	942
query37	111	96	78	78
query38	4070	4292	4098	4098
query39	1497	1461	1432	1432
query40	209	114	100	100
query41	49	45	47	45
query42	116	106	104	104
query43	498	535	495	495
query44	1336	803	796	796
query45	174	170	178	170
query46	844	1037	645	645
query47	1912	1921	1837	1837
query48	393	417	305	305
query49	772	488	380	380
query50	614	649	371	371
query51	7277	7311	7264	7264
query52	101	100	89	89
query53	222	249	186	186
query54	474	479	395	395
query55	82	84	79	79
query56	261	258	258	258
query57	1252	1223	1167	1167
query58	238	235	232	232
query59	3166	3122	3014	3014
query60	282	254	254	254
query61	111	109	107	107
query62	878	789	734	734
query63	229	188	190	188
query64	4478	991	649	649
query65	3422	3286	3246	3246
query66	1041	431	313	313
query67	15950	15856	15547	15547
query68	8609	779	519	519
query69	462	292	254	254
query70	1264	1103	1128	1103
query71	451	273	250	250
query72	5849	3979	3894	3894
query73	654	758	363	363
query74	10354	8921	8860	8860
query75	4673	3160	2684	2684
query76	4356	1240	775	775
query77	830	359	266	266
query78	10602	10506	9997	9997
query79	4245	888	594	594
query80	725	535	445	445
query81	517	281	246	246
query82	611	156	119	119
query83	199	174	157	157
query84	281	89	74	74
query85	803	365	309	309
query86	352	312	286	286
query87	4585	4449	4629	4449
query88	3904	2161	2138	2138
query89	425	340	308	308
query90	1930	189	189	189
query91	139	138	108	108
query92	70	65	55	55
query93	1722	886	529	529
query94	666	405	288	288
query95	332	257	258	257
query96	473	613	281	281
query97	2811	2868	2727	2727
query98	228	205	196	196
query99	1677	1568	1461	1461
Total cold run time: 298733 ms
Total hot run time: 190883 ms

doris-robot · 2024-12-31T11:38:08Z

ClickBench: Total hot run time: 31.25 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f35cad9e25aa9f29f5326faa81764c90cbd7bae8, data reload: false
query1	0.03	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.42	0.41	0.42
query6	1.16	0.64	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.51	0.49
query10	0.54	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.61	0.60
query14	2.86	2.75	2.84
query15	0.90	0.82	0.83
query16	0.39	0.39	0.38
query17	1.08	1.00	1.06
query18	0.23	0.22	0.21
query19	1.91	1.78	2.03
query20	0.02	0.00	0.01
query21	15.35	0.96	0.58
query22	0.76	0.67	0.57
query23	15.54	1.47	0.50
query24	3.02	1.67	1.09
query25	0.19	0.06	0.16
query26	0.24	0.14	0.15
query27	0.07	0.04	0.05
query28	14.17	1.49	1.05
query29	12.61	3.91	3.27
query30	0.25	0.09	0.06
query31	2.83	0.57	0.40
query32	3.25	0.53	0.47
query33	3.10	3.09	3.22
query34	16.90	5.08	4.58
query35	4.47	4.46	4.42
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.92 s
Total hot run time: 31.25 s

doris-robot · 2024-12-31T11:42:47Z

TeamCity be ut coverage result:
Function Coverage: 38.89% (10122/26024)
Line Coverage: 29.88% (85504/286167)
Region Coverage: 29.02% (43721/150671)
Branch Coverage: 25.54% (22297/87298)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f35cad9e25aa9f29f5326faa81764c90cbd7bae8_f35cad9e25aa9f29f5326faa81764c90cbd7bae8/report/index.html

bobhan1 · 2024-12-31T13:25:46Z

run p0

zhannngchen

LGTM

github-actions · 2025-01-02T02:31:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-02T02:31:48Z

PR approved by anyone and no changes requested.

dataroaring

LGTM

…e delete sign columns (apache#46194) Currently, when a partial update load include delete sign columns, it will not read delete sign columns from historical data. This may result in incorrect result because it may read data from rows whose delete sign is 1.

…ial update include delete sign columns #46194 (#46336) pick #46194

…pdate include delete sign columns #46194 (#46543) pick #46194

…e delete sign columns apache#46194 (apache#3742) pick apache#46194

bobhan1 added 3 commits December 31, 2024 14:14

add case
40a8b96

update
a8161e8

correct data
502ee57

fix existing cases
628596b

add publish case
f35cad9

bobhan1 force-pushed the fix-partial-update-with-delete-sign-col branch from 41c458f to f35cad9Compare December 31, 2024 11:00

zhannngchen approved these changes Jan 2, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 2, 2025

github-actionsBot added the reviewed label Jan 2, 2025

dataroaring added dev/3.0.x dev/2.1.x labels Jan 2, 2025

dataroaring approved these changes Jan 2, 2025
View reviewed changes

dataroaring merged commit 77beb58 into apache:masterJan 2, 2025

github-actionsBot added dev/3.0.x-conflict dev/2.1.x-conflict labels Jan 2, 2025

bobhan1 mentioned this pull request Jan 3, 2025
branch-3.0-pick: [Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194 #46336
Merged

dataroaring added the p0_w label Jan 7, 2025

dataroaring pushed a commit that referenced this pull request Jan 7, 2025
branch-3.0-pick: [Fix](partial update) Fix incorrect result when part…
aef3b54
…ial update include delete sign columns #46194 (#46336) pick #46194

dataroaring added dev/3.0.4-merged and removed dev/3.0.x labels Jan 7, 2025

bobhan1 mentioned this pull request Jan 7, 2025
branch-2.1: [Fix](partial update) Fix incorrect result when partial update include delete sign columns #46194 #46543
Merged

dataroaring removed the dev/3.0.x-conflict label Jan 7, 2025

yiguolei pushed a commit that referenced this pull request Jan 9, 2025
branch-2.1: [Fix](partial update) Fix incorrect result when partial u…
0ecb84c
…pdate include delete sign columns #46194 (#46543) pick #46194

yiguolei added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Jan 9, 2025

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

hubgeter pushed a commit to hubgeter/doris that referenced this pull request Mar 12, 2025
[Fix](partial update) Fix incorrect result when partial update includ…
45e9719
…e delete sign columns apache#46194 (apache#3742) pick apache#46194

Uh oh!

Conversation

bobhan1 commented Dec 31, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 31, 2024

Uh oh!

bobhan1 commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

bobhan1 commented Dec 31, 2024

Uh oh!

bobhan1 commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

bobhan1 commented Dec 31, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

github-actionsBot commented Jan 2, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

bobhan1 commented Dec 31, 2024•
edited
Loading