[fix](row binlog): op should be insert but update when insert after delete by Userwhite · Pull Request #66046 · apache/doris

Userwhite · 2026-07-25T10:04:02Z

What problem does this PR solve?

Issue Number: close#65808

The main reason for the OP error is that the data has already been deleted, but in Doris, deletion may only set the delete sign to 1, so the data can still be queried when retrieving historical data.

Therefore, we need to proactively detect this situation.

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-07-25T10:04:07Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Userwhite · 2026-07-25T10:12:30Z

/review

Userwhite · 2026-07-25T10:24:13Z

run buildall

hello-stephen · 2026-07-25T11:26:25Z

TPC-H: Total hot run time: 29756 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6fde9ee23fb09ce05bd9f7027f54764f2538d8be, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17663	4086	4083	4083
q2	2007	336	194	194
q3	10295	1455	843	843
q4	4679	477	346	346
q5	7511	860	568	568
q6	187	171	141	141
q7	762	854	627	627
q8	9842	1684	1530	1530
q9	6129	4309	4329	4309
q10	6827	1780	1495	1495
q11	500	358	334	334
q12	758	581	450	450
q13	18186	3438	2736	2736
q14	271	256	248	248
q15	q16	797	784	716	716
q17	978	960	994	960
q18	6860	5756	5630	5630
q19	1169	1275	1056	1056
q20	811	681	621	621
q21	5761	2673	2567	2567
q22	427	351	302	302
Total cold run time: 102420 ms
Total hot run time: 29756 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4418	4444	4501	4444
q2	287	317	207	207
q3	4568	4987	4452	4452
q4	2041	2164	1397	1397
q5	4453	4384	4303	4303
q6	221	178	127	127
q7	2008	1972	1673	1673
q8	2538	2148	2115	2115
q9	7870	7907	7734	7734
q10	4722	4624	4235	4235
q11	587	413	390	390
q12	950	763	538	538
q13	3328	3558	3014	3014
q14	299	314	286	286
q15	q16	743	744	657	657
q17	1324	1390	1338	1338
q18	7997	7446	7011	7011
q19	1124	1101	1095	1095
q20	2234	2220	1941	1941
q21	5298	4595	4404	4404
q22	520	463	394	394
Total cold run time: 57530 ms
Total hot run time: 51755 ms

hello-stephen · 2026-07-25T11:37:46Z

TPC-DS: Total hot run time: 177500 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6fde9ee23fb09ce05bd9f7027f54764f2538d8be, data reload: false
query5	4324	652	492	492
query6	470	239	229	229
query7	4870	581	333	333
query8	338	192	194	192
query9	8800	4145	4125	4125
query10	454	378	288	288
query11	5963	2349	2114	2114
query12	165	109	103	103
query13	1272	637	454	454
query14	6218	5242	4900	4900
query14_1	4248	4266	4232	4232
query15	220	211	181	181
query16	1041	469	461	461
query17	1122	723	579	579
query18	2506	463	357	357
query19	210	194	150	150
query20	116	111	107	107
query21	236	162	139	139
query22	13491	13503	13279	13279
query23	17520	16610	16135	16135
query23_1	16309	16313	16329	16313
query24	7466	1745	1279	1279
query24_1	1306	1292	1279	1279
query25	562	464	390	390
query26	1339	349	220	220
query27	2583	579	376	376
query28	4440	1982	1998	1982
query29	1074	618	503	503
query30	344	271	228	228
query31	1153	1084	972	972
query32	102	66	60	60
query33	527	309	248	248
query34	1183	1140	656	656
query35	762	792	667	667
query36	1199	1191	1026	1026
query37	159	113	96	96
query38	1881	1708	1674	1674
query39	899	866	852	852
query39_1	817	836	839	836
query40	257	179	160	160
query41	65	62	62	62
query42	93	98	98	98
query43	329	332	280	280
query44	1413	762	779	762
query45	188	178	172	172
query46	1056	1166	698	698
query47	2165	2135	2032	2032
query48	394	427	290	290
query49	576	418	338	338
query50	1069	442	331	331
query51	10737	10398	10752	10398
query52	86	88	76	76
query53	264	284	199	199
query54	275	233	218	218
query55	73	74	67	67
query56	308	308	291	291
query57	1322	1287	1215	1215
query58	266	262	249	249
query59	1567	1666	1469	1469
query60	309	271	252	252
query61	151	151	154	151
query62	543	498	432	432
query63	239	203	201	201
query64	2796	1103	871	871
query65	4735	4631	4597	4597
query66	1801	513	384	384
query67	29691	29226	29099	29099
query68	3091	1590	1057	1057
query69	412	301	264	264
query70	1066	968	951	951
query71	365	345	302	302
query72	3090	2691	2383	2383
query73	807	755	452	452
query74	5085	4907	4729	4729
query75	2538	2504	2137	2137
query76	2328	1164	774	774
query77	359	387	279	279
query78	11810	11933	11313	11313
query79	1358	1207	745	745
query80	1228	539	455	455
query81	515	338	294	294
query82	589	159	117	117
query83	417	328	299	299
query84	282	162	133	133
query85	982	638	537	537
query86	395	294	274	274
query87	1832	1823	1771	1771
query88	3724	2815	2781	2781
query89	438	377	325	325
query90	1817	204	204	204
query91	202	191	160	160
query92	65	58	54	54
query93	1488	1485	1021	1021
query94	642	361	335	335
query95	805	486	565	486
query96	1056	823	352	352
query97	2645	2667	2492	2492
query98	212	206	199	199
query99	1093	1116	974	974
Total cold run time: 263475 ms
Total hot run time: 177500 ms

hello-stephen · 2026-07-25T11:42:46Z

ClickBench: Total hot run time: 25.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6fde9ee23fb09ce05bd9f7027f54764f2538d8be, data reload: false
query1	0.00	0.00	0.00
query2	0.09	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.13	0.13
query5	0.24	0.22	0.22
query6	1.23	1.06	1.14
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.38	0.30	0.31
query10	0.54	0.55	0.56
query11	0.20	0.14	0.13
query12	0.18	0.14	0.14
query13	0.46	0.48	0.47
query14	1.02	1.00	0.99
query15	0.61	0.61	0.60
query16	0.32	0.31	0.31
query17	1.08	1.08	1.12
query18	0.22	0.22	0.21
query19	2.05	2.01	1.93
query20	0.02	0.02	0.02
query21	15.43	0.22	0.13
query22	4.75	0.05	0.06
query23	16.13	0.31	0.13
query24	2.96	0.42	0.32
query25	0.10	0.04	0.06
query26	0.74	0.20	0.15
query27	0.04	0.04	0.03
query28	3.52	0.86	0.53
query29	12.52	4.12	3.28
query30	0.27	0.16	0.17
query31	2.78	0.61	0.31
query32	3.22	0.58	0.47
query33	3.16	3.23	3.26
query34	15.73	4.23	3.53
query35	3.54	3.52	3.53
query36	0.56	0.44	0.41
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.03	0.03
query40	0.19	0.17	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.56 s
Total hot run time: 25.02 s

hello-stephen · 2026-07-25T13:41:24Z

BE Regression && UT Coverage Report

Increment line coverage 46.03% (29/63) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.29% (31073/41271)
Line Coverage	59.82% (346108/578554)
Region Coverage	56.51% (290607/514255)
Branch Coverage	57.89% (129904/224414)

Userwhite · 2026-07-28T08:47:39Z

run buildall

hello-stephen · 2026-07-28T12:33:28Z

TPC-H: Total hot run time: 29718 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39aa0ac4f1002bfd93c5d33c33bac3215593e2a6, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17598	4108	4035	4035
q2	2073	323	206	206
q3	10281	1478	828	828
q4	4685	476	344	344
q5	7499	855	570	570
q6	184	175	142	142
q7	794	837	625	625
q8	9349	1583	1712	1583
q9	6085	4407	4354	4354
q10	6828	1731	1475	1475
q11	520	355	332	332
q12	746	593	460	460
q13	18090	3439	2805	2805
q14	264	269	247	247
q15	q16	795	778	709	709
q17	957	1047	932	932
q18	6990	5760	5650	5650
q19	1293	1308	1092	1092
q20	844	678	579	579
q21	6010	2667	2455	2455
q22	440	356	295	295
Total cold run time: 102325 ms
Total hot run time: 29718 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4390	4333	4340	4333
q2	281	320	214	214
q3	4577	4982	4395	4395
q4	2068	2159	1376	1376
q5	4383	4242	4291	4242
q6	227	173	127	127
q7	1761	2215	1796	1796
q8	2614	2231	2174	2174
q9	8036	8157	7918	7918
q10	4693	4735	4268	4268
q11	595	422	400	400
q12	744	760	537	537
q13	3295	3593	2981	2981
q14	300	288	273	273
q15	q16	709	717	638	638
q17	1414	1325	1518	1325
q18	7775	7311	7255	7255
q19	1139	1064	1073	1064
q20	2205	2213	1958	1958
q21	5256	4562	4411	4411
q22	533	461	396	396
Total cold run time: 56995 ms
Total hot run time: 52081 ms

hello-stephen · 2026-07-28T12:44:41Z

TPC-DS: Total hot run time: 177639 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39aa0ac4f1002bfd93c5d33c33bac3215593e2a6, data reload: false
query5	4322	629	493	493
query6	470	224	204	204
query7	4853	603	342	342
query8	341	190	167	167
query9	8749	4071	4021	4021
query10	449	365	333	333
query11	5944	2349	2072	2072
query12	164	102	100	100
query13	1271	607	410	410
query14	6247	5199	4902	4902
query14_1	4228	4198	4209	4198
query15	217	207	179	179
query16	1037	480	475	475
query17	1127	720	631	631
query18	2713	515	360	360
query19	221	192	158	158
query20	122	108	107	107
query21	235	157	135	135
query22	13567	13521	13371	13371
query23	17393	16516	16144	16144
query23_1	16154	16257	16318	16257
query24	7465	1776	1297	1297
query24_1	1305	1298	1302	1298
query25	600	476	397	397
query26	1359	346	213	213
query27	2567	617	387	387
query28	4421	1996	2002	1996
query29	1072	631	493	493
query30	345	273	232	232
query31	1111	1104	995	995
query32	110	63	63	63
query33	529	326	264	264
query34	1163	1139	626	626
query35	774	781	675	675
query36	1029	1029	863	863
query37	164	106	95	95
query38	1938	1695	1643	1643
query39	865	869	854	854
query39_1	852	842	873	842
query40	249	177	141	141
query41	73	67	66	66
query42	94	93	93	93
query43	315	332	276	276
query44	1408	762	758	758
query45	195	177	170	170
query46	1097	1159	724	724
query47	2164	2107	1992	1992
query48	424	421	299	299
query49	583	424	312	312
query50	1074	446	339	339
query51	10767	10535	10858	10535
query52	86	87	74	74
query53	259	280	202	202
query54	285	228	216	216
query55	81	74	68	68
query56	304	318	288	288
query57	1336	1317	1204	1204
query58	293	263	253	253
query59	1637	1626	1463	1463
query60	299	271	254	254
query61	150	156	151	151
query62	555	488	432	432
query63	246	205	203	203
query64	2779	1049	862	862
query65	4750	4631	4625	4625
query66	1811	492	385	385
query67	29270	29118	29091	29091
query68	3292	1499	1040	1040
query69	404	306	263	263
query70	907	801	834	801
query71	354	334	330	330
query72	3276	2730	2373	2373
query73	836	806	391	391
query74	5078	4871	4730	4730
query75	2559	2491	2121	2121
query76	2301	1187	749	749
query77	341	380	291	291
query78	11890	11824	11409	11409
query79	1381	1167	776	776
query80	1312	551	485	485
query81	531	327	288	288
query82	960	159	120	120
query83	367	327	300	300
query84	293	159	131	131
query85	1005	623	537	537
query86	422	240	233	233
query87	1862	1833	1769	1769
query88	3711	2806	2804	2804
query89	446	391	331	331
query90	1932	197	197	197
query91	204	191	165	165
query92	63	60	54	54
query93	1704	1576	965	965
query94	728	358	323	323
query95	803	508	479	479
query96	1025	790	339	339
query97	2631	2631	2535	2535
query98	206	216	210	210
query99	1085	1111	971	971
Total cold run time: 264391 ms
Total hot run time: 177639 ms

hello-stephen · 2026-07-28T12:49:41Z

ClickBench: Total hot run time: 25.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39aa0ac4f1002bfd93c5d33c33bac3215593e2a6, data reload: false
query1	0.00	0.00	0.01
query2	0.10	0.05	0.05
query3	0.25	0.14	0.14
query4	1.61	0.14	0.13
query5	0.24	0.22	0.23
query6	1.22	1.06	1.13
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.39	0.35	0.30
query10	0.55	0.59	0.54
query11	0.20	0.14	0.14
query12	0.19	0.14	0.14
query13	0.47	0.47	0.48
query14	1.00	1.00	1.01
query15	0.62	0.60	0.61
query16	0.32	0.32	0.32
query17	1.13	1.07	1.18
query18	0.23	0.22	0.21
query19	2.07	1.96	1.89
query20	0.01	0.01	0.02
query21	15.44	0.19	0.13
query22	5.01	0.06	0.05
query23	16.11	0.31	0.13
query24	2.96	0.40	0.34
query25	0.11	0.08	0.04
query26	0.74	0.21	0.15
query27	0.04	0.03	0.03
query28	3.55	0.92	0.56
query29	12.46	4.08	3.31
query30	0.28	0.15	0.15
query31	2.77	0.60	0.31
query32	3.22	0.59	0.48
query33	3.11	3.26	3.19
query34	15.45	4.25	3.53
query35	3.53	3.57	3.54
query36	0.56	0.43	0.41
query37	0.09	0.07	0.07
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.20	0.17	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.58 s
Total hot run time: 25.05 s

hello-stephen · 2026-07-28T13:45:37Z

BE Regression && UT Coverage Report

Increment line coverage 76.92% (50/65) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.32% (31151/41359)
Line Coverage	59.85% (347211/580098)
Region Coverage	56.56% (291772/515874)
Branch Coverage	57.90% (130610/225588)

github-actions · 2026-07-30T03:04:16Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-07-30T03:04:18Z

PR approved by anyone and no changes requested.

gavinchou · 2026-07-31T02:31:46Z

/review

github-actions · 2026-07-31T02:52:50Z

Codex automated review failed and did not complete.

Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Aug 5th, 2026 4:32 AM.
Workflow run: https://github.com/apache/doris/actions/runs/30599211460

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

yujun777

LGTM

github-actions · 2026-07-31T03:07:52Z

PR approved by at least one committer and no changes requested.

hello-stephen · 2026-07-31T03:52:25Z

skip buildall

…elete (apache#66046) ### What problem does this PR solve? Issue Number: closeapache#65808 The main reason for the OP error is that the data has already been deleted, but in Doris, deletion may only set the delete sign to 1, so the data can still be queried when retrieving historical data. Therefore, we need to proactively detect this situation. --------- Co-authored-by: Gavin Chou <gavineaglechou@gmail.com>

…RowBinlogSegmentWriter Last PR of the block-transform-chain stack. RowBinlogDeriveStage rebuilds the load block into a full-width block over the binlog schema -- key + AFTER values, optional __BEFORE__* values, and the TSO/LSN/op columns -- so the ordinary segment writers write it like any other block. build_transform_chain picks PlainRowBinlogDerive (no history probe) or MowRowBinlogDerive (partial update and/or BEFORE image) per flush, and keeps an empty chain for every non-direct write type, whose rows are already binlog shaped. RowBinlogSegmentWriter is deleted, and with it the dead VerticalRowBinlogRowsetWriter template, the retriever's clear() bridge and SegmentWriter::build_key_index's protected visibility. SegmentFlusher no longer special-cases binlog when it picks a writer. Three later fixes to the deleted writer are carried into the stage, since deleting the file would drop them silently: - apache#66046: the op revision from the old delete sign runs between the BEFORE fill and the op column fill, the order the writer used. - apache#65810: the cloud-mode rejection is gone, and the historical lookup uses source.base_tablet. The writer fell back to its own tablet when that was unset; the stage requires it instead, because on a binlog write the writer context's tablet is the binlog tablet, which holds no source rows. Every site that fills source.tablet_schema fills source.base_tablet too. - apache#66553: AFTER value columns are wrapped nullable before they land in the binlog block, which declares them nullable. Behavior change to ack: binlog segments are written by the ordinary segment writer, which is the vertical one under the default enable_vertical_segment_writer=true; the deleted writer was always horizontal. The derived rows are identical -- the plain, MoW and BEFORE binlog goldens recorded through the vertical writer are byte-identical to the baselines recorded from the legacy writer -- but the writer type and the physical page layout are visible to operational tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix insert after delete
283c50b

Userwhite requested review from airborne12, csun5285, eldenmoon, gavinchou, liaoxin01 and yiguolei as code owners July 25, 2026 10:04

add min-delta test
6fde9ee

add historical_row_retriever test
39aa0ac

gavinchou previously approved these changes Jul 30, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2026

github-actionsBot added the reviewed label Jul 30, 2026

Fix formatting in _fill_old_delete_signs method
c2b12a5

gavinchou dismissed their stale review via c2b12a5July 31, 2026 02:53

Fix formatting in old_delete_signs assertion
caee36d

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jul 31, 2026

yujun777 approved these changes Jul 31, 2026
View reviewed changes

gavinchou approved these changes Jul 31, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 31, 2026

hello-stephen merged commit 977eaa7 into apache:masterJul 31, 2026
30 checks passed

gavinchou added the binlog label Jul 31, 2026

HappenLee mentioned this pull request Aug 11, 2026
[feature](query) Generate TopN filters from aggregation #66652
Open

morningman mentioned this pull request Sep 1, 2026
[Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418
Open
69 tasks

Uh oh!

Conversation

Userwhite commented Jul 25, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 25, 2026

Uh oh!

Userwhite commented Jul 25, 2026

Uh oh!

Userwhite commented Jul 25, 2026

Uh oh!

hello-stephen commented Jul 25, 2026

Uh oh!

hello-stephen commented Jul 25, 2026

Uh oh!

hello-stephen commented Jul 25, 2026

Uh oh!

hello-stephen commented Jul 25, 2026

BE Regression && UT Coverage Report

Uh oh!

Userwhite commented Jul 28, 2026

Uh oh!

hello-stephen commented Jul 28, 2026

Uh oh!

hello-stephen commented Jul 28, 2026

Uh oh!

hello-stephen commented Jul 28, 2026

Uh oh!

hello-stephen commented Jul 28, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Jul 30, 2026

Uh oh!

github-actionsBot commented Jul 30, 2026

Uh oh!

gavinchou commented Jul 31, 2026

Uh oh!

github-actionsBot commented Jul 31, 2026

Uh oh!

yujun777 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 31, 2026

Uh oh!

hello-stephen commented Jul 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Userwhite commented Jul 25, 2026•
edited
Loading