[opt](iceberg) able to ignore iceberg dangling delete to do count push down by morningman · Pull Request #58889 · apache/doris

morningman · 2025-12-10T04:08:11Z

What problem does this PR solve?

Related PR: #44039

Problem Summary:

In #44039, we disable the count push down to iceberg table if it has position delete file,
because there may be dangling delete file that will cause the statistics info incorrect.

This PR add a new session variable ignore_iceberg_dangling_delete, default is false,
when set to true, Doris will ignore the dangling delete issue and do count push down for iceberg table,
add user should take care of the result.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…h down

Thearas · 2025-12-10T04:08:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morningman · 2025-12-10T04:59:32Z

run buildall

doris-robot · 2025-12-10T05:36:57Z

TPC-H: Total hot run time: 35634 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
------ Round 1 ----------------------------------
q1	17615	4256	4070	4070
q2	2027	366	240	240
q3	10156	1329	760	760
q4	10265	798	312	312
q5	7561	2158	1945	1945
q6	186	172	142	142
q7	1016	850	715	715
q8	9353	1455	1120	1120
q9	7031	5326	5352	5326
q10	6777	2392	1997	1997
q11	527	321	294	294
q12	684	742	576	576
q13	17768	3739	3072	3072
q14	294	302	270	270
q15	586	523	508	508
q16	941	931	866	866
q17	735	774	561	561
q18	7914	7258	7139	7139
q19	1123	971	620	620
q20	413	367	252	252
q21	4313	3951	3862	3862
q22	1037	992	987	987
Total cold run time: 108322 ms
Total hot run time: 35634 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4112	4046	4060	4046
q2	329	426	307	307
q3	2148	2715	2323	2323
q4	1315	1722	1299	1299
q5	4249	4849	4739	4739
q6	229	180	130	130
q7	2054	1972	1822	1822
q8	2648	2584	2514	2514
q9	7943	7499	7671	7499
q10	3026	3245	2819	2819
q11	607	507	493	493
q12	703	734	623	623
q13	3584	3898	3331	3331
q14	288	293	295	293
q15	581	514	509	509
q16	892	915	936	915
q17	1323	1415	1498	1415
q18	7894	7583	7632	7583
q19	870	855	925	855
q20	2051	2122	1955	1955
q21	4678	4191	4139	4139
q22	1091	1047	1011	1011
Total cold run time: 52615 ms
Total hot run time: 50620 ms

doris-robot · 2025-12-10T05:48:13Z

TPC-DS: Total hot run time: 181181 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
query5	5340	651	482	482
query6	324	226	210	210
query7	4232	465	274	274
query8	303	252	239	239
query9	8778	2572	2591	2572
query10	540	373	340	340
query11	15466	15259	14605	14605
query12	179	120	116	116
query13	1277	488	399	399
query14	6239	3234	2912	2912
query14_1	2877	2856	2829	2829
query15	214	208	184	184
query16	805	450	473	450
query17	1094	730	572	572
query18	2542	435	340	340
query19	236	232	215	215
query20	123	114	114	114
query21	224	141	115	115
query22	4179	4239	4065	4065
query23	16569	15950	15964	15950
query23_1	15977	16046	16020	16020
query24	7423	1681	1232	1232
query24_1	1221	1251	1215	1215
query25	558	468	414	414
query26	1263	269	157	157
query27	2782	453	316	316
query28	4486	2159	2150	2150
query29	818	545	434	434
query30	309	239	217	217
query31	812	686	638	638
query32	77	68	67	67
query33	539	322	291	291
query34	915	913	543	543
query35	802	804	732	732
query36	863	908	831	831
query37	128	90	79	79
query38	3926	3809	3754	3754
query39	749	750	708	708
query39_1	704	697	705	697
query40	229	144	126	126
query41	73	70	68	68
query42	108	106	108	106
query43	430	429	404	404
query44	1321	762	778	762
query45	194	192	187	187
query46	879	994	633	633
query47	1732	1716	1634	1634
query48	336	336	268	268
query49	659	456	370	370
query50	656	310	226	226
query51	3807	3871	3786	3786
query52	110	115	103	103
query53	321	362	300	300
query54	312	286	282	282
query55	77	77	71	71
query56	311	313	302	302
query57	1163	1152	1077	1077
query58	292	272	259	259
query59	2417	2582	2390	2390
query60	345	327	317	317
query61	188	190	189	189
query62	695	663	616	616
query63	331	298	300	298
query64	5133	1442	1128	1128
query65	4018	3925	3957	3925
query66	1444	468	334	334
query67	15369	15179	14944	14944
query68	7888	1028	752	752
query69	499	354	318	318
query70	1078	1052	1017	1017
query71	374	310	293	293
query72	6067	4972	5143	4972
query73	654	584	304	304
query74	8845	8708	8599	8599
query75	3549	3501	3132	3132
query76	3887	1144	753	753
query77	520	413	290	290
query78	9539	9861	8832	8832
query79	1417	881	600	600
query80	735	657	569	569
query81	524	276	236	236
query82	208	128	103	103
query83	270	262	241	241
query84	257	120	97	97
query85	905	517	472	472
query86	376	295	278	278
query87	4093	3984	3918	3918
query88	3177	2289	2300	2289
query89	477	441	396	396
query90	2241	162	148	148
query91	174	168	146	146
query92	86	70	63	63
query93	1224	925	574	574
query94	488	308	278	278
query95	588	339	370	339
query96	605	468	206	206
query97	2600	2647	2605	2605
query98	218	195	198	195
query99	1271	1288	1245	1245
Total cold run time: 263486 ms
Total hot run time: 181181 ms

doris-robot · 2025-12-10T05:53:17Z

ClickBench: Total hot run time: 27.47 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a6bb8511e5260b043c23444ca19867b87f91951e, data reload: false
query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.62	0.11	0.11
query5	0.28	0.26	0.26
query6	1.16	0.65	0.64
query7	0.03	0.03	0.03
query8	0.05	0.04	0.05
query9	0.57	0.51	0.52
query10	0.55	0.55	0.55
query11	0.16	0.10	0.12
query12	0.15	0.11	0.13
query13	0.62	0.61	0.61
query14	1.00	0.99	0.98
query15	0.81	0.81	0.82
query16	0.42	0.40	0.41
query17	1.09	1.07	1.02
query18	0.23	0.21	0.21
query19	1.86	1.90	1.82
query20	0.02	0.01	0.02
query21	15.43	0.29	0.14
query22	4.67	0.05	0.05
query23	16.03	0.28	0.10
query24	2.26	0.59	0.27
query25	0.08	0.08	0.08
query26	0.14	0.13	0.14
query27	0.07	0.06	0.05
query28	3.78	1.23	1.02
query29	12.65	4.06	3.34
query30	0.28	0.15	0.16
query31	2.82	0.63	0.39
query32	3.24	0.56	0.46
query33	3.02	2.98	2.98
query34	16.86	5.18	4.50
query35	4.61	4.61	4.58
query36	0.65	0.51	0.50
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 98.18 s
Total hot run time: 27.47 s

hello-stephen · 2025-12-10T07:14:07Z

FE UT Coverage Report

Increment line coverage 12.50% (1/8) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-12-10T08:53:58Z

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-12-10T14:52:16Z

FE Regression Coverage Report

Increment line coverage 100.00% (8/8) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-12-15T02:37:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-15T02:37:39Z

PR approved by anyone and no changes requested.

…h down (apache#58889) Related PR: apache#44039 Problem Summary: In apache#44039, we disable the count push down to iceberg table if it has position delete file, because there may be dangling delete file that will cause the statistics info incorrect. This PR add a new session variable `ignore_iceberg_dangling_delete`, default is false, when set to true, Doris will ignore the dangling delete issue and do count push down for iceberg table, add user should take care of the result.

…do count push down #58889 (#59069) bp #58889

…h down (apache#58889) Related PR: apache#44039 Problem Summary: In apache#44039, we disable the count push down to iceberg table if it has position delete file, because there may be dangling delete file that will cause the statistics info incorrect. This PR add a new session variable `ignore_iceberg_dangling_delete`, default is false, when set to true, Doris will ignore the dangling delete issue and do count push down for iceberg table, add user should take care of the result.

…do count push down (#58889) (#59068) bp #58889

[opt](iceberg) able to ignore iceberg dangling delete to do count pus…
7fd2cf6
…h down

morningman added 2 commits December 10, 2025 12:29

2
aa7c17b

3
a6bb851

morningman added dev/3.0.x dev/3.1.x dev/4.0.x labels Dec 10, 2025

CalvinKirs approved these changes Dec 15, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 15, 2025

github-actionsBot added the reviewed label Dec 15, 2025

suxiaogang223 approved these changes Dec 15, 2025
View reviewed changes

morningman merged commit 37ec44d into apache:masterDec 15, 2025
29 of 31 checks passed

github-actionsBot added dev/3.1.x-conflict dev/4.0.x-conflict labels Dec 15, 2025

morningman mentioned this pull request Dec 16, 2025
branch-4.0: [opt](iceberg) able to ignore iceberg dangling delete to do count push down (#58889) #59068
Merged

morningman mentioned this pull request Dec 16, 2025
branch-3.1: [opt](iceberg) able to ignore iceberg dangling delete to do count push down #58889 #59069
Merged

morrySnow pushed a commit that referenced this pull request Dec 17, 2025
branch-3.1: [opt](iceberg) able to ignore iceberg dangling delete to …
5ac3928
…do count push down #58889 (#59069) bp #58889

morrySnow removed dev/3.1.x dev/3.1.x-conflict labels Dec 17, 2025

morrySnow added the dev/3.1.4-merged label Dec 17, 2025

morningman added a commit that referenced this pull request Dec 23, 2025
branch-4.0: [opt](iceberg) able to ignore iceberg dangling delete to …
97fa3b1
…do count push down (#58889) (#59068) bp #58889

morningman added dev/4.0.3-merged and removed dev/4.0.x dev/4.0.x-conflict labels Dec 23, 2025

Navigation Menu

Uh oh!

[opt](iceberg) able to ignore iceberg dangling delete to do count push down - #58889

[opt](iceberg) able to ignore iceberg dangling delete to do count push down#58889
morningman merged 3 commits into
apache:masterfrom
morningman:ignore_dangling_delete

morningman commented Dec 10, 2025•
edited
Loading

Uh oh!

Thearas commented Dec 10, 2025

Uh oh!

morningman commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

hello-stephen commented Dec 10, 2025

Uh oh!

hello-stephen commented Dec 10, 2025

Uh oh!

hello-stephen commented Dec 10, 2025

Uh oh!

github-actionsBot commented Dec 15, 2025

Uh oh!

github-actionsBot commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

morningman commented Dec 10, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 10, 2025

Uh oh!

morningman commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

doris-robot commented Dec 10, 2025

Uh oh!

hello-stephen commented Dec 10, 2025

FE UT Coverage Report

Uh oh!

hello-stephen commented Dec 10, 2025

FE Regression Coverage Report

Uh oh!

hello-stephen commented Dec 10, 2025

FE Regression Coverage Report

Uh oh!

github-actionsBot commented Dec 15, 2025

Uh oh!

github-actionsBot commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

morningman commented Dec 10, 2025•
edited
Loading