[fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error. by hubgeter · Pull Request #60262 · apache/doris

hubgeter · 2026-01-27T08:01:18Z

What problem does this PR solve?

Problem Summary:

fix[fix](parquet)fix parquet reader cannot push down conjuncts for min-max filter #60197 :
Null pointer exceptions may occur when processing non-slot topn values.

*** Query id: 45488371394ee0-aac81c6593b2f02a ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1769444368 (unix time) try "date -d @1769444368" if you are using GNU date ***
*** Current BE git commitID: a0eb8c0c71 ***
*** SIGSEGV address not mapped to object (@0x18) received by PID 13103 (TID 13838 OR 0x7ff0423ee640) from PID 24; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/signal_handler.h:420
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so
3# 0x00007FF250EE5520 in /lib/x86_64-linux-gnu/libc.so.6
4# doris::vectorized::RuntimePredicate::update(doris::vectorized::Field const&) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_predicate.cpp:219
5# doris::pipeline::SortSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/sort_sink_operator.cpp:152
6# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:598

fix : file scan node push down bloom_filter & bitmap_filter for min-max filter.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2026-01-27T08:01:27Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hubgeter · 2026-01-27T08:01:27Z

run buildall

github-actions · 2026-01-27T08:21:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-01-27T08:21:39Z

PR approved by anyone and no changes requested.

doris-robot · 2026-01-27T08:38:59Z

TPC-H: Total hot run time: 32843 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 64fa5e1181eedfd5003a5f03ce9f7cfbc01c9513, data reload: false
------ Round 1 ----------------------------------
q1	17628	5340	5090	5090
q2	2100	317	189	189
q3	10181	1342	771	771
q4	10218	850	317	317
q5	7508	2199	1898	1898
q6	200	183	148	148
q7	916	730	594	594
q8	9266	1431	1139	1139
q9	5127	4878	4845	4845
q10	6804	1942	1585	1585
q11	513	293	281	281
q12	331	381	230	230
q13	17774	4073	3250	3250
q14	242	243	215	215
q15	903	826	813	813
q16	693	710	629	629
q17	643	823	470	470
q18	6836	6419	7661	6419
q19	1254	1035	636	636
q20	405	366	264	264
q21	3015	2243	2066	2066
q22	1128	1110	994	994
Total cold run time: 103685 ms
Total hot run time: 32843 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5586	5792	5583	5583
q2	258	351	247	247
q3	2478	2842	2430	2430
q4	1519	1959	1465	1465
q5	4580	4669	4649	4649
q6	235	186	139	139
q7	2098	1943	1806	1806
q8	2574	2678	2432	2432
q9	7424	7458	7576	7458
q10	2835	3172	2643	2643
q11	558	473	454	454
q12	774	781	616	616
q13	3784	4356	3247	3247
q14	269	275	257	257
q15	835	795	790	790
q16	642	680	637	637
q17	1070	1196	1285	1196
q18	7698	7307	7382	7307
q19	818	782	789	782
q20	1961	2034	1918	1918
q21	4616	4166	4055	4055
q22	1061	1061	951	951
Total cold run time: 53673 ms
Total hot run time: 51062 ms

doris-robot · 2026-01-27T08:55:47Z

ClickBench: Total hot run time: 28.39 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 64fa5e1181eedfd5003a5f03ce9f7cfbc01c9513, data reload: false
query1	0.05	0.04	0.05
query2	0.10	0.05	0.04
query3	0.25	0.08	0.08
query4	1.61	0.11	0.11
query5	0.27	0.25	0.25
query6	1.17	0.67	0.67
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.57	0.56	0.55
query11	0.14	0.09	0.09
query12	0.14	0.10	0.10
query13	0.64	0.62	0.63
query14	1.06	1.04	1.05
query15	0.89	0.86	0.87
query16	0.40	0.39	0.38
query17	1.12	1.15	1.09
query18	0.23	0.21	0.21
query19	2.03	2.03	2.00
query20	0.03	0.02	0.02
query21	15.41	0.27	0.15
query22	5.20	0.06	0.04
query23	15.95	0.27	0.10
query24	1.82	0.47	0.80
query25	0.12	0.12	0.07
query26	0.14	0.13	0.13
query27	0.06	0.06	0.04
query28	4.92	1.14	0.96
query29	12.55	3.98	3.16
query30	0.28	0.13	0.11
query31	2.81	0.63	0.41
query32	3.23	0.59	0.51
query33	3.29	3.21	3.26
query34	16.01	5.41	4.70
query35	4.77	4.80	4.77
query36	0.66	0.51	0.49
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.02
query40	0.19	0.17	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.03	0.04
Total cold run time: 99.17 s
Total hot run time: 28.39 s

hello-stephen · 2026-01-27T10:54:55Z

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.49% (19160/36501)
Line Coverage	35.90% (178248/496537)
Region Coverage	32.37% (137861/425947)
Branch Coverage	33.30% (59667/179197)

…t to column predicate error. (#60262) (#60268) ### What problem does this PR solve? Problem Summary: bp #60262

hubgeter · 2026-01-28T10:36:52Z

run buildall

doris-robot · 2026-01-28T11:19:40Z

TPC-H: Total hot run time: 31935 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7a1f6ef34867d93e44eeec4a36450bd976880f58, data reload: false
------ Round 1 ----------------------------------
q1	17609	5232	5088	5088
q2	2022	307	191	191
q3	10277	1297	738	738
q4	10200	818	320	320
q5	7511	2126	1893	1893
q6	194	179	148	148
q7	910	741	603	603
q8	9272	1359	1065	1065
q9	5242	4917	4891	4891
q10	6806	1968	1576	1576
q11	501	295	261	261
q12	332	386	220	220
q13	17774	4032	3201	3201
q14	229	233	221	221
q15	913	828	802	802
q16	670	675	623	623
q17	629	800	456	456
q18	6836	6585	6588	6585
q19	1243	985	624	624
q20	393	341	223	223
q21	2588	1964	1934	1934
q22	354	310	272	272
Total cold run time: 102505 ms
Total hot run time: 31935 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5342	5300	5321	5300
q2	269	336	250	250
q3	2155	2674	2292	2292
q4	1349	1719	1299	1299
q5	4262	4211	4272	4211
q6	215	187	140	140
q7	2114	2114	1892	1892
q8	2566	2524	2396	2396
q9	7517	7502	7518	7502
q10	2878	3120	2604	2604
q11	552	472	455	455
q12	662	750	640	640
q13	3819	4619	3466	3466
q14	298	331	298	298
q15	898	811	832	811
q16	700	750	676	676
q17	1155	1381	1365	1365
q18	8581	7935	7956	7935
q19	840	821	866	821
q20	2057	2138	1971	1971
q21	4844	4348	4048	4048
q22	631	549	531	531
Total cold run time: 53704 ms
Total hot run time: 50903 ms

doris-robot · 2026-01-28T11:36:24Z

ClickBench: Total hot run time: 28.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7a1f6ef34867d93e44eeec4a36450bd976880f58, data reload: false
query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.25	0.09	0.08
query4	1.60	0.11	0.11
query5	0.26	0.26	0.26
query6	1.16	0.68	0.68
query7	0.03	0.03	0.02
query8	0.05	0.03	0.03
query9	0.55	0.50	0.49
query10	0.56	0.55	0.54
query11	0.14	0.10	0.10
query12	0.13	0.10	0.11
query13	0.64	0.62	0.61
query14	1.08	1.06	1.07
query15	0.86	0.86	0.87
query16	0.39	0.38	0.38
query17	1.07	1.12	1.08
query18	0.22	0.20	0.20
query19	2.07	2.06	2.00
query20	0.02	0.02	0.01
query21	15.39	0.27	0.15
query22	5.25	0.05	0.04
query23	15.99	0.28	0.11
query24	0.95	0.23	0.51
query25	0.09	0.08	0.06
query26	0.16	0.13	0.13
query27	0.06	0.11	0.06
query28	3.89	1.16	0.96
query29	12.54	4.01	3.21
query30	0.28	0.13	0.12
query31	2.82	0.63	0.40
query32	3.24	0.59	0.51
query33	3.16	3.37	3.24
query34	16.13	5.43	4.72
query35	4.82	4.78	4.82
query36	0.64	0.49	0.50
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 97.21 s
Total hot run time: 28.29 s

hello-stephen · 2026-01-28T11:57:23Z

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.72% (19255/36526)
Line Coverage	36.09% (178864/495666)
Region Coverage	32.54% (138747/426354)
Branch Coverage	33.48% (60041/179343)

hello-stephen · 2026-01-28T13:34:53Z

BE Regression && UT Coverage Report

Increment line coverage 20.00% (1/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	69.84% (25007/35804)
Line Coverage	52.29% (258518/494433)
Region Coverage	49.80% (214487/430672)
Branch Coverage	51.00% (91831/180059)

hello-stephen · 2026-01-29T05:30:53Z

BE Regression && UT Coverage Report

Increment line coverage 20.00% (1/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	69.84% (25007/35804)
Line Coverage	52.29% (258518/494433)
Region Coverage	49.80% (214487/430672)
Branch Coverage	51.00% (91831/180059)

github-actions · 2026-01-29T06:07:56Z

PR approved by at least one committer and no changes requested.

@0x18

… predicate error. (#60262) ### What problem does this PR solve? Problem Summary: 1. fix#60197 : Null pointer exceptions may occur when processing non-slot topn values. ``` *** Query id: 45488371394ee0-aac81c6593b2f02a *** *** is nereids: 1 *** *** tablet id: 0 *** *** Aborted at 1769444368 (unix time) try "date -d @1769444368" if you are using GNU date *** *** Current BE git commitID: a0eb8c0 *** *** SIGSEGV address not mapped to object (@0x18) received by PID 13103 (TID 13838 OR 0x7ff0423ee640) from PID 24; stack trace: *** 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 2# JVM_handle_linux_signal in /usr/lib/jvm/java-17-openjdk-amd64/lib/server/libjvm.so 3# 0x00007FF250EE5520 in /lib/x86_64-linux-gnu/libc.so.6 4# doris::vectorized::RuntimePredicate::update(doris::vectorized::Field const&) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/runtime/runtime_predicate.cpp:219 5# doris::pipeline::SortSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/exec/sort_sink_operator.cpp:152 6# doris::pipeline::PipelineTask::execute(bool*) at /home/zcp/repo_center/doris_branch-4.0/doris/be/src/pipeline/pipeline_task.cpp:598 ```` 2. fix : file scan node push down bloom_filter & bitmap_filter for min-max filter.

…t to column predicate error. (apache#60262) (apache#60268) ### What problem does this PR solve? Problem Summary: bp apache#60262

This PR backports the external Docker startup improvements and the applicable follow-up fixes to `branch-4.0`. ### Included backports | PR | Backported scope | | --- | --- | | #61852 | Split Hive bootstrap data into common, Hive2-only, and Hive3-only groups | | #62103 | Hive baseline restore plus incremental module refresh | | #62664 | External Docker startup cleanup, diagnostics, and health handling | | #62690 | ClickHouse startup and health-check stabilization | | #63719 | Only the deterministic `find \| sort` changes required by Iceberg/Paimon initialization | | #61671 | Reduce duplicate Hive2/Hive3 regression runs | | #65460 | Remove obsolete Hive transactional cases and their initialization work | | #65564 | Backport the lightweight Kerberos environment | | #61869 | Add external regression stage timing and diagnostics | | #65459 | Move resource-intensive external cases and Docker components to P2 | | #65825 | Apply the follow-up fixes for the lightweight Kerberos environment | | #65251 | Remove the obsolete Parquet complex cross-page case and expected output | The later fixes made while validating the companion `branch-4.1` backport #65913 are also included: - Switch `test_hive_topn_rf_null` to Hive2 so it matches the Hive2-only `run81.hql` bootstrap group. - Change the location-backed `decimals_1_10` table in `run82.hql` to `CREATE TABLE IF NOT EXISTS`, avoiding deletion of preinstalled HDFS data during refresh. - Keep `HIVE_BASELINE_VERSION=20260415`, the same baseline version used by `master` and the companion 4.1 backport. ### Intentionally not included | PR | Reason | | --- | --- | | #63673 | Already present in `branch-4.0` | | #63542 | Already present in `branch-4.0` | | #64886 | `branch-4.0` still uses the older Iceberg/Spark layout; this cannot be picked independently | | #64671 | Contains a BE feature change together with new regression cases; it is outside this Docker backport | | #60262 | Not cherry-picked as a whole; only the final TopN Hive2 mapping needed by the bootstrap grouping is retained | ## Why `branch-4.0` still uses the older external Docker initialization path. Without these backports, Hive modules can remain incompletely refreshed, Iceberg/Paimon SQL execution order can be nondeterministic, and external jobs repeat unnecessary Hive2/Hive3 coverage. During conflict resolution, this backport preserves the `branch-4.0` Iceberg/Paimon 1.10 data archive and jar bootstrap while integrating the new startup and health-check framework. It does not introduce the newer Spark Thrift JDBC architecture. ## Validation - `git diff --check` - Conflict-marker scan across `docker`, `regression-test`, and `be/src` - Bash syntax checks for all changed `.sh` and `.sh.tpl` files - `bash docker/thirdparties/test/run-thirdparties-docker-hive-bootstrap-groups-test.sh` - `python3 regression-test/pipeline/common/test_stage_timer.py` - Confirmed the BE and new serde regression files from #64671 are absent from the final diff Compilation and local Docker startup were intentionally skipped. The final scope contains Docker, scripts, pipeline support, and regression-case organization only. --------- Co-authored-by: Chenjunwei <138805230+xylaaaaa@users.noreply.github.com>

[fix](topn)fix topn&rf expr convert to column predicate error.
64fa5e1

fix
ec02c42

hubgeter changed the title ~~[fix](topn)fix topn&rf expr convert to column predicate error.~~[fix](topn)fix topn&bloom&bitmap filter convert to column predicate error.Jan 27, 2026

hubgeter changed the title ~~[fix](topn)fix topn&bloom&bitmap filter convert to column predicate error.~~[fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error.Jan 27, 2026

BiteTheDDDDt previously approved these changes Jan 27, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2026

github-actionsBot added the reviewed label Jan 27, 2026

hubgeter mentioned this pull request Jan 27, 2026
branch-4.0:[fix](column predicate)fix topn&bloom&bitmap filter convert to column predicate error. (#60262) #60268
Merged

yiguolei pushed a commit that referenced this pull request Jan 27, 2026
branch-4.0:[fix](column predicate)fix topn&bloom&bitmap filter conver…
dca74d8
…t to column predicate error. (#60262) (#60268) ### What problem does this PR solve? Problem Summary: bp #60262

add case for column predicate.
7a1f6ef

hubgeter dismissed BiteTheDDDDt’s stale review via 7a1f6efJanuary 28, 2026 10:36

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jan 28, 2026

zclllyybb approved these changes Jan 29, 2026
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2026

morningman approved these changes Jan 29, 2026
View reviewed changes

morningman added the dev/4.0.3-merged label Jan 29, 2026

morningman merged commit 22ce979 into apache:masterJan 29, 2026
31 of 32 checks passed

suxiaogang223 mentioned this pull request Jul 23, 2026
branch-4.0: [fix](docker) Backport external Docker improvements #65939
Merged

Uh oh!

Conversation

hubgeter commented Jan 27, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Jan 27, 2026

Uh oh!

hubgeter commented Jan 27, 2026

Uh oh!

github-actionsBot commented Jan 27, 2026

Uh oh!

github-actionsBot commented Jan 27, 2026

Uh oh!

doris-robot commented Jan 27, 2026

Uh oh!

doris-robot commented Jan 27, 2026

Uh oh!

hello-stephen commented Jan 27, 2026

BE UT Coverage Report

Uh oh!

hubgeter commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

doris-robot commented Jan 28, 2026

Uh oh!

hello-stephen commented Jan 28, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 28, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jan 29, 2026

BE Regression && UT Coverage Report

Uh oh!

github-actionsBot commented Jan 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hubgeter commented Jan 27, 2026•
edited
Loading