[fix](parquet/orc) Disable string dictionary filtering when predicate express is not binary pred and in pred by suxiaogang223 · Pull Request #48532 · apache/doris

suxiaogang223 · 2025-03-03T03:54:00Z

What problem does this PR solve?

Related PR: #42113

Problem Summary:
In the previous fix, the conditions for when dictionary filtering can be performed were wrong.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-03-03T03:54:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2025-04-25T10:38:55Z

run buildall

suxiaogang223 · 2025-05-07T16:37:03Z

run buildall

doris-robot · 2025-05-07T17:15:47Z

TPC-H: Total hot run time: 33895 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 281a4c33fc59ae9528d4343afc6e30f9128d41ba, data reload: false
------ Round 1 ----------------------------------
q1	25972	4987	4952	4952
q2	2067	270	186	186
q3	10388	1247	671	671
q4	10217	998	512	512
q5	7510	2381	2299	2299
q6	181	162	132	132
q7	917	751	601	601
q8	9296	1244	1104	1104
q9	6930	5030	5118	5030
q10	6815	2304	1887	1887
q11	507	281	276	276
q12	357	353	207	207
q13	17779	3753	3089	3089
q14	226	221	215	215
q15	547	491	495	491
q16	427	431	374	374
q17	600	839	360	360
q18	7832	7363	7234	7234
q19	1653	967	558	558
q20	345	351	224	224
q21	4212	3469	2513	2513
q22	1105	1045	980	980
Total cold run time: 115883 ms
Total hot run time: 33895 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5095	5048	5166	5048
q2	230	324	230	230
q3	2123	2593	2247	2247
q4	1326	1768	1369	1369
q5	4434	4375	4381	4375
q6	212	165	127	127
q7	1994	1902	1750	1750
q8	2575	2529	2530	2529
q9	7208	7216	7090	7090
q10	2991	3147	2734	2734
q11	591	509	488	488
q12	702	777	596	596
q13	3456	3828	3363	3363
q14	306	294	272	272
q15	521	464	464	464
q16	426	487	452	452
q17	1128	1468	1435	1435
q18	7685	7619	7419	7419
q19	794	827	1029	827
q20	1980	1987	1828	1828
q21	5141	4741	4690	4690
q22	1118	1071	1016	1016
Total cold run time: 52036 ms
Total hot run time: 50349 ms

doris-robot · 2025-05-07T17:27:59Z

TPC-DS: Total hot run time: 191962 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 281a4c33fc59ae9528d4343afc6e30f9128d41ba, data reload: false
query1	1412	1106	1054	1054
query2	6263	1843	1795	1795
query3	11109	4556	4630	4556
query4	54014	25199	22830	22830
query5	5220	508	445	445
query6	373	220	193	193
query7	5003	511	289	289
query8	337	269	235	235
query9	6319	2525	2544	2525
query10	415	305	262	262
query11	15066	15000	14874	14874
query12	152	108	104	104
query13	1090	487	381	381
query14	9947	6281	6213	6213
query15	189	192	195	192
query16	7082	655	476	476
query17	1094	744	605	605
query18	1555	415	330	330
query19	202	195	167	167
query20	126	118	115	115
query21	208	133	110	110
query22	4511	4677	4552	4552
query23	34112	33691	33465	33465
query24	6633	2395	2452	2395
query25	469	450	406	406
query26	722	273	163	163
query27	2171	504	334	334
query28	3027	2090	2097	2090
query29	623	568	433	433
query30	266	227	188	188
query31	857	869	780	780
query32	76	65	64	64
query33	460	357	326	326
query34	810	882	537	537
query35	799	818	754	754
query36	943	997	902	902
query37	114	98	71	71
query38	4197	4240	4160	4160
query39	1499	1474	1448	1448
query40	226	120	109	109
query41	56	56	55	55
query42	124	115	108	108
query43	515	509	465	465
query44	1345	843	819	819
query45	182	177	170	170
query46	844	1015	639	639
query47	1861	1886	1822	1822
query48	375	415	313	313
query49	687	488	404	404
query50	676	719	409	409
query51	4107	4281	4130	4130
query52	120	119	104	104
query53	234	261	185	185
query54	605	584	512	512
query55	87	84	83	83
query56	316	303	284	284
query57	1228	1183	1120	1120
query58	261	256	257	256
query59	2767	2835	2732	2732
query60	322	319	302	302
query61	128	129	126	126
query62	719	752	674	674
query63	223	190	182	182
query64	1873	1019	652	652
query65	4294	4224	4263	4224
query66	715	399	300	300
query67	15835	15670	15407	15407
query68	6858	827	502	502
query69	539	299	254	254
query70	1209	1146	1058	1058
query71	511	309	287	287
query72	5989	4739	4723	4723
query73	1312	689	345	345
query74	8945	9030	8912	8912
query75	3774	3205	2685	2685
query76	4221	1181	758	758
query77	596	367	282	282
query78	10145	10130	9187	9187
query79	2546	808	561	561
query80	653	507	429	429
query81	481	251	262	251
query82	533	120	99	99
query83	249	242	248	242
query84	298	110	82	82
query85	759	373	309	309
query86	375	304	258	258
query87	4364	4400	4437	4400
query88	3476	2245	2230	2230
query89	400	309	285	285
query90	1780	211	213	211
query91	141	142	110	110
query92	81	58	56	56
query93	2316	941	571	571
query94	661	417	303	303
query95	371	297	278	278
query96	486	560	271	271
query97	3128	3224	3137	3137
query98	225	207	194	194
query99	1329	1406	1287	1287
Total cold run time: 298941 ms
Total hot run time: 191962 ms

doris-robot · 2025-05-07T17:33:10Z

ClickBench: Total hot run time: 29.29 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 281a4c33fc59ae9528d4343afc6e30f9128d41ba, data reload: false
query1	0.04	0.04	0.03
query2	0.12	0.10	0.12
query3	0.26	0.20	0.19
query4	1.59	0.19	0.18
query5	0.59	0.60	0.59
query6	1.18	0.71	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.51
query10	0.56	0.58	0.56
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.60
query14	0.78	0.81	0.80
query15	0.86	0.85	0.85
query16	0.39	0.37	0.40
query17	1.01	1.01	1.00
query18	0.21	0.19	0.19
query19	1.86	1.82	1.81
query20	0.01	0.01	0.01
query21	15.40	0.90	0.52
query22	0.75	1.30	0.65
query23	15.54	1.37	0.62
query24	6.77	1.33	1.08
query25	0.52	0.13	0.17
query26	0.64	0.16	0.14
query27	0.05	0.05	0.05
query28	9.62	0.82	0.45
query29	12.52	3.93	3.20
query30	0.25	0.09	0.06
query31	2.83	0.57	0.38
query32	3.23	0.55	0.46
query33	3.02	3.00	3.09
query34	15.76	5.11	4.49
query35	4.51	4.54	4.51
query36	0.66	0.50	0.47
query37	0.09	0.07	0.07
query38	0.04	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.13
query41	0.08	0.04	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.53 s
Total hot run time: 29.29 s

hello-stephen · 2025-05-07T18:08:40Z

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.59% (14868/26745)
Line Coverage	44.63% (131186/293967)
Region Coverage	43.35% (66824/154148)
Branch Coverage	37.94% (33727/88894)

suxiaogang223 · 2025-05-08T16:48:42Z

run buildall

doris-robot · 2025-05-08T17:09:53Z

TPC-H: Total hot run time: 34061 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d35693eb1afcaaee0cb51936a7d82d4b518958b4, data reload: false
------ Round 1 ----------------------------------
q1	26055	5065	4967	4967
q2	2075	279	184	184
q3	10496	1254	721	721
q4	10282	1040	533	533
q5	9046	2388	2421	2388
q6	185	164	132	132
q7	938	773	615	615
q8	9308	1305	1150	1150
q9	6855	5106	5122	5106
q10	6808	2317	1885	1885
q11	482	293	265	265
q12	348	348	209	209
q13	17793	3690	3107	3107
q14	223	221	213	213
q15	536	478	484	478
q16	428	433	377	377
q17	622	872	367	367
q18	7653	7151	7200	7151
q19	1545	988	586	586
q20	330	347	229	229
q21	4095	3372	2450	2450
q22	1059	1003	948	948
Total cold run time: 117162 ms
Total hot run time: 34061 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5190	5066	5062	5062
q2	240	336	237	237
q3	2213	2686	2277	2277
q4	1405	1883	1510	1510
q5	4541	4476	4412	4412
q6	251	183	127	127
q7	2066	1923	1762	1762
q8	2878	2760	2834	2760
q9	7143	7173	7153	7153
q10	3055	3215	2752	2752
q11	626	495	491	491
q12	694	749	622	622
q13	3666	3971	3185	3185
q14	273	294	274	274
q15	546	494	493	493
q16	471	489	416	416
q17	1211	1721	1497	1497
q18	7731	7645	7363	7363
q19	868	970	1297	970
q20	2011	2069	1870	1870
q21	5176	4681	4636	4636
q22	1053	1034	993	993
Total cold run time: 53308 ms
Total hot run time: 50862 ms

doris-robot · 2025-05-08T17:21:26Z

TPC-DS: Total hot run time: 186183 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d35693eb1afcaaee0cb51936a7d82d4b518958b4, data reload: false
query1	997	484	496	484
query2	6568	1753	1746	1746
query3	6769	226	225	225
query4	26005	24338	23366	23366
query5	5266	628	463	463
query6	296	195	189	189
query7	4634	506	280	280
query8	283	248	225	225
query9	8644	2530	2531	2530
query10	498	320	258	258
query11	15547	15031	14873	14873
query12	172	107	110	107
query13	1631	500	396	396
query14	9269	6052	6136	6052
query15	207	193	166	166
query16	7222	645	506	506
query17	1132	694	560	560
query18	1947	400	303	303
query19	183	181	158	158
query20	124	114	118	114
query21	211	132	118	118
query22	4089	4228	4195	4195
query23	33830	32883	32918	32883
query24	8399	2349	2365	2349
query25	516	454	404	404
query26	1244	271	159	159
query27	2758	504	336	336
query28	4338	2092	2071	2071
query29	769	537	421	421
query30	276	214	183	183
query31	927	837	736	736
query32	72	62	64	62
query33	541	356	312	312
query34	777	836	520	520
query35	767	826	761	761
query36	952	993	906	906
query37	113	100	80	80
query38	4089	4099	4106	4099
query39	1438	1449	1396	1396
query40	213	122	109	109
query41	64	61	60	60
query42	123	104	110	104
query43	482	509	468	468
query44	1306	791	797	791
query45	183	181	170	170
query46	830	1040	629	629
query47	1754	1816	1777	1777
query48	376	434	301	301
query49	801	531	458	458
query50	645	687	410	410
query51	4101	4091	4144	4091
query52	111	101	100	100
query53	221	255	183	183
query54	574	564	502	502
query55	82	80	79	79
query56	297	297	292	292
query57	1105	1128	1086	1086
query58	259	243	246	243
query59	2571	2645	2608	2608
query60	334	326	298	298
query61	127	155	129	129
query62	829	729	657	657
query63	225	190	183	183
query64	4339	1013	716	716
query65	4300	4209	4251	4209
query66	1099	422	311	311
query67	15711	15510	15356	15356
query68	7724	879	518	518
query69	487	300	266	266
query70	1165	1080	1137	1080
query71	432	328	292	292
query72	5528	4951	4909	4909
query73	752	668	340	340
query74	9261	8857	8896	8857
query75	3331	3232	2690	2690
query76	3392	1190	760	760
query77	664	385	324	324
query78	9866	10043	9364	9364
query79	2235	809	566	566
query80	564	518	446	446
query81	493	262	218	218
query82	446	126	102	102
query83	256	254	235	235
query84	253	106	85	85
query85	747	352	313	313
query86	381	307	301	301
query87	4418	4461	4274	4274
query88	3782	2206	2230	2206
query89	396	311	287	287
query90	1997	206	211	206
query91	136	138	111	111
query92	83	60	56	56
query93	1802	935	582	582
query94	715	408	272	272
query95	372	299	285	285
query96	485	581	283	283
query97	3132	3222	3160	3160
query98	230	209	203	203
query99	1369	1381	1280	1280
Total cold run time: 273456 ms
Total hot run time: 186183 ms

doris-robot · 2025-05-08T17:26:38Z

ClickBench: Total hot run time: 29.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d35693eb1afcaaee0cb51936a7d82d4b518958b4, data reload: false
query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.24	0.19	0.19
query4	1.60	0.19	0.10
query5	0.56	0.58	0.55
query6	1.21	0.71	0.72
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.57	0.58	0.57
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	0.78	0.80	0.80
query15	0.88	0.85	0.86
query16	0.38	0.37	0.39
query17	1.00	1.02	1.05
query18	0.21	0.19	0.20
query19	1.96	1.82	1.80
query20	0.01	0.01	0.02
query21	15.40	0.91	0.55
query22	0.75	1.16	0.68
query23	14.98	1.36	0.64
query24	7.21	1.66	0.86
query25	0.45	0.19	0.09
query26	0.66	0.17	0.14
query27	0.06	0.05	0.05
query28	9.54	0.82	0.44
query29	12.56	3.97	3.27
query30	0.25	0.10	0.07
query31	2.81	0.63	0.38
query32	3.23	0.55	0.46
query33	3.00	3.12	3.04
query34	15.72	5.08	4.52
query35	4.58	4.57	4.49
query36	0.65	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 103.46 s
Total hot run time: 29.15 s

doris-robot · 2025-05-08T18:24:50Z

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	55.69% (14896/26750)
Line Coverage	44.49% (131367/295265)
Region Coverage	43.39% (66188/152533)
Branch Coverage	38.06% (33796/88792)

hello-stephen · 2025-05-08T19:31:06Z

BE Regression && UT Coverage Report

Increment line coverage 75.00% (6/8) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.30% (20813/26247)
Line Coverage	72.54% (213809/294763)
Region Coverage	70.72% (126514/178893)
Branch Coverage	64.34% (65190/101324)

morningman added dev/2.1.x dev/3.0.x labels Mar 3, 2025

morningman changed the title ~~[fix](multi-catalog) Disable string dictionary filtering when predicate express is not binary pred and in pred~~[fix](parquet/orc) Disable string dictionary filtering when predicate express is not binary pred and in predMar 4, 2025

morningman reviewed Mar 4, 2025
View reviewed changes

Comment threadbe/src/vec/exec/format/parquet/vparquet_group_reader.cpp

suxiaogang223 force-pushed the fix_dict_filter branch from 35f6eec to 8486c93Compare April 25, 2025 02:29

suxiaogang223 requested a review from CalvinKirs as a code owner April 25, 2025 06:53

suxiaogang223 force-pushed the fix_dict_filter branch from bda9aee to 8486c93Compare April 25, 2025 10:31

suxiaogang223 force-pushed the fix_dict_filter branch from 8486c93 to 74365a4Compare April 27, 2025 15:51

fix
281a4c3

suxiaogang223 force-pushed the fix_dict_filter branch from 74365a4 to 281a4c3Compare May 7, 2025 16:24

Merge branch 'master' into fix_dict_filter
d35693e

suxiaogang223 closed this May 9, 2025

suxiaogang223 deleted the fix_dict_filter branch May 9, 2025 06:06

morningman removed dev/2.1.x dev/3.0.x labels Aug 11, 2025

Uh oh!

Conversation

suxiaogang223 commented Mar 3, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 3, 2025

Uh oh!

Uh oh!

suxiaogang223 commented Apr 25, 2025

Uh oh!

suxiaogang223 commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

doris-robot commented May 7, 2025

Uh oh!

hello-stephen commented May 7, 2025

BE UT Coverage Report

Uh oh!

suxiaogang223 commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

Uh oh!

doris-robot commented May 8, 2025

BE UT Coverage Report

Uh oh!

hello-stephen commented May 8, 2025

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants