[fix](bloomfilter) fix inlist support for date/datetimev1 in bloomfilter index by airborne12 · Pull Request #46961 · apache/doris

airborne12 · 2025-01-14T04:45:26Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43351

Problem Summary:
#43351 fix date/datetime v1 support in comparison predicate, this PR try to fix it in inlist predicate.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

…ter index

hello-stephen · 2025-01-14T04:45:37Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2025-01-14T04:45:41Z

run buildall

doris-robot · 2025-01-14T05:34:32Z

TPC-H: Total hot run time: 32684 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8b191310f2fc6acc894a905e449e69c927278ada, data reload: false
------ Round 1 ----------------------------------
q1	17562	6124	6055	6055
q2	2053	311	170	170
q3	10410	1245	750	750
q4	10206	868	428	428
q5	7726	2167	1989	1989
q6	235	190	146	146
q7	914	764	609	609
q8	9252	1334	1142	1142
q9	5271	4889	4907	4889
q10	6759	2306	1849	1849
q11	483	279	253	253
q12	348	352	217	217
q13	17782	3713	3111	3111
q14	233	221	226	221
q15	579	544	513	513
q16	627	638	590	590
q17	583	842	326	326
q18	7121	6390	6443	6390
q19	1726	953	528	528
q20	316	322	186	186
q21	3027	2189	2011	2011
q22	367	342	311	311
Total cold run time: 103580 ms
Total hot run time: 32684 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6265	6185	6196	6185
q2	246	333	238	238
q3	2252	2635	2340	2340
q4	1461	1828	1396	1396
q5	4318	4762	4825	4762
q6	188	189	141	141
q7	2058	1956	1788	1788
q8	2619	2799	2693	2693
q9	7285	7265	7352	7265
q10	3029	3357	2698	2698
q11	563	527	486	486
q12	670	798	603	603
q13	3568	4022	3287	3287
q14	277	299	286	286
q15	581	537	525	525
q16	623	689	640	640
q17	1239	1766	1251	1251
q18	7623	7428	7433	7428
q19	801	990	1124	990
q20	1986	2034	1879	1879
q21	5787	5150	4916	4916
q22	624	650	577	577
Total cold run time: 54063 ms
Total hot run time: 52374 ms

doris-robot · 2025-01-14T05:47:09Z

TPC-DS: Total hot run time: 194778 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8b191310f2fc6acc894a905e449e69c927278ada, data reload: false
query1	1334	1000	944	944
query2	6359	2044	2018	2018
query3	11111	4675	4492	4492
query4	61830	29104	23307	23307
query5	5457	618	455	455
query6	436	190	191	190
query7	5541	505	298	298
query8	331	243	228	228
query9	8401	2708	2691	2691
query10	442	322	259	259
query11	17838	15262	15662	15262
query12	157	112	116	112
query13	1457	524	445	445
query14	11189	7515	7084	7084
query15	224	191	184	184
query16	7330	645	449	449
query17	1097	715	549	549
query18	1936	396	322	322
query19	193	178	154	154
query20	124	113	111	111
query21	206	122	110	110
query22	4357	4763	4384	4384
query23	34145	33958	34081	33958
query24	5554	2355	2283	2283
query25	480	474	374	374
query26	699	283	161	161
query27	2144	475	333	333
query28	4186	2481	2446	2446
query29	534	562	416	416
query30	211	209	154	154
query31	979	893	827	827
query32	71	56	56	56
query33	484	359	311	311
query34	765	891	500	500
query35	805	834	724	724
query36	1028	1042	970	970
query37	123	104	73	73
query38	4376	4226	4147	4147
query39	1506	1483	1411	1411
query40	210	125	102	102
query41	49	56	49	49
query42	118	103	104	103
query43	527	529	513	513
query44	1318	849	841	841
query45	189	177	163	163
query46	880	1072	668	668
query47	1892	1924	1871	1871
query48	394	398	334	334
query49	757	524	410	410
query50	672	687	401	401
query51	7026	7159	6977	6977
query52	106	103	94	94
query53	235	256	184	184
query54	487	495	417	417
query55	85	78	82	78
query56	273	253	263	253
query57	1221	1207	1089	1089
query58	232	233	226	226
query59	3072	3013	2951	2951
query60	275	264	262	262
query61	116	116	114	114
query62	794	774	705	705
query63	217	183	185	183
query64	1670	1024	711	711
query65	3244	3160	3141	3141
query66	664	419	294	294
query67	16131	15765	15456	15456
query68	4628	817	530	530
query69	484	302	266	266
query70	1182	1158	1100	1100
query71	419	277	265	265
query72	6099	3872	3806	3806
query73	776	750	357	357
query74	10059	8964	8595	8595
query75	3212	3191	2640	2640
query76	3547	1192	766	766
query77	481	371	272	272
query78	10157	10211	9353	9353
query79	2309	826	605	605
query80	666	516	429	429
query81	506	282	235	235
query82	289	224	123	123
query83	175	163	149	149
query84	286	99	75	75
query85	745	350	295	295
query86	399	305	299	299
query87	4298	4382	4469	4382
query88	3661	2163	2125	2125
query89	396	328	300	300
query90	1678	189	196	189
query91	142	136	105	105
query92	76	58	55	55
query93	2731	877	533	533
query94	674	406	306	306
query95	323	267	256	256
query96	485	605	288	288
query97	2822	2896	2746	2746
query98	224	203	196	196
query99	1406	1530	1355	1355
Total cold run time: 313629 ms
Total hot run time: 194778 ms

doris-robot · 2025-01-14T05:52:30Z

ClickBench: Total hot run time: 31.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8b191310f2fc6acc894a905e449e69c927278ada, data reload: false
query1	0.03	0.03	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.11
query5	0.41	0.44	0.40
query6	1.13	0.66	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.49	0.50
query10	0.56	0.56	0.55
query11	0.14	0.11	0.10
query12	0.13	0.11	0.10
query13	0.61	0.59	0.60
query14	2.72	2.85	2.73
query15	0.91	0.83	0.83
query16	0.40	0.38	0.40
query17	1.05	1.06	1.08
query18	0.23	0.20	0.22
query19	1.97	1.83	1.96
query20	0.01	0.01	0.01
query21	15.38	0.92	0.57
query22	0.77	0.82	0.88
query23	15.07	1.41	0.55
query24	3.09	1.67	0.90
query25	0.18	0.29	0.15
query26	0.25	0.13	0.13
query27	0.06	0.04	0.04
query28	13.93	1.52	1.06
query29	12.62	4.01	3.27
query30	0.26	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.56	0.46
query33	2.95	3.02	3.04
query34	16.73	5.17	4.54
query35	4.63	4.56	4.56
query36	0.64	0.52	0.47
query37	0.11	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 105.99 s
Total hot run time: 31.46 s

doris-robot · 2025-01-14T06:25:40Z

TeamCity be ut coverage result:
Function Coverage: 40.60% (10584/26070)
Line Coverage: 31.26% (89562/286504)
Region Coverage: 30.38% (45808/150787)
Branch Coverage: 26.66% (23285/87350)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8b191310f2fc6acc894a905e449e69c927278ada_8b191310f2fc6acc894a905e449e69c927278ada/report/index.html

airborne12 · 2025-01-14T07:38:26Z

run buildall

doris-robot · 2025-01-14T08:08:43Z

TPC-H: Total hot run time: 32774 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bc10a8c1145bff3d179ecdd3b7eda84d4aea4486, data reload: false
------ Round 1 ----------------------------------
q1	17599	6187	6073	6073
q2	2047	328	176	176
q3	10491	1275	728	728
q4	10240	866	449	449
q5	8337	2221	2013	2013
q6	209	181	150	150
q7	903	751	618	618
q8	9234	1360	1146	1146
q9	5286	4854	4923	4854
q10	6751	2286	1849	1849
q11	462	275	272	272
q12	347	354	222	222
q13	17767	3678	3175	3175
q14	231	246	211	211
q15	565	526	521	521
q16	630	627	594	594
q17	586	857	321	321
q18	7147	6479	6348	6348
q19	1321	961	548	548
q20	319	323	192	192
q21	2882	2248	2000	2000
q22	367	340	314	314
Total cold run time: 103721 ms
Total hot run time: 32774 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6284	6222	6240	6222
q2	234	333	233	233
q3	2269	2656	2365	2365
q4	1432	1800	1374	1374
q5	4365	4766	4889	4766
q6	185	174	140	140
q7	2106	1956	1807	1807
q8	2618	2770	2696	2696
q9	7249	7214	7298	7214
q10	3022	3345	2820	2820
q11	593	538	505	505
q12	676	763	630	630
q13	3487	3878	3356	3356
q14	280	293	273	273
q15	575	548	519	519
q16	646	687	649	649
q17	1216	1725	1279	1279
q18	7751	7291	7386	7291
q19	802	836	1147	836
q20	2011	2032	1880	1880
q21	5846	5338	4907	4907
q22	601	573	553	553
Total cold run time: 54248 ms
Total hot run time: 52315 ms

doris-robot · 2025-01-14T08:20:39Z

TPC-DS: Total hot run time: 189056 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bc10a8c1145bff3d179ecdd3b7eda84d4aea4486, data reload: false
query1	959	391	372	372
query2	7158	2039	2036	2036
query3	6707	212	222	212
query4	33926	24135	23545	23545
query5	4400	612	476	476
query6	294	184	170	170
query7	4616	479	303	303
query8	277	226	224	224
query9	9606	2685	2688	2685
query10	490	307	258	258
query11	18120	15279	15054	15054
query12	149	109	104	104
query13	1648	518	405	405
query14	10566	7380	7367	7367
query15	261	188	187	187
query16	8253	614	417	417
query17	1608	727	543	543
query18	2061	396	287	287
query19	225	174	150	150
query20	118	107	112	107
query21	209	133	102	102
query22	4375	4435	4013	4013
query23	34106	33266	33290	33266
query24	6498	2366	2235	2235
query25	493	457	384	384
query26	1210	269	163	163
query27	1981	476	325	325
query28	5347	2453	2396	2396
query29	686	537	413	413
query30	229	200	161	161
query31	973	876	794	794
query32	85	61	60	60
query33	517	353	312	312
query34	757	860	493	493
query35	794	801	779	779
query36	991	1064	942	942
query37	129	103	76	76
query38	4105	4191	4092	4092
query39	1454	1427	1382	1382
query40	203	111	99	99
query41	53	56	51	51
query42	119	101	100	100
query43	507	531	495	495
query44	1328	809	804	804
query45	179	169	168	168
query46	860	1052	643	643
query47	1804	1822	1793	1793
query48	378	402	327	327
query49	771	484	452	452
query50	628	663	379	379
query51	6859	7036	6820	6820
query52	99	102	89	89
query53	232	247	183	183
query54	497	496	419	419
query55	89	91	82	82
query56	273	278	251	251
query57	1235	1116	1106	1106
query58	262	245	252	245
query59	3033	3065	2863	2863
query60	290	267	267	267
query61	174	148	123	123
query62	834	791	690	690
query63	230	195	187	187
query64	4186	1008	647	647
query65	3246	3127	3185	3127
query66	1073	422	305	305
query67	15959	15591	15519	15519
query68	4429	839	532	532
query69	466	292	257	257
query70	1253	1128	1131	1128
query71	385	285	264	264
query72	5886	3919	3801	3801
query73	644	739	361	361
query74	10208	9023	9093	9023
query75	3153	3177	2676	2676
query76	3116	1169	784	784
query77	457	372	288	288
query78	9964	9823	9283	9283
query79	3156	805	596	596
query80	1669	525	462	462
query81	576	278	247	247
query82	360	147	120	120
query83	274	184	160	160
query84	246	87	74	74
query85	777	365	293	293
query86	470	316	297	297
query87	4461	4499	4436	4436
query88	4686	2201	2122	2122
query89	395	334	293	293
query90	1853	190	198	190
query91	132	137	111	111
query92	67	64	53	53
query93	2725	884	527	527
query94	740	396	286	286
query95	345	261	251	251
query96	495	614	281	281
query97	2800	2884	2757	2757
query98	229	199	198	198
query99	1397	1493	1381	1381
Total cold run time: 289596 ms
Total hot run time: 189056 ms

doris-robot · 2025-01-14T08:25:58Z

ClickBench: Total hot run time: 30.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bc10a8c1145bff3d179ecdd3b7eda84d4aea4486, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.60	0.11	0.10
query5	0.43	0.41	0.40
query6	1.15	0.66	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.49	0.51
query10	0.55	0.57	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.71	2.85	2.79
query15	0.90	0.83	0.84
query16	0.40	0.38	0.38
query17	1.03	1.03	1.01
query18	0.24	0.21	0.21
query19	1.90	1.80	1.92
query20	0.02	0.01	0.01
query21	15.36	0.94	0.57
query22	0.76	0.83	0.54
query23	15.39	1.38	0.52
query24	3.11	1.52	0.69
query25	0.21	0.12	0.15
query26	0.30	0.14	0.14
query27	0.06	0.08	0.04
query28	13.82	1.53	1.04
query29	12.54	3.92	3.22
query30	0.25	0.09	0.06
query31	2.84	0.62	0.38
query32	3.24	0.55	0.47
query33	2.93	2.99	3.01
query34	16.80	5.14	4.47
query35	4.51	4.54	4.50
query36	0.67	0.48	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.15	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.09 s
Total hot run time: 30.74 s

doris-robot · 2025-01-14T09:26:00Z

TeamCity be ut coverage result:
Function Coverage: 40.65% (10595/26062)
Line Coverage: 31.29% (89624/286455)
Region Coverage: 30.39% (45823/150793)
Branch Coverage: 26.66% (23289/87354)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc10a8c1145bff3d179ecdd3b7eda84d4aea4486_bc10a8c1145bff3d179ecdd3b7eda84d4aea4486/report/index.html

csun5285

LGTM

github-actions · 2025-01-15T01:52:06Z

PR approved by anyone and no changes requested.

qidaye

LGTM

github-actions · 2025-01-15T02:18:24Z

PR approved by at least one committer and no changes requested.

…ter index (#46961) Problem Summary: #43351 fix date/datetime v1 support in comparison predicate, this PR try to fix it in inlist predicate.

…ter index (apache#46961) Problem Summary: apache#43351 fix date/datetime v1 support in comparison predicate, this PR try to fix it in inlist predicate.

… in bloomfilter index #46961 (#47005) Cherry-picked from #46961 Co-authored-by: airborne12 <jiangkai@selectdb.com>

[fix](bloomfilter) fix inlist support for date/datetimev1 in bloomfil…
8b19131
…ter index

airborne12 added 2 commits January 14, 2025 15:03

add UT
867f4ed

add out
bc10a8c

csun5285 approved these changes Jan 15, 2025
View reviewed changes

github-actionsBot added the reviewed label Jan 15, 2025

qidaye approved these changes Jan 15, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jan 15, 2025

airborne12 merged commit a0f4c4f into apache:masterJan 15, 2025

airborne12 deleted the fix1 branch January 15, 2025 02:56

airborne12 added dev/2.1.x dev/3.0.x labels Jan 15, 2025

github-actionsBot mentioned this pull request Jan 15, 2025
branch-2.1: [fix](bloomfilter) fix inlist support for date/datetimev1 in bloomfilter index #46961 #47004
Closed

github-actionsBot mentioned this pull request Jan 15, 2025
branch-3.0: [fix](bloomfilter) fix inlist support for date/datetimev1 in bloomfilter index #46961 #47005
Merged

airborne12 removed the dev/2.1.x label Jan 18, 2025

dataroaring pushed a commit that referenced this pull request Feb 24, 2025
branch-3.0: [fix](bloomfilter) fix inlist support for date/datetimev1…
e6bfa54
… in bloomfilter index #46961 (#47005) Cherry-picked from #46961 Co-authored-by: airborne12 <jiangkai@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Feb 24, 2025

Uh oh!

Conversation

airborne12 commented Jan 14, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 14, 2025

Uh oh!

airborne12 commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

airborne12 commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

doris-robot commented Jan 14, 2025

Uh oh!

csun5285 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 15, 2025

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jan 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

airborne12 commented Jan 14, 2025•
edited
Loading