[opt](bloomfilter index) optimize memory usage for bloom filter index writer by airborne12 · Pull Request #45833 · apache/doris

airborne12 · 2024-12-24T01:59:08Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Optimize memory usage when adding string values for bloom filter index.
Using uint64 hash value instead of string values itself, it is expected to save a lot of memory for especially long text

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2024-12-24T01:59:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

airborne12 · 2024-12-24T01:59:38Z

run buildall

doris-robot · 2024-12-24T02:46:23Z

TPC-H: Total hot run time: 39975 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a1f8dc2f89121851216a36a37c9e3b321998bc68, data reload: false
------ Round 1 ----------------------------------
q1	17595	7483	7247	7247
q2	2048	185	174	174
q3	10608	1117	1177	1117
q4	10557	714	770	714
q5	7586	2772	2687	2687
q6	247	154	150	150
q7	994	637	611	611
q8	9270	1883	1959	1883
q9	6642	6428	6478	6428
q10	7002	2334	2312	2312
q11	468	262	256	256
q12	423	235	227	227
q13	17752	2898	2982	2898
q14	254	212	221	212
q15	558	514	500	500
q16	667	609	585	585
q17	1003	579	581	579
q18	7359	6721	6708	6708
q19	1359	965	1011	965
q20	465	182	185	182
q21	4029	3292	3242	3242
q22	377	322	298	298
Total cold run time: 107263 ms
Total hot run time: 39975 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7219	7175	7171	7171
q2	332	224	222	222
q3	2956	2838	2949	2838
q4	2039	1835	1819	1819
q5	5728	5717	5660	5660
q6	223	139	142	139
q7	2259	1831	1809	1809
q8	3409	3567	3583	3567
q9	8918	9054	9042	9042
q10	3618	3584	3554	3554
q11	608	517	504	504
q12	825	622	631	622
q13	11070	3060	3203	3060
q14	303	291	292	291
q15	582	517	499	499
q16	698	639	666	639
q17	1856	1658	1637	1637
q18	8302	7780	7695	7695
q19	1799	1504	1583	1504
q20	2097	1838	1861	1838
q21	5863	5512	5438	5438
q22	677	610	589	589
Total cold run time: 71381 ms
Total hot run time: 60137 ms

doris-robot · 2024-12-24T02:57:37Z

TeamCity be ut coverage result:
Function Coverage: 38.80% (10094/26016)
Line Coverage: 29.78% (85119/285873)
Region Coverage: 28.90% (43462/150407)
Branch Coverage: 25.43% (22154/87120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a1f8dc2f89121851216a36a37c9e3b321998bc68_a1f8dc2f89121851216a36a37c9e3b321998bc68/report/index.html

doris-robot · 2024-12-24T02:58:49Z

TPC-DS: Total hot run time: 197250 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a1f8dc2f89121851216a36a37c9e3b321998bc68, data reload: false
query1	1304	1004	920	920
query2	6227	2323	2308	2308
query3	10963	4716	4527	4527
query4	33583	23923	23475	23475
query5	4809	470	466	466
query6	284	192	214	192
query7	4011	310	308	308
query8	329	250	242	242
query9	9534	2713	2715	2713
query10	446	250	242	242
query11	18115	15094	15154	15094
query12	160	104	100	100
query13	1585	420	408	408
query14	10988	7399	7558	7399
query15	266	181	189	181
query16	8081	515	467	467
query17	1642	577	582	577
query18	2190	308	327	308
query19	352	157	166	157
query20	127	116	119	116
query21	209	106	124	106
query22	4951	4428	4616	4428
query23	35653	33773	33717	33717
query24	10514	2586	2475	2475
query25	662	414	433	414
query26	1420	162	157	157
query27	2770	337	349	337
query28	7904	2493	2518	2493
query29	949	439	444	439
query30	238	151	158	151
query31	1098	861	848	848
query32	99	60	59	59
query33	755	299	292	292
query34	1140	530	523	523
query35	899	775	761	761
query36	1117	960	947	947
query37	145	85	74	74
query38	4390	4346	4510	4346
query39	1516	1515	1471	1471
query40	264	106	105	105
query41	45	46	44	44
query42	116	108	103	103
query43	545	506	504	504
query44	1275	826	836	826
query45	195	174	187	174
query46	1184	717	719	717
query47	2071	1962	1925	1925
query48	429	338	349	338
query49	867	395	390	390
query50	841	398	413	398
query51	7434	7184	7134	7134
query52	110	99	93	93
query53	261	195	203	195
query54	1010	413	414	413
query55	86	80	80	80
query56	290	247	258	247
query57	1331	1178	1148	1148
query58	241	232	242	232
query59	3397	3109	3114	3109
query60	288	247	250	247
query61	108	109	107	107
query62	844	663	694	663
query63	217	193	193	193
query64	5003	695	653	653
query65	3340	3310	3289	3289
query66	1104	313	310	310
query67	16493	15725	15581	15581
query68	6601	553	543	543
query69	472	255	263	255
query70	1244	1165	1098	1098
query71	519	269	252	252
query72	6929	4107	4114	4107
query73	799	370	362	362
query74	10205	8828	8803	8803
query75	3792	2628	2683	2628
query76	4362	1139	1019	1019
query77	634	266	292	266
query78	10264	9470	9781	9470
query79	1510	598	595	595
query80	1076	437	419	419
query81	535	247	228	228
query82	223	125	121	121
query83	209	146	145	145
query84	286	72	67	67
query85	965	382	304	304
query86	357	298	281	281
query87	4618	4418	4405	4405
query88	3831	2243	2222	2222
query89	430	291	289	289
query90	2145	191	193	191
query91	145	110	107	107
query92	65	51	57	51
query93	1689	550	556	550
query94	748	281	279	279
query95	360	260	318	260
query96	635	270	275	270
query97	2904	2698	2752	2698
query98	220	197	192	192
query99	1588	1317	1321	1317
Total cold run time: 311174 ms
Total hot run time: 197250 ms

doris-robot · 2024-12-24T03:04:23Z

ClickBench: Total hot run time: 33.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a1f8dc2f89121851216a36a37c9e3b321998bc68, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.24	0.08	0.07
query4	1.88	0.11	0.11
query5	0.43	0.43	0.39
query6	1.18	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.55
query10	0.55	0.58	0.56
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.61	0.60
query14	2.75	2.73	2.74
query15	0.90	0.83	0.83
query16	0.39	0.38	0.39
query17	1.08	1.06	1.05
query18	0.24	0.21	0.21
query19	1.89	1.91	2.05
query20	0.02	0.01	0.01
query21	15.36	0.58	0.57
query22	2.72	2.18	2.53
query23	16.86	1.07	0.83
query24	3.38	1.88	1.24
query25	0.31	0.15	0.17
query26	0.40	0.14	0.13
query27	0.05	0.04	0.04
query28	9.49	1.11	1.08
query29	12.57	3.23	3.26
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.23	0.48	0.48
query33	3.06	3.14	3.10
query34	16.88	4.40	4.46
query35	4.54	4.47	4.47
query36	0.71	0.49	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.39 s
Total hot run time: 33.48 s

… writer

airborne12 · 2024-12-25T01:17:56Z

run buildall

qidaye

LGTM

github-actions · 2024-12-25T01:58:25Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-25T01:58:27Z

PR approved by anyone and no changes requested.

doris-robot · 2024-12-25T02:23:41Z

TPC-H: Total hot run time: 32746 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 57a827a6aa122b8a67869aa107053022a4a9d933, data reload: false
------ Round 1 ----------------------------------
q1	17619	6182	6069	6069
q2	2068	302	161	161
q3	10444	1292	778	778
q4	10283	898	445	445
q5	9082	2245	1985	1985
q6	208	186	150	150
q7	892	760	606	606
q8	9255	1380	1178	1178
q9	5303	4897	4912	4897
q10	6789	2344	1900	1900
q11	471	271	252	252
q12	351	401	224	224
q13	17790	3590	2934	2934
q14	240	239	234	234
q15	566	498	503	498
q16	628	614	579	579
q17	571	855	327	327
q18	7094	6438	6484	6438
q19	1972	971	577	577
q20	312	322	190	190
q21	2879	2229	2020	2020
q22	365	338	304	304
Total cold run time: 105182 ms
Total hot run time: 32746 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6262	6211	6211	6211
q2	243	329	227	227
q3	2262	2692	2267	2267
q4	1486	1887	1398	1398
q5	4378	4807	4961	4807
q6	197	174	143	143
q7	2145	2011	1879	1879
q8	2667	2814	2678	2678
q9	7309	7318	7417	7318
q10	3102	3363	2794	2794
q11	593	513	491	491
q12	711	781	589	589
q13	3444	3753	3083	3083
q14	299	316	286	286
q15	582	499	495	495
q16	654	707	648	648
q17	1257	1742	1243	1243
q18	7703	7620	7367	7367
q19	856	1225	1131	1131
q20	2045	2041	1919	1919
q21	5668	5271	4977	4977
q22	628	617	585	585
Total cold run time: 54491 ms
Total hot run time: 52536 ms

doris-robot · 2024-12-25T02:28:26Z

TeamCity be ut coverage result:
Function Coverage: 38.80% (10095/26017)
Line Coverage: 29.79% (85165/285915)
Region Coverage: 28.91% (43495/150436)
Branch Coverage: 25.45% (22178/87142)
Coverage Report: http://coverage.selectdb-in.cc/coverage/57a827a6aa122b8a67869aa107053022a4a9d933_57a827a6aa122b8a67869aa107053022a4a9d933/report/index.html

doris-robot · 2024-12-25T02:36:11Z

TPC-DS: Total hot run time: 198228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 57a827a6aa122b8a67869aa107053022a4a9d933, data reload: false
query1	1301	958	950	950
query2	6499	2388	2388	2388
query3	11010	4742	4939	4742
query4	32779	23940	23554	23554
query5	3638	630	478	478
query6	284	203	198	198
query7	3981	486	302	302
query8	288	238	231	231
query9	9262	2735	2720	2720
query10	457	299	252	252
query11	17902	15316	15302	15302
query12	159	118	104	104
query13	1569	556	423	423
query14	10068	6956	7352	6956
query15	255	221	193	193
query16	8083	646	470	470
query17	1588	760	639	639
query18	2146	404	334	334
query19	202	184	169	169
query20	137	113	122	113
query21	214	119	103	103
query22	4523	4759	4436	4436
query23	34684	33736	34635	33736
query24	7318	2291	2272	2272
query25	468	463	406	406
query26	1036	273	165	165
query27	2533	482	337	337
query28	5873	2481	2507	2481
query29	550	556	428	428
query30	230	217	154	154
query31	1004	936	860	860
query32	77	67	56	56
query33	493	374	305	305
query34	825	878	519	519
query35	852	825	765	765
query36	1024	1043	993	993
query37	117	103	85	85
query38	4403	4437	4339	4339
query39	1515	1473	1476	1473
query40	203	117	104	104
query41	48	43	48	43
query42	121	114	103	103
query43	532	526	500	500
query44	1342	828	821	821
query45	184	178	176	176
query46	891	1075	670	670
query47	1995	1992	1951	1951
query48	386	421	327	327
query49	761	517	408	408
query50	638	683	416	416
query51	7325	7383	7211	7211
query52	108	111	98	98
query53	231	263	187	187
query54	509	523	441	441
query55	87	88	84	84
query56	285	304	270	270
query57	1226	1219	1166	1166
query58	233	225	232	225
query59	3139	3261	3178	3178
query60	288	286	292	286
query61	109	112	110	110
query62	922	823	751	751
query63	230	202	203	202
query64	4212	1040	675	675
query65	3375	3246	3270	3246
query66	1036	421	323	323
query67	16437	15903	15704	15704
query68	9996	740	490	490
query69	482	301	246	246
query70	1275	1176	1154	1154
query71	431	283	250	250
query72	5874	4032	3979	3979
query73	1723	773	375	375
query74	10124	9421	9253	9253
query75	4535	3173	2639	2639
query76	5611	1186	810	810
query77	1007	345	282	282
query78	10902	10324	9513	9513
query79	5595	842	581	581
query80	615	518	429	429
query81	470	271	231	231
query82	214	153	122	122
query83	196	159	142	142
query84	282	91	78	78
query85	740	355	301	301
query86	344	323	303	303
query87	4690	4413	4526	4413
query88	3777	2230	2202	2202
query89	414	322	300	300
query90	2093	184	187	184
query91	142	132	102	102
query92	69	55	55	55
query93	2378	741	525	525
query94	661	391	276	276
query95	342	270	251	251
query96	487	636	283	283
query97	2727	2800	2719	2719
query98	220	201	231	201
query99	1614	1562	1432	1432
Total cold run time: 306230 ms
Total hot run time: 198228 ms

doris-robot · 2024-12-25T02:41:31Z

ClickBench: Total hot run time: 31.34 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 57a827a6aa122b8a67869aa107053022a4a9d933, data reload: false
query1	0.03	0.04	0.05
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.11
query5	0.40	0.43	0.39
query6	1.18	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.56	0.58	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.82	2.85	2.74
query15	0.90	0.82	0.84
query16	0.39	0.40	0.39
query17	1.00	1.04	1.08
query18	0.22	0.20	0.21
query19	1.97	1.79	2.00
query20	0.02	0.01	0.01
query21	15.37	0.93	0.58
query22	0.75	0.83	0.60
query23	15.35	1.46	0.59
query24	2.83	1.29	1.04
query25	0.21	0.10	0.16
query26	0.37	0.14	0.14
query27	0.06	0.05	0.06
query28	13.77	1.50	1.04
query29	12.58	4.00	3.34
query30	0.25	0.09	0.07
query31	2.81	0.60	0.37
query32	3.24	0.55	0.45
query33	3.07	3.07	3.10
query34	16.83	5.08	4.46
query35	4.51	4.48	4.49
query36	0.62	0.50	0.47
query37	0.10	0.06	0.05
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.08 s
Total hot run time: 31.34 s

… writer

airborne12 · 2024-12-26T09:20:21Z

run buildall

zzzxl1993

LGTM

github-actions · 2024-12-26T09:56:30Z

PR approved by at least one committer and no changes requested.

doris-robot · 2024-12-26T10:13:17Z

TPC-H: Total hot run time: 32766 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f55790900ac9cb8c705f362b43bf3cc77d66e9b6, data reload: false
------ Round 1 ----------------------------------
q1	17639	6181	6056	6056
q2	2056	321	181	181
q3	10529	1257	749	749
q4	10273	874	438	438
q5	8543	2246	2028	2028
q6	210	184	148	148
q7	898	754	601	601
q8	9239	1398	1197	1197
q9	5378	5006	4939	4939
q10	6794	2321	1861	1861
q11	506	289	258	258
q12	342	362	230	230
q13	17790	3633	2934	2934
q14	239	240	221	221
q15	578	502	500	500
q16	637	624	595	595
q17	600	873	339	339
q18	7165	6529	6448	6448
q19	2313	989	572	572
q20	304	316	185	185
q21	2913	2187	1978	1978
q22	366	345	308	308
Total cold run time: 105312 ms
Total hot run time: 32766 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6360	6209	6231	6209
q2	236	335	238	238
q3	2281	2649	2299	2299
q4	1413	1867	1358	1358
q5	4342	4811	4936	4811
q6	187	177	143	143
q7	2102	1952	1802	1802
q8	2635	2804	2717	2717
q9	7456	7280	7407	7280
q10	3124	3379	2793	2793
q11	573	513	512	512
q12	736	786	642	642
q13	3401	3852	3101	3101
q14	288	302	290	290
q15	571	518	565	518
q16	670	732	663	663
q17	1250	1772	1252	1252
q18	7920	7564	7340	7340
q19	871	1143	1122	1122
q20	2020	2051	1908	1908
q21	5812	5164	5010	5010
q22	664	622	615	615
Total cold run time: 54912 ms
Total hot run time: 52623 ms

doris-robot · 2024-12-26T10:24:12Z

TeamCity be ut coverage result:
Function Coverage: 38.83% (10104/26021)
Line Coverage: 29.84% (85352/286061)
Region Coverage: 28.98% (43619/150523)
Branch Coverage: 25.52% (22245/87184)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f55790900ac9cb8c705f362b43bf3cc77d66e9b6_f55790900ac9cb8c705f362b43bf3cc77d66e9b6/report/index.html

doris-robot · 2024-12-26T10:25:44Z

TPC-DS: Total hot run time: 197754 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f55790900ac9cb8c705f362b43bf3cc77d66e9b6, data reload: false
query1	1317	1026	909	909
query2	6503	2566	2567	2566
query3	10969	4624	4605	4605
query4	33217	23692	23470	23470
query5	4413	624	488	488
query6	296	205	208	205
query7	3980	497	312	312
query8	313	255	249	249
query9	9537	2746	2741	2741
query10	463	324	248	248
query11	17913	15434	15260	15260
query12	168	110	106	106
query13	1576	571	431	431
query14	11132	6865	7707	6865
query15	230	208	199	199
query16	7815	590	450	450
query17	1562	756	575	575
query18	2123	409	315	315
query19	214	187	159	159
query20	120	119	113	113
query21	204	129	110	110
query22	4939	4694	4498	4498
query23	34515	34705	34152	34152
query24	6370	2321	2378	2321
query25	463	456	393	393
query26	770	253	161	161
query27	2090	465	330	330
query28	5687	2512	2505	2505
query29	552	548	428	428
query30	210	184	150	150
query31	1012	944	873	873
query32	72	64	66	64
query33	492	361	325	325
query34	780	869	522	522
query35	817	835	776	776
query36	1050	1056	983	983
query37	128	110	80	80
query38	4427	4229	4210	4210
query39	1524	1465	1484	1465
query40	215	134	111	111
query41	52	49	47	47
query42	128	105	110	105
query43	591	560	520	520
query44	1407	827	835	827
query45	192	187	178	178
query46	904	1074	672	672
query47	1971	2011	1938	1938
query48	400	437	347	347
query49	722	481	398	398
query50	656	697	389	389
query51	7389	7148	7182	7148
query52	104	102	99	99
query53	238	263	188	188
query54	501	533	414	414
query55	80	80	81	80
query56	253	264	248	248
query57	1259	1256	1187	1187
query58	238	223	231	223
query59	3359	3523	3288	3288
query60	280	265	266	265
query61	107	105	114	105
query62	880	821	745	745
query63	244	202	203	202
query64	3173	1061	689	689
query65	3396	3235	3249	3235
query66	876	438	303	303
query67	16567	16057	15612	15612
query68	9631	775	507	507
query69	497	299	254	254
query70	1228	1097	1124	1097
query71	444	292	253	253
query72	5829	3931	3864	3864
query73	834	784	356	356
query74	10343	9270	9100	9100
query75	4709	3162	2628	2628
query76	5603	1198	770	770
query77	1016	354	272	272
query78	10156	10345	9385	9385
query79	3654	908	586	586
query80	750	531	434	434
query81	486	274	222	222
query82	480	155	127	127
query83	199	160	172	160
query84	282	93	79	79
query85	747	398	300	300
query86	341	320	279	279
query87	4515	4678	4357	4357
query88	3073	2241	2220	2220
query89	442	329	291	291
query90	2110	186	185	185
query91	133	135	106	106
query92	64	54	54	54
query93	2079	893	525	525
query94	675	384	283	283
query95	350	263	253	253
query96	495	621	286	286
query97	2733	2856	2734	2734
query98	230	199	195	195
query99	1707	1553	1417	1417
Total cold run time: 301980 ms
Total hot run time: 197754 ms

doris-robot · 2024-12-26T10:31:04Z

ClickBench: Total hot run time: 31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f55790900ac9cb8c705f362b43bf3cc77d66e9b6, data reload: false
query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.43	0.43	0.43
query6	1.16	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.52
query10	0.56	0.57	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	2.85	2.71	2.83
query15	0.90	0.82	0.83
query16	0.42	0.38	0.39
query17	1.08	0.98	0.99
query18	0.23	0.22	0.20
query19	1.85	1.88	2.02
query20	0.01	0.01	0.02
query21	15.37	0.88	0.58
query22	0.74	0.81	0.71
query23	15.24	1.41	0.53
query24	3.30	0.63	0.87
query25	0.23	0.12	0.12
query26	0.31	0.15	0.15
query27	0.04	0.05	0.06
query28	13.59	1.55	1.05
query29	12.55	3.91	3.21
query30	0.26	0.09	0.06
query31	2.81	0.60	0.39
query32	3.23	0.54	0.46
query33	3.09	3.26	3.20
query34	16.89	5.08	4.46
query35	4.48	4.42	4.47
query36	0.65	0.49	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.27 s
Total hot run time: 31 s

… writer (apache#45833) Issue Number: close #xxx Related PR: #xxx Problem Summary: Optimize memory usage when adding string values for bloom filter index. Using uint64 hash value instead of string values itself, it is expected to save a lot of memory for especially long text

…filter index writer #45833 (#46048) cherry pick from #45833

… writer #45833 (#46047) cherry pick from #45833

… writer apache#45833 (apache#46047) cherry pick from apache#45833

[opt](bloomfilter index) optimize memory usage for bloom filter index…
57a827a
… writer

airborne12 force-pushed the opt-bf-memory branch from a1f8dc2 to 57a827aCompare December 24, 2024 10:01

xiaokang reviewed Dec 24, 2024
View reviewed changes

Comment threadbe/src/olap/rowset/segment_v2/bloom_filter.h
Comment threadbe/src/olap/rowset/segment_v2/bloom_filter.h

eldenmoon added dev/2.1.x dev/3.0.x labels Dec 25, 2024

qidaye previously approved these changes Dec 25, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 25, 2024

github-actionsBot added the reviewed label Dec 25, 2024

[opt](bloomfilter index) optimize memory usage for bloom filter index…
f557909
… writer

airborne12 dismissed qidaye’s stale review via f557909December 26, 2024 09:18

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

zzzxl1993 approved these changes Dec 26, 2024
View reviewed changes

qidaye approved these changes Dec 26, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024

airborne12 merged commit 30ebe42 into apache:masterDec 26, 2024

airborne12 deleted the opt-bf-memory branch December 26, 2024 11:42

github-actionsBot added dev/3.0.x-conflict dev/2.1.x-conflict labels Dec 26, 2024

airborne12 mentioned this pull request Dec 26, 2024
[opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833 #46047
Merged

airborne12 mentioned this pull request Dec 26, 2024
branch-3.0: [opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833 #46048
Merged

airborne12 added a commit that referenced this pull request Dec 27, 2024
branch-3.0: [opt](bloomfilter index) optimize memory usage for bloom …
cac25be
…filter index writer #45833 (#46048) cherry pick from #45833

airborne12 added dev/3.0.4-merged and removed dev/3.0.x dev/3.0.x-conflict labels Dec 27, 2024

airborne12 added a commit that referenced this pull request Dec 27, 2024
[opt](bloomfilter index) optimize memory usage for bloom filter index…
d2c1087
… writer #45833 (#46047) cherry pick from #45833

airborne12 added dev/2.1.8-merged and removed dev/2.1.x dev/2.1.x-conflict labels Dec 27, 2024

deardeng pushed a commit to deardeng/incubator-doris that referenced this pull request Jan 6, 2025
[opt](bloomfilter index) optimize memory usage for bloom filter index…
a11209a
… writer apache#45833 (apache#46047) cherry pick from apache#45833

gavinchou mentioned this pull request Feb 18, 2025
Release Notes 3.0.4 #48013
Closed

Uh oh!

Conversation

airborne12 commented Dec 24, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 24, 2024

Uh oh!

airborne12 commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

doris-robot commented Dec 24, 2024

Uh oh!

Uh oh!

Uh oh!

airborne12 commented Dec 25, 2024

Uh oh!

qidaye left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 25, 2024

Uh oh!

github-actionsBot commented Dec 25, 2024

Uh oh!

doris-robot commented Dec 25, 2024

Uh oh!

doris-robot commented Dec 25, 2024

Uh oh!

doris-robot commented Dec 25, 2024

Uh oh!

doris-robot commented Dec 25, 2024

Uh oh!

airborne12 commented Dec 26, 2024

Uh oh!

zzzxl1993 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

doris-robot commented Dec 26, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

airborne12 commented Dec 24, 2024•
edited
Loading