Skip to content

branch-3.0: [Bug](hash) fix wrong HashLen16 implement and add cityhash64 unit test #46928 - #46948

Merged
BiteTheDDDDt merged 1 commit into
branch-3.0from
auto-pick-46928-branch-3.0
Jan 14, 2025
Merged

branch-3.0: [Bug](hash) fix wrong HashLen16 implement and add cityhash64 unit test #46928#46948
BiteTheDDDDt merged 1 commit into
branch-3.0from
auto-pick-46928-branch-3.0

Conversation

@github-actions

@github-actionsgithub-actionsBot commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

Cherry-picked from #46928

@BiteTheDDDDtYour Name
[Bug](hash) fix wrong HashLen16 implement and add cityhash64 unit test (
ee5f442
#46928)
### What problem does this PR solve?
fix wrtong HashLen16 implement and add cityhash64 unit test
HashLen16 was incorrectly modified in #35841, which would cause ngram
indexes built in 2.0/2.1 to get incorrect results in 3.0
@Thearas

Thearas commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Thearas

Thearas commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

run buildall

airborne12
airborne12 approved these changes Jan 14, 2025
View reviewed changes

@airborne12airborne12 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jan 14, 2025

Copy link
Copy Markdown
ContributorAuthor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 14, 2025
@github-actions

github-actionsBot commented Jan 14, 2025

Copy link
Copy Markdown
ContributorAuthor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 41302 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
------ Round 1 ----------------------------------
q1	17581	7430	7283	7283
q2	2063	171	174	171
q3	10603	1161	1174	1161
q4	10274	763	808	763
q5	7800	2881	2898	2881
q6	235	148	151	148
q7	978	618	590	590
q8	9358	2101	2058	2058
q9	6641	6382	6423	6382
q10	7003	2292	2279	2279
q11	466	261	269	261
q12	407	218	212	212
q13	17786	2995	3025	2995
q14	246	216	215	215
q15	580	529	517	517
q16	689	622	611	611
q17	988	613	639	613
q18	7614	6788	6736	6736
q19	1393	1092	1110	1092
q20	477	200	197	197
q21	3980	3147	3203	3147
q22	1076	991	990	990
Total cold run time: 108238 ms
Total hot run time: 41302 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7239	7261	7260	7260
q2	331	229	237	229
q3	2982	3049	3074	3049
q4	2045	1780	1805	1780
q5	5693	5790	5737	5737
q6	218	138	139	138
q7	2218	1775	1866	1775
q8	3422	3506	3505	3505
q9	8878	8817	8894	8817
q10	3558	3560	3560	3560
q11	597	506	502	502
q12	811	624	674	624
q13	9586	3106	3234	3106
q14	294	276	266	266
q15	578	538	520	520
q16	706	673	662	662
q17	1831	1642	1626	1626
q18	8192	7734	7703	7703
q19	1684	1492	1536	1492
q20	2119	1839	1841	1839
q21	5396	5255	5162	5162
q22	1112	1012	1006	1006
Total cold run time: 69490 ms
Total hot run time: 60358 ms

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191459 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
query1	993	394	372	372
query2	6513	2144	2091	2091
query3	6706	213	225	213
query4	33707	23406	23358	23358
query5	4319	453	448	448
query6	277	174	190	174
query7	4621	313	316	313
query8	285	233	235	233
query9	9571	2654	2653	2653
query10	486	263	261	261
query11	18035	15435	15050	15050
query12	154	104	104	104
query13	1646	438	417	417
query14	10198	7297	6901	6901
query15	256	178	175	175
query16	8079	427	461	427
query17	1626	578	582	578
query18	2118	311	326	311
query19	272	160	157	157
query20	114	108	113	108
query21	212	105	103	103
query22	4510	4196	4131	4131
query23	35066	34203	33963	33963
query24	11322	2831	2874	2831
query25	696	407	406	406
query26	1437	173	175	173
query27	2917	345	349	345
query28	7988	2419	2431	2419
query29	943	463	442	442
query30	328	170	167	167
query31	1036	822	830	822
query32	100	62	66	62
query33	793	303	301	301
query34	959	490	519	490
query35	873	745	747	745
query36	1106	950	952	950
query37	143	77	76	76
query38	4054	3920	3849	3849
query39	1496	1457	1451	1451
query40	292	105	102	102
query41	56	52	53	52
query42	113	100	103	100
query43	554	489	494	489
query44	1240	827	813	813
query45	183	170	174	170
query46	1150	704	703	703
query47	1914	1820	1813	1813
query48	469	388	375	375
query49	1208	426	416	416
query50	813	418	418	418
query51	7310	6918	7098	6918
query52	105	92	91	91
query53	259	187	192	187
query54	1226	489	461	461
query55	78	79	82	79
query56	291	274	267	267
query57	1225	1148	1089	1089
query58	243	216	227	216
query59	3270	3044	3218	3044
query60	318	253	252	252
query61	115	117	150	117
query62	893	699	681	681
query63	235	195	183	183
query64	5314	684	647	647
query65	3259	3208	3197	3197
query66	1457	310	325	310
query67	16111	15741	15549	15549
query68	4464	590	573	573
query69	420	270	290	270
query70	1181	1130	1120	1120
query71	341	268	263	263
query72	6411	4028	4065	4028
query73	741	342	355	342
query74	10417	8984	8880	8880
query75	3396	2626	2647	2626
query76	2723	1119	1209	1119
query77	453	285	278	278
query78	10593	9744	9567	9567
query79	1479	619	613	613
query80	1191	441	436	436
query81	550	242	246	242
query82	937	123	124	123
query83	228	155	151	151
query84	243	82	83	82
query85	1356	309	302	302
query86	378	302	301	301
query87	4328	4305	4265	4265
query88	3417	2382	2336	2336
query89	404	290	296	290
query90	2020	191	190	190
query91	187	152	151	151
query92	62	51	55	51
query93	1096	553	542	542
query94	987	312	297	297
query95	356	265	270	265
query96	595	275	287	275
query97	3280	3180	3183	3180
query98	216	202	200	200
query99	1496	1318	1292	1292
Total cold run time: 301602 ms
Total hot run time: 191459 ms

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 33.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.61	0.11	0.11
query5	0.53	0.50	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.03	0.04
query9	0.56	0.51	0.50
query10	0.55	0.55	0.55
query11	0.15	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.97	3.11	2.93
query15	0.90	0.84	0.83
query16	0.37	0.39	0.38
query17	1.08	1.01	0.99
query18	0.23	0.22	0.22
query19	1.87	1.83	2.11
query20	0.01	0.00	0.01
query21	15.36	0.62	0.60
query22	2.60	1.97	2.67
query23	16.93	0.92	0.70
query24	2.96	1.58	1.21
query25	0.26	0.24	0.15
query26	0.33	0.14	0.15
query27	0.04	0.06	0.04
query28	10.45	1.12	1.07
query29	12.57	3.26	3.26
query30	0.24	0.07	0.06
query31	2.87	0.38	0.37
query32	3.24	0.46	0.47
query33	2.99	3.02	3.03
query34	17.06	4.44	4.44
query35	4.48	4.56	4.55
query36	0.66	0.52	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.64 s
Total hot run time: 33.35 s

@BiteTheDDDDt

BiteTheDDDDt commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

Failed case depends on #46974

@BiteTheDDDDt

BiteTheDDDDt commented Jan 14, 2025

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40863 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
------ Round 1 ----------------------------------
q1	17580	7391	7223	7223
q2	2061	179	201	179
q3	10566	1099	1200	1099
q4	10473	723	694	694
q5	7770	2874	2830	2830
q6	241	153	151	151
q7	1009	608	590	590
q8	9351	1949	2042	1949
q9	6677	6385	6456	6385
q10	7031	2278	2281	2278
q11	461	273	280	273
q12	400	226	220	220
q13	17787	2982	3010	2982
q14	232	210	211	210
q15	571	519	542	519
q16	691	603	619	603
q17	986	565	561	561
q18	7369	6832	6715	6715
q19	1416	1082	1064	1064
q20	468	216	206	206
q21	4221	3274	3191	3191
q22	1086	958	941	941
Total cold run time: 108447 ms
Total hot run time: 40863 ms
----- Round 2, with runtime_filter_mode=off -----
q1	7233	7253	7241	7241
q2	329	237	224	224
q3	2908	2940	2965	2940
q4	2008	1816	1804	1804
q5	5781	5738	5739	5738
q6	224	140	145	140
q7	2265	1850	1827	1827
q8	3411	3555	3522	3522
q9	8841	8921	8895	8895
q10	3559	3576	3546	3546
q11	609	508	509	508
q12	808	619	610	610
q13	9182	3238	3112	3112
q14	294	280	265	265
q15	584	521	518	518
q16	711	674	683	674
q17	1856	1639	1606	1606
q18	8296	7726	7734	7726
q19	1708	1595	1626	1595
q20	2045	1850	1873	1850
q21	5483	5358	5364	5358
q22	1140	1083	1038	1038
Total cold run time: 69275 ms
Total hot run time: 60737 ms

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 198872 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
query1	1306	910	900	900
query2	6319	2076	2014	2014
query3	10796	4314	4179	4179
query4	61911	29139	23530	23530
query5	5199	468	463	463
query6	417	205	182	182
query7	5440	321	311	311
query8	319	223	221	221
query9	8760	2702	2690	2690
query10	483	282	280	280
query11	17364	15351	15773	15351
query12	161	109	101	101
query13	1430	494	445	445
query14	9831	7707	7573	7573
query15	204	176	183	176
query16	7103	525	481	481
query17	1177	589	593	589
query18	1860	344	322	322
query19	215	168	163	163
query20	122	117	110	110
query21	206	99	103	99
query22	4551	4557	4766	4557
query23	34760	34208	34435	34208
query24	6145	2989	2909	2909
query25	530	390	406	390
query26	656	169	171	169
query27	1886	347	357	347
query28	4256	2572	2494	2494
query29	711	480	476	476
query30	251	175	173	173
query31	1012	828	831	828
query32	67	58	58	58
query33	414	286	287	286
query34	929	528	528	528
query35	842	751	753	751
query36	1085	940	972	940
query37	127	80	73	73
query38	4223	4033	3991	3991
query39	1494	1470	1456	1456
query40	202	104	96	96
query41	53	51	51	51
query42	114	99	100	99
query43	534	491	489	489
query44	1208	843	838	838
query45	188	170	170	170
query46	1143	726	735	726
query47	2042	1923	1941	1923
query48	485	386	373	373
query49	732	416	395	395
query50	839	434	443	434
query51	7358	7416	7321	7321
query52	110	92	94	92
query53	274	206	201	201
query54	611	468	469	468
query55	79	78	77	77
query56	268	261	243	243
query57	1292	1158	1187	1158
query58	218	228	214	214
query59	3337	2912	2921	2912
query60	280	263	251	251
query61	121	107	131	107
query62	767	664	668	664
query63	210	191	186	186
query64	1361	682	634	634
query65	3256	3194	3178	3178
query66	692	296	333	296
query67	16208	15580	15675	15580
query68	4223	611	576	576
query69	426	263	267	263
query70	1166	1130	1145	1130
query71	347	256	258	256
query72	6333	4170	4212	4170
query73	758	350	360	350
query74	10018	8854	9024	8854
query75	3365	2664	2677	2664
query76	2007	1113	1048	1048
query77	530	301	308	301
query78	10558	9723	9641	9641
query79	2237	603	609	603
query80	1428	427	433	427
query81	527	238	238	238
query82	1230	125	114	114
query83	195	143	146	143
query84	285	79	81	79
query85	1005	301	288	288
query86	383	299	261	261
query87	4564	4373	4339	4339
query88	3811	2420	2406	2406
query89	417	289	289	289
query90	1983	187	180	180
query91	177	147	145	145
query92	65	51	50	50
query93	2756	560	559	559
query94	792	309	293	293
query95	362	258	258	258
query96	637	284	285	284
query97	3313	3226	3188	3188
query98	218	216	201	201
query99	1628	1308	1284	1284
Total cold run time: 316288 ms
Total hot run time: 198872 ms

@doris-robot

doris-robot commented Jan 14, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 33.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee5f442b92f8db6a1db8ef86ea7e4d84100e89f5, data reload: false
query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.10	0.10
query5	0.51	0.51	0.51
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.06	0.03	0.04
query9	0.56	0.50	0.49
query10	0.55	0.56	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.93	2.94	2.93
query15	0.92	0.83	0.82
query16	0.38	0.38	0.41
query17	1.08	1.04	0.98
query18	0.24	0.21	0.21
query19	1.99	1.89	2.02
query20	0.01	0.01	0.01
query21	15.36	0.60	0.59
query22	2.39	2.13	2.05
query23	17.10	0.95	0.77
query24	2.85	1.38	1.21
query25	0.34	0.10	0.19
query26	0.34	0.14	0.14
query27	0.04	0.04	0.04
query28	10.16	1.11	1.07
query29	12.58	3.25	3.22
query30	0.24	0.06	0.06
query31	2.85	0.40	0.39
query32	3.22	0.46	0.46
query33	3.01	3.00	3.05
query34	16.59	4.51	4.44
query35	4.55	4.48	4.49
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.01 s
Total hot run time: 33.36 s

@BiteTheDDDDt
BiteTheDDDDt merged commit f41ac10 into branch-3.0Jan 14, 2025
@github-actions
github-actionsBot deleted the auto-pick-46928-branch-3.0 branch January 14, 2025 16:07
hubgeter pushed a commit to hubgeter/doris that referenced this pull request Mar 12, 2025
@BiteTheDDDDt
branch-3.0: [Bug](hash) fix wrong HashLen16 implement and add cityha…
ef47ad1
…sh64 unit test apache#46928 (apache#3721)
pick from apache#46948
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12airborne12airborne12 approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/3.0.4-mergedreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Thearas@doris-robot@BiteTheDDDDt@airborne12@yiguolei@dataroaring