[fix](inverted index)Support Chinese column name with inverted index #36321 by qidaye · Pull Request #36375 · apache/doris

qidaye · 2024-06-17T04:26:25Z

std::string to std::wstring conversion only supports ASCII characters. For non-ASCII characters, we need to use StringUtil::string_to_wstring
Fix index_tool check_terms_stats_v2 and add field info to print

Issue Number: #34118
pick from master #36321

…pache#36321) 1. `std::string` to `std::wstring` conversion only supports ASCII characters. For non-ASCII characters, we need to use `StringUtil::string_to_wstring` 2. Fix index_tool check_terms_stats_v2 and add field info to print Issue Number: apache#34118

doris-robot · 2024-06-17T04:26:30Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

qidaye · 2024-06-17T04:26:35Z

run buildall

github-actions · 2024-06-17T04:33:15Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-17T04:50:33Z

TPC-H: Total hot run time: 50193 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e2d79dcac63cc36dcd0163739b72afca6cc442fe, data reload: false
------ Round 1 ----------------------------------
q1	17531	4413	4343	4343
q2	2062	153	146	146
q3	10271	2199	1919	1919
q4	10113	1263	1304	1263
q5	8525	3919	3934	3919
q6	222	126	125	125
q7	2071	1603	1613	1603
q8	9272	2759	2728	2728
q9	10792	11105	10627	10627
q10	8661	3487	3518	3487
q11	426	250	247	247
q12	472	301	307	301
q13	18337	3940	4039	3940
q14	352	331	326	326
q15	505	459	460	459
q16	669	576	569	569
q17	1165	995	996	995
q18	7357	6824	6777	6777
q19	1844	1607	1647	1607
q20	534	303	296	296
q21	4409	4135	4078	4078
q22	526	456	438	438
Total cold run time: 116116 ms
Total hot run time: 50193 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4319	4330	4270	4270
q2	324	225	229	225
q3	4191	4193	4185	4185
q4	2777	2760	2772	2760
q5	7198	7165	7172	7165
q6	238	118	120	118
q7	3260	2869	2849	2849
q8	4375	4524	4526	4524
q9	16929	16779	16693	16693
q10	4187	4271	4228	4228
q11	785	705	692	692
q12	1027	882	872	872
q13	7022	3727	3739	3727
q14	453	429	421	421
q15	496	474	455	455
q16	735	694	698	694
q17	3871	3856	3871	3856
q18	8842	8671	8904	8671
q19	1707	1708	1668	1668
q20	2380	2091	2098	2091
q21	8518	8431	8450	8431
q22	1068	980	967	967
Total cold run time: 84702 ms
Total hot run time: 79562 ms

doris-robot · 2024-06-17T04:57:49Z

TeamCity be ut coverage result:
Function Coverage: 37.80% (8096/21416)
Line Coverage: 29.45% (66141/224607)
Region Coverage: 28.93% (34099/117874)
Branch Coverage: 24.80% (17503/70588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e2d79dcac63cc36dcd0163739b72afca6cc442fe_e2d79dcac63cc36dcd0163739b72afca6cc442fe/report/index.html

doris-robot · 2024-06-17T05:03:16Z

TPC-DS: Total hot run time: 203413 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e2d79dcac63cc36dcd0163739b72afca6cc442fe, data reload: false
query1	930	423	374	374
query2	6531	2890	2631	2631
query3	6927	207	203	203
query4	21327	17931	17880	17880
query5	19747	6516	6475	6475
query6	294	222	230	222
query7	4162	301	322	301
query8	442	431	439	431
query9	3120	2696	2614	2614
query10	406	292	303	292
query11	11387	10653	10816	10653
query12	130	78	74	74
query13	5594	698	682	682
query14	17587	13585	13644	13585
query15	364	242	243	242
query16	6478	290	262	262
query17	1691	1447	870	870
query18	2334	414	410	410
query19	207	153	149	149
query20	83	76	80	76
query21	193	99	103	99
query22	5207	5037	4930	4930
query23	32475	31946	31698	31698
query24	7049	6549	6546	6546
query25	504	420	420	420
query26	636	174	162	162
query27	2027	295	296	295
query28	6243	2380	2329	2329
query29	2942	2799	2815	2799
query30	266	163	164	163
query31	920	744	738	738
query32	71	62	60	60
query33	405	261	249	249
query34	847	469	493	469
query35	1162	912	943	912
query36	1314	1123	1277	1123
query37	91	64	64	64
query38	3103	2904	2902	2902
query39	1372	1344	1348	1344
query40	250	97	95	95
query41	46	46	42	42
query42	83	87	87	87
query43	734	687	676	676
query44	1135	719	731	719
query45	248	238	239	238
query46	1234	955	943	943
query47	2261	1671	1884	1671
query48	1033	717	707	707
query49	631	379	376	376
query50	861	629	632	629
query51	4779	4645	4739	4645
query52	90	80	74	74
query53	456	323	322	322
query54	2646	2453	2477	2453
query55	95	82	87	82
query56	237	226	203	203
query57	1200	1170	1082	1082
query58	210	220	203	203
query59	4009	4149	4167	4149
query60	206	192	195	192
query61	95	90	94	90
query62	819	459	470	459
query63	483	338	343	338
query64	2498	1536	1509	1509
query65	3636	3583	3565	3565
query66	826	368	377	368
query67	16365	15280	16787	15280
query68	8454	632	656	632
query69	568	362	355	355
query70	1646	1355	1432	1355
query71	392	307	313	307
query72	6514	3454	3517	3454
query73	734	320	317	317
query74	6322	5844	5822	5822
query75	4871	3750	3646	3646
query76	4968	1138	1100	1100
query77	706	260	258	258
query78	12684	11765	11516	11516
query79	8550	622	652	622
query80	2013	402	397	397
query81	511	241	244	241
query82	1636	103	95	95
query83	168	133	130	130
query84	267	70	72	70
query85	1083	328	314	314
query86	348	291	306	291
query87	3237	3058	3079	3058
query88	5208	2282	2283	2282
query89	402	285	299	285
query90	1834	214	207	207
query91	173	137	148	137
query92	63	53	52	52
query93	5299	547	551	547
query94	749	217	211	211
query95	1108	1064	1060	1060
query96	636	330	323	323
query97	6415	6415	6445	6415
query98	201	177	169	169
query99	2889	820	877	820
Total cold run time: 314383 ms
Total hot run time: 203413 ms

doris-robot · 2024-06-17T05:08:41Z

ClickBench: Total hot run time: 30.6 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e2d79dcac63cc36dcd0163739b72afca6cc442fe, data reload: false
query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.79	0.07	0.06
query5	0.54	0.53	0.52
query6	1.25	0.61	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.02
query9	0.54	0.49	0.48
query10	0.53	0.54	0.54
query11	0.12	0.08	0.09
query12	0.12	0.08	0.09
query13	0.61	0.61	0.62
query14	0.80	0.80	0.79
query15	0.78	0.76	0.77
query16	0.37	0.36	0.36
query17	0.98	1.00	1.03
query18	0.20	0.27	0.23
query19	1.90	1.89	1.82
query20	0.02	0.01	0.01
query21	15.49	0.53	0.56
query22	1.97	1.94	1.91
query23	17.15	0.95	1.04
query24	6.17	0.78	0.76
query25	0.34	0.11	0.05
query26	0.66	0.16	0.15
query27	0.04	0.04	0.04
query28	7.46	0.73	0.74
query29	12.62	2.16	2.31
query30	0.60	0.50	0.52
query31	2.81	0.38	0.37
query32	3.38	0.50	0.49
query33	3.07	3.04	3.04
query34	15.25	4.79	4.79
query35	4.84	4.86	4.83
query36	1.04	1.01	1.00
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.22 s
Total hot run time: 30.6 s

doris-robot · 2024-06-17T05:11:54Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e2d79dcac63cc36dcd0163739b72afca6cc442fe with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.8 seconds inserted 10000000 Rows, about 458K ops/s

qidaye · 2024-06-17T06:01:57Z

run p0

qidaye · 2024-06-17T06:30:22Z

run buildall

github-actions · 2024-06-17T06:36:17Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-06-17T07:01:16Z

TeamCity be ut coverage result:
Function Coverage: 37.78% (8092/21416)
Line Coverage: 29.43% (66104/224607)
Region Coverage: 28.90% (34070/117874)
Branch Coverage: 24.77% (17486/70588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852_502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852/report/index.html

doris-robot · 2024-06-17T07:03:44Z

TPC-H: Total hot run time: 50034 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852, data reload: false
------ Round 1 ----------------------------------
q1	17788	4361	4345	4345
q2	2086	161	145	145
q3	10459	1900	1926	1900
q4	10312	1265	1350	1265
q5	8307	3841	3939	3841
q6	229	149	141	141
q7	2045	1610	1597	1597
q8	9293	2728	2697	2697
q9	11052	10578	10532	10532
q10	8666	3484	3539	3484
q11	425	237	242	237
q12	465	305	306	305
q13	18380	3949	4003	3949
q14	347	336	332	332
q15	503	463	462	462
q16	685	577	573	573
q17	1122	978	1007	978
q18	7185	6927	6773	6773
q19	1768	1637	1605	1605
q20	556	309	295	295
q21	4488	4141	4148	4141
q22	522	437	463	437
Total cold run time: 116683 ms
Total hot run time: 50034 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4399	4411	4361	4361
q2	318	225	226	225
q3	4147	4145	4140	4140
q4	2756	2765	2764	2764
q5	7326	7175	7162	7162
q6	242	123	120	120
q7	3256	2841	2814	2814
q8	4400	4489	4519	4489
q9	17709	17173	17369	17173
q10	4262	4268	4316	4268
q11	754	704	709	704
q12	1052	856	865	856
q13	9101	3895	3868	3868
q14	500	454	441	441
q15	505	473	465	465
q16	804	713	715	713
q17	3846	4102	3877	3877
q18	9004	8956	8870	8870
q19	1745	1781	1672	1672
q20	2389	2125	2086	2086
q21	8448	8434	8453	8434
q22	1069	950	959	950
Total cold run time: 88032 ms
Total hot run time: 80452 ms

doris-robot · 2024-06-17T07:16:28Z

TPC-DS: Total hot run time: 203306 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852, data reload: false
query1	928	429	374	374
query2	6527	2800	2475	2475
query3	6921	208	197	197
query4	19976	17999	17981	17981
query5	19757	6575	6507	6507
query6	370	213	245	213
query7	4674	298	306	298
query8	419	422	388	388
query9	3030	2624	2573	2573
query10	407	300	313	300
query11	11318	10769	10751	10751
query12	125	78	79	78
query13	5594	691	686	686
query14	18742	13124	13465	13124
query15	365	231	230	230
query16	6448	282	275	275
query17	1343	1573	888	888
query18	2270	412	403	403
query19	218	150	153	150
query20	79	80	81	80
query21	192	102	98	98
query22	5215	5112	5002	5002
query23	32463	32019	31886	31886
query24	6894	6606	6591	6591
query25	537	439	431	431
query26	505	165	169	165
query27	1721	299	294	294
query28	6095	2358	2311	2311
query29	2922	2700	2822	2700
query30	236	172	168	168
query31	896	739	739	739
query32	69	64	63	63
query33	405	262	267	262
query34	858	468	467	467
query35	1168	869	980	869
query36	1254	1225	1329	1225
query37	91	61	64	61
query38	3025	2936	2975	2936
query39	1375	1343	1322	1322
query40	212	93	99	93
query41	48	43	45	43
query42	79	87	80	80
query43	764	730	728	728
query44	1213	724	731	724
query45	248	236	237	236
query46	1234	973	993	973
query47	1902	1767	1694	1694
query48	993	693	685	685
query49	625	360	374	360
query50	864	611	608	608
query51	4762	4712	4611	4611
query52	92	84	75	75
query53	450	321	328	321
query54	2656	2468	2487	2468
query55	88	77	86	77
query56	244	218	213	213
query57	1267	1346	1053	1053
query58	220	211	187	187
query59	4235	4027	3952	3952
query60	213	190	199	190
query61	102	97	98	97
query62	774	461	468	461
query63	485	337	339	337
query64	2504	1528	1510	1510
query65	3620	3563	3538	3538
query66	753	384	379	379
query67	15695	15288	15461	15288
query68	9259	642	641	641
query69	564	373	355	355
query70	1581	1403	1617	1403
query71	422	315	309	309
query72	6598	3512	3534	3512
query73	736	322	320	320
query74	6316	5907	5870	5870
query75	5372	3665	3676	3665
query76	5640	1173	1191	1173
query77	969	255	255	255
query78	12941	11602	11589	11589
query79	9542	633	632	632
query80	1496	395	395	395
query81	482	250	237	237
query82	1652	99	97	97
query83	167	130	133	130
query84	249	73	69	69
query85	877	318	325	318
query86	321	292	290	290
query87	3237	2989	3038	2989
query88	4895	2287	2268	2268
query89	486	287	315	287
query90	1974	198	200	198
query91	173	153	137	137
query92	62	52	56	52
query93	6543	560	572	560
query94	700	200	214	200
query95	1080	1057	1040	1040
query96	633	326	325	325
query97	6529	6467	6407	6407
query98	188	187	170	170
query99	3062	885	871	871
Total cold run time: 316520 ms
Total hot run time: 203306 ms

doris-robot · 2024-06-17T07:21:55Z

ClickBench: Total hot run time: 30.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852, data reload: false
query1	0.03	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.05	0.05
query4	1.79	0.08	0.06
query5	0.53	0.53	0.52
query6	1.24	0.62	0.62
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.51	0.47	0.49
query10	0.53	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.60	0.62	0.61
query14	0.79	0.79	0.76
query15	0.77	0.77	0.76
query16	0.40	0.39	0.39
query17	0.98	0.99	1.05
query18	0.24	0.24	0.26
query19	1.91	1.87	1.86
query20	0.01	0.02	0.02
query21	15.47	0.54	0.53
query22	2.03	2.08	1.25
query23	16.85	1.03	1.07
query24	7.14	0.60	1.52
query25	0.39	0.11	0.05
query26	0.80	0.16	0.15
query27	0.04	0.04	0.03
query28	5.94	0.76	0.74
query29	12.69	2.36	2.25
query30	0.57	0.52	0.52
query31	2.80	0.39	0.38
query32	3.38	0.49	0.50
query33	3.03	3.05	3.11
query34	15.29	4.85	4.83
query35	4.89	4.83	4.85
query36	1.08	1.01	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.13	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 103.67 s
Total hot run time: 30.11 s

doris-robot · 2024-06-17T07:25:08Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 502ed4e8c1ab6b0ed12e1f4cfe82dfa81df46852 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 22.0 seconds inserted 10000000 Rows, about 454K ops/s

…pache#36321 (apache#36375)

github-actionsBot added the kind/test label Jun 17, 2024

fix out
502ed4e

xiaokang changed the title ~~[fix](inverted index)Support Chinese column name with inverted index …~~[fix](inverted index)Support Chinese column name with inverted index #36321Jun 17, 2024

xiaokang merged commit 1fb6dca into apache:branch-2.0Jun 17, 2024

qidaye deleted the pick_chinese_col_idx_2.0 branch June 18, 2024 02:35

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
[fix](inverted index)Support Chinese column name with inverted indexa…
1ac2ec2
…pache#36321 (apache#36375)

Uh oh!

Conversation

qidaye commented Jun 17, 2024• edited by xiaokang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

qidaye commented Jun 17, 2024

Uh oh!

github-actionsBot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

qidaye commented Jun 17, 2024

Uh oh!

qidaye commented Jun 17, 2024

Uh oh!

github-actionsBot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

doris-robot commented Jun 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

qidaye commented Jun 17, 2024•
edited by xiaokang
Loading