[improve](array) forbid create inverted index with parser for array and add cases for write correctness check by amorynan · Pull Request #39062 · apache/doris

amorynan · 2024-08-07T14:26:41Z

Proposed changes

this pr

forbid creating array with inverted index which has parser
add debug code for array creation with inverted index and add regress test to check number of field which inverted index should create
add regress test to check result when query array_contains according profile sign: RowsInvertedIndexFiltered.
Issue Number: close #xxx

doris-robot · 2024-08-07T14:26:46Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-08-07T14:33:54Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2024-08-08T09:40:31Z

run buildall

airborne12

LGTM

github-actions · 2024-08-08T09:44:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-08-08T09:44:57Z

PR approved by anyone and no changes requested.

airborne12

Need to fix case

github-actions · 2024-08-08T09:46:48Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-08T10:19:56Z

TPC-H: Total hot run time: 39647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c9a1a11e4e1f0b47ec63b5d0c4629656913b44f, data reload: false
------ Round 1 ----------------------------------
q1	17608	4341	4295	4295
q2	2018	179	186	179
q3	10476	1160	1071	1071
q4	10139	767	699	699
q5	7492	2517	2426	2426
q6	222	134	136	134
q7	962	591	584	584
q8	9215	1962	1900	1900
q9	8621	6580	6569	6569
q10	7068	2172	2140	2140
q11	448	240	238	238
q12	454	222	219	219
q13	18823	3027	2995	2995
q14	281	245	240	240
q15	525	500	478	478
q16	511	379	386	379
q17	968	649	690	649
q18	8011	7551	7473	7473
q19	5111	1119	1047	1047
q20	673	333	343	333
q21	5251	4648	4583	4583
q22	1123	1016	1024	1016
Total cold run time: 116000 ms
Total hot run time: 39647 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4396	4241	4237	4237
q2	390	274	264	264
q3	2830	2633	2764	2633
q4	2010	1712	1733	1712
q5	5614	5570	5525	5525
q6	221	132	145	132
q7	2081	1734	1801	1734
q8	3285	3431	3413	3413
q9	8875	8769	8970	8769
q10	3497	3279	3319	3279
q11	596	489	484	484
q12	784	590	598	590
q13	15776	3176	3218	3176
q14	324	316	286	286
q15	530	488	487	487
q16	509	436	440	436
q17	1828	1531	1492	1492
q18	8268	7781	7807	7781
q19	1720	1493	1531	1493
q20	2141	1884	1904	1884
q21	5691	5438	5311	5311
q22	1135	1013	1011	1011
Total cold run time: 72501 ms
Total hot run time: 56129 ms

amorynan · 2024-08-08T10:24:47Z

run buildall

github-actions · 2024-08-08T10:28:49Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-08-08T10:30:15Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-08T11:00:45Z

TPC-H: Total hot run time: 39132 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 035fe8f1f12f7d574f4e06b0bc4f120181a75b52, data reload: false
------ Round 1 ----------------------------------
q1	17607	4336	4298	4298
q2	2023	179	175	175
q3	10489	1179	1111	1111
q4	10143	810	711	711
q5	7503	2509	2462	2462
q6	226	139	139	139
q7	966	602	595	595
q8	9221	1908	1877	1877
q9	8871	6526	6495	6495
q10	7087	2205	2163	2163
q11	457	232	240	232
q12	461	217	218	217
q13	17768	3029	2987	2987
q14	279	243	241	241
q15	525	476	495	476
q16	504	387	381	381
q17	949	621	747	621
q18	8074	7428	7482	7428
q19	6229	1073	1054	1054
q20	684	327	339	327
q21	5287	4134	4117	4117
q22	1136	1025	1026	1025
Total cold run time: 116489 ms
Total hot run time: 39132 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4533	4228	4257	4228
q2	364	276	273	273
q3	2862	2602	2770	2602
q4	1982	1697	1729	1697
q5	5533	5502	5501	5501
q6	228	134	132	132
q7	2081	1740	1777	1740
q8	3261	3449	3409	3409
q9	8731	8661	8847	8661
q10	3513	3280	3331	3280
q11	585	506	490	490
q12	787	611	601	601
q13	16930	3157	3236	3157
q14	302	307	286	286
q15	537	486	487	486
q16	519	459	458	458
q17	1793	1538	1486	1486
q18	8062	7975	7818	7818
q19	3188	1724	1518	1518
q20	2061	1884	1903	1884
q21	5458	5364	5288	5288
q22	1115	1015	1036	1015
Total cold run time: 74425 ms
Total hot run time: 56010 ms

doris-robot · 2024-08-08T11:13:18Z

TPC-DS: Total hot run time: 202770 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 035fe8f1f12f7d574f4e06b0bc4f120181a75b52, data reload: false
query1	959	404	392	392
query2	6427	1948	1806	1806
query3	6630	210	219	210
query4	34150	23197	23050	23050
query5	3604	496	485	485
query6	297	163	169	163
query7	4570	289	287	287
query8	240	198	191	191
query9	8714	2377	2347	2347
query10	921	882	854	854
query11	16962	14960	15000	14960
query12	137	95	100	95
query13	1623	370	358	358
query14	10265	8020	7913	7913
query15	410	339	322	322
query16	7663	473	452	452
query17	1702	583	581	581
query18	1965	386	397	386
query19	271	221	231	221
query20	127	112	107	107
query21	206	109	111	109
query22	4562	4292	4300	4292
query23	34656	33633	33771	33633
query24	11105	2887	2920	2887
query25	558	365	358	358
query26	696	150	144	144
query27	2140	279	284	279
query28	5996	2011	1989	1989
query29	771	400	406	400
query30	255	149	145	145
query31	969	761	721	721
query32	97	51	52	51
query33	675	274	279	274
query34	862	467	467	467
query35	940	864	842	842
query36	1081	923	900	900
query37	132	82	79	79
query38	4272	4159	4082	4082
query39	1455	1366	1385	1366
query40	193	113	112	112
query41	44	44	42	42
query42	113	93	95	93
query43	495	460	447	447
query44	1097	733	743	733
query45	419	384	378	378
query46	1119	779	765	765
query47	1859	1767	1771	1767
query48	358	302	305	302
query49	817	424	420	420
query50	814	397	400	397
query51	6715	6668	6625	6625
query52	106	87	89	87
query53	248	183	176	176
query54	884	445	438	438
query55	75	74	72	72
query56	277	244	239	239
query57	1138	1076	1064	1064
query58	226	226	255	226
query59	2988	2754	2780	2754
query60	288	260	266	260
query61	100	96	90	90
query62	789	635	658	635
query63	212	181	172	172
query64	9246	2405	1979	1979
query65	3225	3139	3105	3105
query66	745	322	324	322
query67	15357	14774	14663	14663
query68	4491	543	543	543
query69	402	388	384	384
query70	1141	1108	1117	1108
query71	432	278	267	267
query72	17918	16382	16346	16346
query73	761	323	320	320
query74	9307	8757	8771	8757
query75	3482	2654	2688	2654
query76	2678	964	944	944
query77	455	302	308	302
query78	9493	8965	9680	8965
query79	2108	525	523	523
query80	1060	493	502	493
query81	605	232	224	224
query82	815	136	132	132
query83	233	149	149	149
query84	238	77	138	77
query85	1708	333	302	302
query86	343	279	307	279
query87	4801	4470	4614	4470
query88	4509	2488	2503	2488
query89	389	285	287	285
query90	1731	197	184	184
query91	139	116	114	114
query92	57	51	49	49
query93	2370	527	521	521
query94	672	291	296	291
query95	350	271	258	258
query96	605	274	272	272
query97	3235	3108	3060	3060
query98	223	203	194	194
query99	1509	1267	1254	1254
Total cold run time: 307272 ms
Total hot run time: 202770 ms

doris-robot · 2024-08-08T11:18:50Z

ClickBench: Total hot run time: 30.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 035fe8f1f12f7d574f4e06b0bc4f120181a75b52, data reload: false
query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.49	0.48
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.54	0.52
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.60	0.59
query14	0.77	0.77	0.79
query15	0.85	0.82	0.81
query16	0.36	0.35	0.36
query17	0.98	0.96	1.04
query18	0.23	0.21	0.22
query19	1.86	1.68	1.71
query20	0.01	0.00	0.00
query21	15.40	0.73	0.65
query22	4.74	6.90	1.81
query23	18.30	1.35	1.20
query24	2.06	0.24	0.21
query25	0.15	0.08	0.08
query26	0.29	0.22	0.21
query27	0.46	0.23	0.22
query28	13.32	1.01	1.02
query29	12.66	3.34	3.33
query30	0.24	0.05	0.05
query31	2.89	0.38	0.38
query32	3.29	0.48	0.46
query33	2.90	2.90	2.90
query34	17.20	4.41	4.39
query35	4.49	4.40	4.40
query36	0.66	0.48	0.47
query37	0.20	0.16	0.15
query38	0.16	0.15	0.14
query39	0.05	0.04	0.04
query40	0.15	0.12	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.59 s
Total hot run time: 30.44 s

…an/doris into improve-array-invertedIdx

amorynan · 2024-08-09T10:32:33Z

run buildall

github-actions · 2024-08-09T10:36:43Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2024-08-10T05:39:41Z

run buildall

github-actions · 2024-08-10T05:46:22Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2024-08-13T03:10:15Z

run buildall

github-actions · 2024-08-13T03:17:21Z

clang-tidy review says "All clean, LGTM! 👍"

amorynan · 2024-08-14T07:41:46Z

run buildall

amorynan · 2024-08-14T09:26:47Z

run buildall

doris-robot · 2024-08-14T10:35:04Z

TPC-H: Total hot run time: 37697 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1313355a827ad034728f10c685b1fb6a66e4f87, data reload: false
------ Round 1 ----------------------------------
q1	17716	4659	4340	4340
q2	2028	186	188	186
q3	11654	1035	1123	1035
q4	10497	764	814	764
q5	7772	2811	2803	2803
q6	223	138	136	136
q7	965	620	599	599
q8	9539	2019	2057	2019
q9	8679	6507	6494	6494
q10	7008	2206	2192	2192
q11	447	244	249	244
q12	395	228	222	222
q13	18938	2997	3030	2997
q14	284	240	232	232
q15	528	485	487	485
q16	501	416	391	391
q17	968	720	679	679
q18	7377	6823	6771	6771
q19	6669	959	1040	959
q20	656	342	353	342
q21	3927	2906	2819	2819
q22	1091	1004	988	988
Total cold run time: 117862 ms
Total hot run time: 37697 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4528	4280	4298	4280
q2	374	264	268	264
q3	2871	2619	2653	2619
q4	2073	1735	1745	1735
q5	5645	5707	5627	5627
q6	239	144	140	140
q7	2157	1709	1867	1709
q8	3297	3518	3738	3518
q9	8851	8772	8910	8772
q10	3521	3323	3318	3318
q11	627	512	503	503
q12	827	622	641	622
q13	17245	3175	3199	3175
q14	320	288	282	282
q15	541	491	483	483
q16	523	475	459	459
q17	1808	1581	1577	1577
q18	8276	8095	7849	7849
q19	1824	1635	1581	1581
q20	2427	1926	1948	1926
q21	11365	5557	5216	5216
q22	1104	1072	1038	1038
Total cold run time: 80443 ms
Total hot run time: 56693 ms

doris-robot · 2024-08-14T10:47:35Z

TPC-DS: Total hot run time: 189618 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1313355a827ad034728f10c685b1fb6a66e4f87, data reload: false
query1	1255	894	871	871
query2	6459	1982	1948	1948
query3	10596	3912	3839	3839
query4	57949	26230	23046	23046
query5	5711	504	488	488
query6	494	170	157	157
query7	6492	295	300	295
query8	307	211	218	211
query9	8957	2479	2451	2451
query10	512	283	285	283
query11	17797	15020	15195	15020
query12	160	102	100	100
query13	1596	400	392	392
query14	11722	7183	7277	7183
query15	278	177	185	177
query16	7625	533	519	519
query17	1125	599	604	599
query18	2100	317	311	311
query19	295	162	153	153
query20	122	113	119	113
query21	219	105	102	102
query22	4481	4249	4377	4249
query23	34192	33018	33355	33018
query24	5572	2887	2900	2887
query25	537	397	416	397
query26	699	162	161	161
query27	1776	282	282	282
query28	3664	2068	2059	2059
query29	717	423	422	422
query30	242	157	148	148
query31	920	741	741	741
query32	81	56	56	56
query33	479	295	286	286
query34	837	466	473	466
query35	823	737	737	737
query36	1071	910	907	907
query37	136	81	79	79
query38	3940	3817	3903	3817
query39	1454	1415	1395	1395
query40	197	120	118	118
query41	51	46	45	45
query42	116	98	98	98
query43	514	492	482	482
query44	1060	740	732	732
query45	200	164	165	164
query46	1096	703	754	703
query47	1835	1744	1773	1744
query48	360	299	296	296
query49	766	426	438	426
query50	804	404	402	402
query51	6718	6625	6732	6625
query52	104	90	92	90
query53	252	188	187	187
query54	587	557	448	448
query55	75	75	74	74
query56	268	241	240	240
query57	1129	1029	1031	1029
query58	238	221	235	221
query59	2950	2946	2731	2731
query60	286	264	266	264
query61	100	99	97	97
query62	755	636	645	636
query63	214	178	178	178
query64	3186	1722	1700	1700
query65	3223	3171	3156	3156
query66	689	335	332	332
query67	15362	14759	14629	14629
query68	5882	588	551	551
query69	461	280	278	278
query70	1220	1133	1149	1133
query71	503	276	268	268
query72	6586	2230	2060	2060
query73	788	320	320	320
query74	9147	8807	8804	8804
query75	3643	2673	2705	2673
query76	3187	1066	996	996
query77	551	309	324	309
query78	10549	9662	8874	8874
query79	8125	535	531	531
query80	2637	505	490	490
query81	570	220	235	220
query82	1029	138	131	131
query83	312	146	145	145
query84	275	79	74	74
query85	1687	289	274	274
query86	364	288	294	288
query87	4434	4264	4281	4264
query88	4890	2283	2313	2283
query89	494	291	273	273
query90	1564	198	186	186
query91	126	96	95	95
query92	59	49	49	49
query93	5651	534	531	531
query94	673	298	294	294
query95	345	259	260	259
query96	630	271	270	270
query97	3218	3086	3033	3033
query98	217	252	198	198
query99	1580	1252	1253	1252
Total cold run time: 327924 ms
Total hot run time: 189618 ms

doris-robot · 2024-08-14T10:53:06Z

ClickBench: Total hot run time: 30.62 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1313355a827ad034728f10c685b1fb6a66e4f87, data reload: false
query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.69	0.07	0.08
query5	0.50	0.47	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.05
query9	0.55	0.49	0.49
query10	0.53	0.53	0.55
query11	0.16	0.11	0.11
query12	0.14	0.13	0.12
query13	0.59	0.60	0.60
query14	0.76	0.79	0.77
query15	0.86	0.81	0.81
query16	0.37	0.37	0.39
query17	0.95	0.95	0.95
query18	0.22	0.22	0.22
query19	1.79	1.79	1.70
query20	0.01	0.00	0.01
query21	15.41	0.75	0.67
query22	3.87	6.96	1.94
query23	18.34	1.33	1.28
query24	2.09	0.23	0.21
query25	0.14	0.08	0.08
query26	0.31	0.21	0.22
query27	0.46	0.24	0.22
query28	13.35	1.02	1.00
query29	12.55	3.30	3.30
query30	0.23	0.06	0.04
query31	2.89	0.40	0.39
query32	3.25	0.48	0.48
query33	2.98	2.99	2.92
query34	17.00	4.35	4.35
query35	4.43	4.40	4.37
query36	0.66	0.49	0.47
query37	0.18	0.16	0.15
query38	0.15	0.15	0.16
query39	0.05	0.04	0.03
query40	0.14	0.12	0.12
query41	0.09	0.05	0.04
query42	0.05	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.34 s
Total hot run time: 30.62 s

amorynan · 2024-08-15T02:42:03Z

run cloud_p1

airborne12

LGTM

github-actions · 2024-08-15T06:21:19Z

PR approved by at least one committer and no changes requested.

airborne12

Fill your Proposed changes, DO NOT leave it empty

airborne12

LGTM

github-actions · 2024-08-15T06:28:32Z

PR approved by at least one committer and no changes requested.

xiaokang

LGTM

…nd add cases for write correctness check (#39062) 1. forbid creating array with inverted index which has parser 2. add debug code for array creation with inverted index and add regress test to check number of field which inverted index should create 3. add regress test to check result when query array_contains according profile sign: RowsInvertedIndexFiltered.

…nd add cases for write correctness check (apache#39062) 1. forbid creating array with inverted index which has parser 2. add debug code for array creation with inverted index and add regress test to check number of field which inverted index should create 3. add regress test to check result when query array_contains according profile sign: RowsInvertedIndexFiltered.

add cases and fix fe code
1ba4788

github-actionsBot added the doing label Aug 7, 2024

update
3c9a1a1

airborne12 approved these changes Aug 8, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 8, 2024

github-actionsBot added the reviewed label Aug 8, 2024

airborne12 reviewed Aug 8, 2024
View reviewed changes

Comment thread...ession-test/suites/inverted_index_p0/test_array_inverted_index/test_array_index_write.groovy Outdated

airborne12 requested changes Aug 8, 2024
View reviewed changes

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 8, 2024

amorynan added 2 commits August 8, 2024 18:22

fix cases
dd49cba

Merge branch 'master' into improve-array-invertedIdx
035fe8f

amorynan requested a review from airborne12 August 8, 2024 10:23

amorynan added 2 commits August 9, 2024 18:29

fix case and fe code
b42a060

Merge branch 'improve-array-invertedIdx' ofhttps://github.com/amoryn…
214d536
…an/doris into improve-array-invertedIdx

add query instrumentation and regress case with profile
2374ae3

xiaokang changed the title ~~[improve](array) ban create inverted index with parser for array and add cases for write correctness check~~ [improve](array) forbid create inverted index with parser for array and add cases for write correctness check Aug 11, 2024

Merge branch 'master' into improve-array-invertedIdx
0651296

fix cases
f44c8f6

fix unsigned to int
d131335

airborne12 approved these changes Aug 15, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2024

airborne12 requested changes Aug 15, 2024
View reviewed changes

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Aug 15, 2024

amorynan requested a review from airborne12 August 15, 2024 06:24

airborne12 approved these changes Aug 15, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Aug 15, 2024

xiaokang approved these changes Aug 15, 2024
View reviewed changes

xiaokang merged commit 237fca2 into apache:masterAug 15, 2024

xiaokang removed the doing label Aug 15, 2024

dataroaring added the dev/3.0.2-merged label Aug 17, 2024

gavinchou mentioned this pull request Oct 13, 2024
Release Note 3.0.2 #41558
Closed

Uh oh!

Conversation

amorynan commented Aug 7, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Aug 7, 2024

Uh oh!

github-actionsBot commented Aug 7, 2024

Uh oh!

amorynan commented Aug 8, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

amorynan commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

github-actionsBot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

doris-robot commented Aug 8, 2024

Uh oh!

amorynan commented Aug 9, 2024

Uh oh!

github-actionsBot commented Aug 9, 2024

Uh oh!

amorynan commented Aug 10, 2024

Uh oh!

github-actionsBot commented Aug 10, 2024

Uh oh!

amorynan commented Aug 13, 2024

Uh oh!

github-actionsBot commented Aug 13, 2024

Uh oh!

amorynan commented Aug 14, 2024

Uh oh!

amorynan commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

doris-robot commented Aug 14, 2024

Uh oh!

amorynan commented Aug 15, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 15, 2024

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Aug 15, 2024

Uh oh!

xiaokang left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

amorynan commented Aug 7, 2024•
edited
Loading