Skip to content

[fix](array index) Fix index build failure when all array column data is null - #48231

Merged
airborne12 merged 2 commits into
apache:masterfrom
airborne12:fix-array-index
Feb 24, 2025
Merged

[fix](array index) Fix index build failure when all array column data is null#48231
airborne12 merged 2 commits into
apache:masterfrom
airborne12:fix-array-index

Conversation

@airborne12

@airborne12airborne12 commented Feb 24, 2025•
edited
Loading

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #47846

Problem Summary:
Fix wrong adding array index for building index.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas

Thearas commented Feb 24, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@airborne12

airborne12 commented Feb 24, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32654 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 390113dc2fd84fcc0c0e06ab834adccc76dea6f1, data reload: false
------ Round 1 ----------------------------------
q1	17605	5695	5307	5307
q2	2045	297	166	166
q3	10419	1333	773	773
q4	10198	1094	529	529
q5	7558	2684	2608	2608
q6	215	181	133	133
q7	987	783	605	605
q8	9304	1466	1361	1361
q9	5085	4869	4819	4819
q10	6894	2344	1885	1885
q11	505	280	265	265
q12	353	390	222	222
q13	17789	3779	3080	3080
q14	236	224	211	211
q15	524	457	457	457
q16	612	604	588	588
q17	595	892	335	335
q18	6511	6338	6270	6270
q19	1199	1088	579	579
q20	323	333	189	189
q21	3041	2309	1970	1970
q22	365	342	302	302
Total cold run time: 102363 ms
Total hot run time: 32654 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5529	5495	5545	5495
q2	247	349	231	231
q3	2182	2753	2327	2327
q4	1419	1910	1357	1357
q5	4320	4157	4300	4157
q6	251	181	126	126
q7	2069	2006	1798	1798
q8	2983	3033	2864	2864
q9	7413	7238	7247	7238
q10	3160	3290	2783	2783
q11	631	513	486	486
q12	702	742	556	556
q13	3574	4096	3316	3316
q14	284	311	260	260
q15	545	466	458	458
q16	647	689	658	658
q17	1217	1812	1393	1393
q18	7590	7453	7348	7348
q19	890	925	1147	925
q20	1993	2079	1859	1859
q21	5582	5080	4849	4849
q22	609	576	553	553
Total cold run time: 53837 ms
Total hot run time: 51037 ms

@airborne12

airborne12 commented Feb 24, 2025

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 31530 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false
------ Round 1 ----------------------------------
q1	17615	5418	5110	5110
q2	2047	321	175	175
q3	10373	1288	733	733
q4	10216	1023	529	529
q5	7494	2468	2316	2316
q6	194	171	140	140
q7	908	759	592	592
q8	9278	1326	1187	1187
q9	4933	4613	4696	4613
q10	6837	2312	1892	1892
q11	487	271	265	265
q12	361	362	230	230
q13	17774	3710	3092	3092
q14	233	237	222	222
q15	503	462	459	459
q16	631	606	570	570
q17	580	876	333	333
q18	6503	6180	6123	6123
q19	1074	947	543	543
q20	304	328	187	187
q21	2800	2430	1920	1920
q22	362	328	299	299
Total cold run time: 101507 ms
Total hot run time: 31530 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5187	5126	5157	5126
q2	247	344	235	235
q3	2169	2726	2335	2335
q4	1456	1855	1384	1384
q5	4231	4149	4137	4137
q6	210	166	128	128
q7	1875	1809	1692	1692
q8	2525	2455	2485	2455
q9	6859	6847	6753	6753
q10	2910	3106	2640	2640
q11	570	503	476	476
q12	662	733	585	585
q13	3285	3703	3098	3098
q14	276	277	261	261
q15	503	446	456	446
q16	631	687	623	623
q17	1109	1582	1266	1266
q18	7073	6947	7061	6947
q19	786	847	867	847
q20	1916	1975	1807	1807
q21	5168	4857	4658	4658
q22	652	557	536	536
Total cold run time: 50300 ms
Total hot run time: 48435 ms

@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 184798 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false
query1	849	368	361	361
query2	5625	1908	1868	1868
query3	6793	227	214	214
query4	26067	23458	23477	23458
query5	4972	694	527	527
query6	295	203	188	188
query7	4600	530	306	306
query8	291	233	224	224
query9	8627	2749	2785	2749
query10	516	323	272	272
query11	15172	15389	14883	14883
query12	172	114	107	107
query13	1663	556	423	423
query14	10814	6495	6377	6377
query15	219	195	191	191
query16	7633	634	460	460
query17	1381	725	582	582
query18	1985	414	324	324
query19	215	189	177	177
query20	138	117	116	116
query21	218	124	103	103
query22	4224	4095	4283	4095
query23	33889	33068	32928	32928
query24	7200	2374	2438	2374
query25	547	473	400	400
query26	1231	278	156	156
query27	2298	494	350	350
query28	4132	2497	2475	2475
query29	701	561	433	433
query30	256	186	161	161
query31	946	852	806	806
query32	75	64	62	62
query33	559	360	304	304
query34	788	870	556	556
query35	805	806	763	763
query36	968	989	869	869
query37	120	93	81	81
query38	4165	4153	4256	4153
query39	1472	1405	1401	1401
query40	212	117	135	117
query41	62	54	50	50
query42	128	110	105	105
query43	517	542	503	503
query44	1340	831	830	830
query45	182	169	163	163
query46	880	1132	681	681
query47	1742	1767	1718	1718
query48	424	452	348	348
query49	794	537	444	444
query50	739	730	455	455
query51	4196	4207	4134	4134
query52	106	115	98	98
query53	235	267	189	189
query54	494	486	421	421
query55	91	90	93	90
query56	270	286	270	270
query57	1114	1145	1078	1078
query58	257	235	241	235
query59	2677	2688	2519	2519
query60	293	293	277	277
query61	121	120	118	118
query62	775	720	657	657
query63	248	199	191	191
query64	4234	1000	661	661
query65	3308	3179	3129	3129
query66	1117	400	318	318
query67	15631	15401	15413	15401
query68	7875	791	552	552
query69	462	303	281	281
query70	1212	1146	1114	1114
query71	419	309	295	295
query72	5683	3587	3672	3587
query73	712	766	374	374
query74	9349	9146	8738	8738
query75	3173	3161	2699	2699
query76	3094	1176	755	755
query77	541	372	280	280
query78	9868	10104	9331	9331
query79	2037	837	628	628
query80	603	528	454	454
query81	479	277	244	244
query82	443	134	102	102
query83	175	180	159	159
query84	244	92	73	73
query85	856	343	298	298
query86	338	304	264	264
query87	4526	4564	4415	4415
query88	3842	2450	2380	2380
query89	399	323	292	292
query90	1910	202	194	194
query91	131	138	110	110
query92	70	60	58	58
query93	1151	994	573	573
query94	671	425	284	284
query95	359	269	252	252
query96	534	582	317	317
query97	2760	2838	2742	2742
query98	236	203	207	203
query99	1344	1429	1273	1273
Total cold run time: 270304 ms
Total hot run time: 184798 ms

@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.7 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e95d98d6a294d7538a928e20dda39aaf57dd6281, data reload: false
query1	0.04	0.05	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.11
query5	0.42	0.42	0.40
query6	1.17	0.67	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.55
query10	0.58	0.57	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.70	2.74	2.73
query15	0.93	0.85	0.85
query16	0.39	0.38	0.37
query17	1.05	1.01	1.01
query18	0.21	0.19	0.19
query19	1.91	1.97	1.87
query20	0.01	0.01	0.02
query21	15.36	0.91	0.54
query22	0.76	1.24	0.66
query23	14.91	1.36	0.58
query24	7.91	1.35	0.75
query25	0.46	0.25	0.10
query26	0.60	0.17	0.14
query27	0.05	0.05	0.05
query28	9.21	0.85	0.42
query29	12.62	3.95	3.29
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.56	0.47
query33	3.02	3.06	3.05
query34	15.80	5.17	4.58
query35	4.56	4.57	4.58
query36	0.69	0.51	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.15	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.66 s
Total hot run time: 30.7 s

amorynan
amorynan approved these changes Feb 24, 2025
View reviewed changes

@amorynanamorynan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Feb 24, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 44.51% (11752/26406)
Line Coverage: 34.00% (98248/288944)
Region Coverage: 33.09% (50327/152094)
Branch Coverage: 28.74% (25305/88038)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/report/index.html

@airborne12airborne12 mentioned this pull request Feb 24, 2025
Merged
@doris-robot

doris-robot commented Feb 24, 2025

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 44.51% (11753/26406)
Line Coverage: 34.01% (98273/288944)
Region Coverage: 33.09% (50324/152094)
Branch Coverage: 28.74% (25303/88038)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/report/index.html
Increment Report: http://coverage.selectdb-in.cc/coverage/e95d98d6a294d7538a928e20dda39aaf57dd6281_e95d98d6a294d7538a928e20dda39aaf57dd6281/increment_report/index.html

eldenmoon
eldenmoon approved these changes Feb 24, 2025
View reviewed changes

@eldenmooneldenmoon left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 24, 2025
@github-actions

github-actionsBot commented Feb 24, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@airborne12
airborne12 merged commit 7819adf into apache:masterFeb 24, 2025
@airborne12
airborne12 deleted the fix-array-index branch February 24, 2025 12:10
github-actionsBot pushed a commit that referenced this pull request Feb 24, 2025
@airborne12Your Name
[fix](array index) Fix index build failure when all array column data…
839ee38
… is null (#48231)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #47846
Problem Summary:
Fix wrong adding array index for building index.
@github-actionsgithub-actionsBot mentioned this pull request Feb 24, 2025
Merged
dataroaring pushed a commit that referenced this pull request Feb 25, 2025
@github-actions@airborne12
branch-3.0: [fix](array index) Fix index build failure when all array…
d9926da
… column data is null #48231 (#48264)
Cherry-picked from #48231
Co-authored-by: airborne12 <jiangkai@selectdb.com>
yiguolei pushed a commit that referenced this pull request Feb 25, 2025
@airborne12
branch-2.1: [fix](array index) Correct null bitmap writing for invert…
1aa57a3
…ed index #47846 (#48214)
cherry pick from #47846#48231
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
@airborne12@seawinde
[fix](array index) Fix index build failure when all array column data…
2889072
… is null (apache#48231)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: apache#47846
Problem Summary:
Fix wrong adding array index for building index.
@yiguoleiyiguolei mentioned this pull request Mar 25, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@airborne12koarz
[fix](array index) Fix index build failure when all array column data…
dceee6a
… is null (apache#48231)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: apache#47846
Problem Summary:
Fix wrong adding array index for building index.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmooneldenmooneldenmoon approved these changes
+1 more reviewer
@amorynanamorynanamorynan approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.dev/2.1.9-mergeddev/3.0.5-mergedp0_creviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@airborne12@Thearas@doris-robot@amorynan@eldenmoon@yiguolei@dataroaring