[fix](orc) remove unnecessary fields of orc_reader by suxiaogang223 · Pull Request #47506 · apache/doris

suxiaogang223 · 2025-02-05T08:07:23Z

What problem does this PR solve?

remove unnecessary fields of orc_reader:

remove _col_name_to_file_col_name_low_case by storing original field name in type_map
add comment to describe the the functionality of these mappings

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-02-05T08:07:29Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

suxiaogang223 · 2025-02-05T08:08:25Z

run buildall

doris-robot · 2025-02-05T08:46:44Z

TPC-H: Total hot run time: 32342 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 499728b8943ab9db863a12122ebfd86d631aa02d, data reload: false
------ Round 1 ----------------------------------
q1	17596	5445	5342	5342
q2	2044	302	176	176
q3	10443	1221	737	737
q4	10208	965	538	538
q5	7552	2374	2156	2156
q6	189	165	131	131
q7	894	739	626	626
q8	9235	1351	1152	1152
q9	5219	4932	4920	4920
q10	6857	2314	1877	1877
q11	473	285	256	256
q12	338	356	223	223
q13	17776	3704	3167	3167
q14	228	234	213	213
q15	518	477	461	461
q16	618	645	582	582
q17	570	844	323	323
q18	6823	6399	6414	6399
q19	1963	950	547	547
q20	321	342	194	194
q21	2881	2243	2008	2008
q22	364	339	314	314
Total cold run time: 103110 ms
Total hot run time: 32342 ms
----- Round 2, with runtime_filter_mode=off -----
q1	5579	5436	5516	5436
q2	236	331	228	228
q3	2215	2637	2320	2320
q4	1438	1818	1385	1385
q5	4292	4750	4600	4600
q6	168	163	125	125
q7	2030	1946	1838	1838
q8	2634	2801	2681	2681
q9	7275	7170	7204	7170
q10	3024	3246	2765	2765
q11	565	518	496	496
q12	668	752	578	578
q13	3557	4024	3392	3392
q14	273	293	290	290
q15	528	485	474	474
q16	648	678	650	650
q17	1223	1748	1281	1281
q18	7624	7509	7317	7317
q19	864	1201	1067	1067
q20	1976	2022	1871	1871
q21	5905	5182	4917	4917
q22	603	583	579	579
Total cold run time: 53325 ms
Total hot run time: 51460 ms

doris-robot · 2025-02-05T08:59:11Z

TPC-DS: Total hot run time: 191220 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 499728b8943ab9db863a12122ebfd86d631aa02d, data reload: false
query1	1302	965	924	924
query2	6234	2052	2061	2052
query3	10995	4616	4280	4280
query4	60762	28930	23236	23236
query5	5563	605	440	440
query6	424	207	195	195
query7	5518	506	304	304
query8	337	250	230	230
query9	8353	2672	2664	2664
query10	461	313	259	259
query11	17685	15101	15455	15101
query12	158	114	110	110
query13	1467	553	421	421
query14	10381	7613	6615	6615
query15	216	201	202	201
query16	7264	625	500	500
query17	1125	742	588	588
query18	1910	417	316	316
query19	204	200	164	164
query20	119	111	113	111
query21	217	134	113	113
query22	4578	4663	4410	4410
query23	34074	33462	33294	33294
query24	5473	2267	2359	2267
query25	462	456	379	379
query26	647	276	150	150
query27	1938	475	340	340
query28	4484	2460	2421	2421
query29	533	555	431	431
query30	212	194	159	159
query31	959	895	840	840
query32	76	62	55	55
query33	429	388	288	288
query34	752	872	513	513
query35	796	853	757	757
query36	1006	1038	960	960
query37	128	108	85	85
query38	4324	4515	4292	4292
query39	1486	1458	1492	1458
query40	204	122	103	103
query41	49	47	48	47
query42	117	99	103	99
query43	522	551	493	493
query44	1335	817	831	817
query45	184	182	170	170
query46	873	1061	647	647
query47	1944	1906	1844	1844
query48	381	404	332	332
query49	770	521	428	428
query50	669	682	395	395
query51	4288	4301	4260	4260
query52	115	102	97	97
query53	239	259	183	183
query54	504	493	432	432
query55	83	78	78	78
query56	260	262	241	241
query57	1211	1220	1194	1194
query58	248	233	252	233
query59	3125	3433	2974	2974
query60	274	269	259	259
query61	154	117	116	116
query62	748	734	638	638
query63	225	186	186	186
query64	1291	1040	680	680
query65	3243	3146	3165	3146
query66	729	402	290	290
query67	15750	15706	15721	15706
query68	4236	829	526	526
query69	499	301	253	253
query70	1195	1144	1133	1133
query71	408	282	251	251
query72	6076	3842	3818	3818
query73	703	754	350	350
query74	9971	8947	8982	8947
query75	3219	3138	2669	2669
query76	3278	1166	777	777
query77	478	373	277	277
query78	10146	9991	9281	9281
query79	3459	795	574	574
query80	1672	538	442	442
query81	561	282	238	238
query82	374	156	119	119
query83	270	168	156	156
query84	291	97	74	74
query85	777	338	297	297
query86	463	321	300	300
query87	4413	4485	4346	4346
query88	4567	2143	2136	2136
query89	406	324	291	291
query90	1551	193	187	187
query91	138	135	108	108
query92	69	66	54	54
query93	2797	851	534	534
query94	762	397	296	296
query95	331	265	253	253
query96	493	619	276	276
query97	2801	2842	2776	2776
query98	230	200	191	191
query99	1299	1344	1279	1279
Total cold run time: 310520 ms
Total hot run time: 191220 ms

doris-robot · 2025-02-05T09:04:29Z

ClickBench: Total hot run time: 30.23 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 499728b8943ab9db863a12122ebfd86d631aa02d, data reload: false
query1	0.03	0.03	0.03
query2	0.08	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.43	0.40	0.42
query6	1.15	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.51	0.49
query10	0.55	0.58	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.60
query14	2.87	2.73	2.72
query15	0.88	0.84	0.86
query16	0.39	0.39	0.38
query17	1.07	1.00	1.05
query18	0.23	0.20	0.20
query19	1.96	1.88	2.02
query20	0.01	0.01	0.01
query21	15.37	0.88	0.59
query22	0.75	0.73	0.88
query23	15.13	1.44	0.62
query24	2.95	0.37	0.91
query25	0.20	0.22	0.12
query26	0.48	0.14	0.14
query27	0.07	0.07	0.05
query28	13.19	1.03	0.44
query29	12.62	3.95	3.26
query30	0.26	0.08	0.06
query31	2.84	0.60	0.40
query32	3.23	0.56	0.47
query33	2.96	3.01	3.09
query34	16.68	5.14	4.49
query35	4.52	4.56	4.48
query36	0.66	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.47 s
Total hot run time: 30.23 s

doris-robot · 2025-02-05T09:10:48Z

TeamCity be ut coverage result:
Function Coverage: 42.09% (11004/26142)
Line Coverage: 32.35% (92859/287028)
Region Coverage: 31.51% (47620/151142)
Branch Coverage: 27.53% (24098/87534)
Coverage Report: http://coverage.selectdb-in.cc/coverage/499728b8943ab9db863a12122ebfd86d631aa02d_499728b8943ab9db863a12122ebfd86d631aa02d/report/index.html

morningman

LGTM

github-actions · 2025-02-08T04:49:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-02-08T04:49:48Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? remove unnecessary fields of orc_reader: - remove `_col_name_to_file_col_name_low_case` by storing original field name in `type_map` - add comment to describe the the functionality of these mappings

remove unnecessary fields of orc_reader: - remove `_col_name_to_file_col_name_low_case` by storing original field name in `type_map` - add comment to describe the the functionality of these mappings

…#52054) bp #47506

remove unnecessary fields of orc_reader: - remove `_col_name_to_file_col_name_low_case` by storing original field name in `type_map` - add comment to describe the the functionality of these mappings

…nected Predicate #43255 #44615 #45104 #47506 #47625 #49088 #49835 #49927 (#52192) picks: #43255 #44615 #45104 #47506 #47625 #49088 #49835 #49927 --------- Co-authored-by: Qi Chen <chenqi@selectdb.com>

remove _col_name_to_file_col_name_low_case because it is unnecessary
499728b

suxiaogang223 changed the title ~~remove unnecess~~[fix](orc) remove unnecessary fields of orc_readerFeb 5, 2025

suxiaogang223 marked this pull request as draft February 5, 2025 08:08

suxiaogang223 marked this pull request as ready for review February 6, 2025 08:47

morningman added dev/2.1.x-experimental dev/3.0.x-experimental labels Feb 8, 2025

morningman approved these changes Feb 8, 2025
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Feb 8, 2025

github-actionsBot added the reviewed label Feb 8, 2025

hubgeter approved these changes Feb 8, 2025
View reviewed changes

morningman merged commit ac83d85 into apache:masterFeb 11, 2025

morningman added the dev/2.1.x-lh label Feb 17, 2025

suxiaogang223 deleted the fix_orc_reader branch March 11, 2025 04:15

suxiaogang223 mentioned this pull request Jun 20, 2025
branch-3.1: [fix](orc) remove unnecessary fields of orc_reader (#47506) #52054
Merged

morrySnow pushed a commit that referenced this pull request Jun 20, 2025
branch-3.1: [fix](orc) remove unnecessary fields of orc_reader#47506(…
a3385f3
…#52054) bp #47506

morrySnow added the dev/3.1.0-merged label Jun 20, 2025

suxiaogang223 mentioned this pull request Jun 25, 2025
branch-3.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate (#43255 #44615 #45104 #47506 #47625 #49088 #49835 #49927) #52192
Merged

morningman removed the dev/3.0.x-experimental label Jul 4, 2025

Navigation Menu

Uh oh!

[fix](orc) remove unnecessary fields of orc_reader - #47506

[fix](orc) remove unnecessary fields of orc_reader#47506
morningman merged 1 commit into
apache:masterfrom
suxiaogang223:fix_orc_reader

suxiaogang223 commented Feb 5, 2025•
edited
Loading

Uh oh!

Thearas commented Feb 5, 2025

Uh oh!

suxiaogang223 commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

morningman left a comment

Uh oh!

github-actionsBot commented Feb 8, 2025

Uh oh!

github-actionsBot commented Feb 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

suxiaogang223 commented Feb 5, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Feb 5, 2025

Uh oh!

suxiaogang223 commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

doris-robot commented Feb 5, 2025

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Feb 8, 2025

Uh oh!

github-actionsBot commented Feb 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

suxiaogang223 commented Feb 5, 2025•
edited
Loading