Skip to content

[feature](schema change) unified schema change for parquet and orc reader (#32873) - #38408

Merged
morningman merged 3 commits into
apache:branch-2.0from
hubgeter:pick_20_support_orc_struct_add_column
Jul 30, 2024
Merged

[feature](schema change) unified schema change for parquet and orc reader (#32873)#38408
morningman merged 3 commits into
apache:branch-2.0from
hubgeter:pick_20_support_orc_struct_add_column

Conversation

@hubgeter

@hubgeterhubgeter commented Jul 26, 2024

Copy link
Copy Markdown
Contributor

Proposed changes

bp #32873

Scenario: Reading a hive table after adding fields to a struct column
Since there are still problems with reading tables in parquet and text formats on the master in this scenario, only tables in orc format are picked here and some cases are added.

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hubgeter

hubgeter commented Jul 26, 2024

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

github-actionsBot commented Jul 26, 2024

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown
TPC-H: Total hot run time: 49531 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92ee1a07a697e294eb67643f3cedf5e1efbb07ca, data reload: false
------ Round 1 ----------------------------------
q1	17774	4420	4352	4352
q2	2141	153	148	148
q3	10481	1966	1896	1896
q4	10314	1215	1299	1215
q5	8510	3854	3929	3854
q6	239	124	122	122
q7	2040	1628	1631	1628
q8	9281	2717	2695	2695
q9	10415	10208	10347	10208
q10	8654	3487	3491	3487
q11	434	250	241	241
q12	472	301	293	293
q13	18360	3959	4054	3959
q14	350	314	325	314
q15	524	452	451	451
q16	676	573	568	568
q17	1131	951	958	951
q18	7288	7009	6727	6727
q19	1795	1679	1645	1645
q20	514	290	307	290
q21	4426	4105	4046	4046
q22	533	441	447	441
Total cold run time: 116352 ms
Total hot run time: 49531 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4315	4300	4312	4300
q2	327	231	221	221
q3	4132	4120	4117	4117
q4	2760	2726	2745	2726
q5	7204	7074	7026	7026
q6	236	115	114	114
q7	3273	2842	2882	2842
q8	4363	4418	4481	4418
q9	16881	16828	16687	16687
q10	4211	4260	4274	4260
q11	766	669	695	669
q12	1038	859	839	839
q13	6897	3725	3739	3725
q14	463	426	423	423
q15	506	452	453	452
q16	750	698	686	686
q17	3821	3900	3886	3886
q18	8761	8680	8703	8680
q19	1760	1694	1662	1662
q20	2375	2086	2117	2086
q21	8508	8485	8441	8441
q22	1051	972	953	953
Total cold run time: 84398 ms
Total hot run time: 79213 ms

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 37.82% (8120/21469)
Line Coverage: 29.48% (66583/225870)
Region Coverage: 28.97% (34329/118505)
Branch Coverage: 24.84% (17635/71000)
Coverage Report: http://coverage.selectdb-in.cc/coverage/92ee1a07a697e294eb67643f3cedf5e1efbb07ca_92ee1a07a697e294eb67643f3cedf5e1efbb07ca/report/index.html

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown
TPC-DS: Total hot run time: 202552 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92ee1a07a697e294eb67643f3cedf5e1efbb07ca, data reload: false
query1	932	419	379	379
query2	6526	2883	2811	2811
query3	6920	207	201	201
query4	20780	18003	17822	17822
query5	19730	6526	6503	6503
query6	295	222	234	222
query7	4164	297	311	297
query8	437	460	391	391
query9	3165	2698	2650	2650
query10	405	309	296	296
query11	11326	10850	10689	10689
query12	123	79	78	78
query13	5602	699	697	697
query14	17569	13113	13310	13113
query15	361	256	240	240
query16	6470	294	264	264
query17	1685	1460	871	871
query18	2323	411	408	408
query19	213	148	147	147
query20	84	77	79	77
query21	196	93	99	93
query22	5297	4944	5051	4944
query23	32519	31790	31670	31670
query24	7045	6541	6508	6508
query25	559	429	422	422
query26	524	163	163	163
query27	1832	294	293	293
query28	6172	2391	2365	2365
query29	2865	2777	2685	2685
query30	244	168	167	167
query31	935	760	746	746
query32	70	65	59	59
query33	408	254	253	253
query34	840	479	486	479
query35	1134	911	928	911
query36	1283	1151	1117	1117
query37	90	59	63	59
query38	3114	2918	2907	2907
query39	1371	1311	1311	1311
query40	201	93	93	93
query41	48	46	44	44
query42	84	89	84	84
query43	789	729	695	695
query44	1118	724	715	715
query45	243	239	240	239
query46	1217	958	971	958
query47	1984	1794	1843	1794
query48	1040	688	719	688
query49	627	377	371	371
query50	861	629	619	619
query51	4729	4624	4681	4624
query52	100	87	90	87
query53	444	326	316	316
query54	2644	2446	2457	2446
query55	98	77	83	77
query56	224	224	193	193
query57	1079	1078	1182	1078
query58	225	214	195	195
query59	4087	3851	4144	3851
query60	217	200	200	200
query61	100	94	98	94
query62	813	445	457	445
query63	483	347	343	343
query64	2527	1531	1399	1399
query65	3620	3559	3567	3559
query66	745	378	387	378
query67	16589	14997	15577	14997
query68	8861	680	644	644
query69	584	349	347	347
query70	1507	1567	1369	1369
query71	410	301	313	301
query72	6491	3476	3490	3476
query73	743	329	326	326
query74	6281	5843	5887	5843
query75	5387	3742	3683	3683
query76	5519	1149	1167	1149
query77	930	252	263	252
query78	12539	11571	12995	11571
query79	10171	637	649	637
query80	890	403	410	403
query81	491	231	231	231
query82	1210	96	100	96
query83	173	140	132	132
query84	269	71	72	71
query85	875	336	339	336
query86	331	296	336	296
query87	3230	2991	3017	2991
query88	4535	2317	2325	2317
query89	445	304	298	298
query90	1947	191	209	191
query91	172	140	155	140
query92	55	51	51	51
query93	5448	556	582	556
query94	697	209	206	206
query95	1101	1067	1059	1059
query96	648	330	327	327
query97	6460	6347	6325	6325
query98	192	183	180	180
query99	2790	858	962	858
Total cold run time: 313926 ms
Total hot run time: 202552 ms

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92ee1a07a697e294eb67643f3cedf5e1efbb07ca, data reload: false
query1	0.02	0.02	0.02
query2	0.08	0.03	0.02
query3	0.25	0.06	0.06
query4	1.76	0.10	0.08
query5	0.54	0.52	0.52
query6	1.24	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.53	0.47	0.50
query10	0.54	0.54	0.53
query11	0.13	0.08	0.09
query12	0.11	0.10	0.09
query13	0.62	0.61	0.61
query14	0.78	0.77	0.78
query15	0.78	0.76	0.77
query16	0.38	0.36	0.36
query17	1.01	1.02	1.01
query18	0.21	0.26	0.25
query19	1.90	1.82	1.81
query20	0.01	0.01	0.01
query21	15.47	0.54	0.54
query22	2.26	2.33	1.57
query23	17.23	1.06	0.95
query24	5.68	1.04	1.53
query25	0.40	0.08	0.06
query26	0.62	0.16	0.14
query27	0.03	0.05	0.04
query28	6.88	0.76	0.71
query29	12.67	2.35	2.47
query30	0.62	0.53	0.53
query31	2.82	0.38	0.38
query32	3.37	0.49	0.49
query33	3.03	3.04	3.04
query34	15.26	4.79	4.81
query35	4.84	4.84	4.84
query36	1.06	1.02	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.17	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.57 s
Total hot run time: 30.83 s

@doris-robot

doris-robot commented Jul 26, 2024

Copy link
Copy Markdown

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 92ee1a07a697e294eb67643f3cedf5e1efbb07ca with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 21.5 seconds inserted 10000000 Rows, about 465K ops/s

morningman
morningman reviewed Jul 26, 2024
View reviewed changes
Comment threadbe/src/vec/data_types/data_type_struct.cpp
morningman
morningman approved these changes Jul 30, 2024
View reviewed changes

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 30, 2024
@github-actions

github-actionsBot commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actionsBot commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit ce64963 into apache:branch-2.0Jul 30, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
@hubgetercaokaihua1
[feature](schema change) unified schema change for parquet and orc re…
b787d39
…ader (apache#32873) (apache#38408)
## Proposed changes
bp apache#32873 Scenario: Reading a hive table after adding fields to a struct column
Since there are still problems with reading tables in parquet and text
formats on the master in this scenario, only tables in orc format are
picked here and some cases are added.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.kind/testreviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hubgeter@doris-robot@morningman@xiaokang