Skip to content

branch-3.0:[echcement](hive)support read hive table that change timestamp column to bigint. (#52954) - #53307

Merged
morningman merged 2 commits into
apache:branch-3.0from
hubgeter:pick_sc_ts_bigint
Jul 20, 2025
Merged

branch-3.0:[echcement](hive)support read hive table that change timestamp column to bigint. (#52954)#53307
morningman merged 2 commits into
apache:branch-3.0from
hubgeter:pick_sc_ts_bigint

Conversation

@hubgeter

@hubgeterhubgeter commented Jul 15, 2025

Copy link
Copy Markdown
Contributor

bp #52954

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hubgeter
[echcement](hive)support read hive table that change timestamp column…
52ef8db
… to bigint. (apache#52954)
Related PR: apache#47471
Problem Summary:
This pr is a supplement to apache#47471.
This pr is used to support reading hive tables that convert timestamp
columns to bigint columns and display them in `ms` precision.
(parquet/orc hive table.)
@hello-stephen

hello-stephen commented Jul 15, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hubgeter

hubgeter commented Jul 15, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hubgeterhubgeter changed the title [echcement](hive)support read hive table that change timestamp column to bigint. (#52954)branch-3.0:[echcement](hive)support read hive table that change timestamp column to bigint. (#52954)Jul 15, 2025
@hubgeter

hubgeter commented Jul 16, 2025

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

doris-robot commented Jul 16, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 39713 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6140dc83a990c2f8232cd4372980b008c634c14b, data reload: false
------ Round 1 ----------------------------------
q1	17721	6958	6576	6576
q2	2083	185	160	160
q3	10670	1130	1179	1130
q4	10546	704	726	704
q5	7742	3160	2796	2796
q6	215	133	130	130
q7	984	628	604	604
q8	9577	1963	2033	1963
q9	6689	6384	6315	6315
q10	7003	2241	2270	2241
q11	456	278	269	269
q12	390	209	226	209
q13	17789	2928	2955	2928
q14	247	201	217	201
q15	511	461	459	459
q16	450	375	371	371
q17	984	575	546	546
q18	7524	6747	6764	6747
q19	1401	1123	1001	1001
q20	496	210	213	210
q21	4044	3208	3188	3188
q22	1096	965	1000	965
Total cold run time: 108618 ms
Total hot run time: 39713 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6621	6612	6612	6612
q2	332	224	226	224
q3	2950	2979	2918	2918
q4	2069	1770	1777	1770
q5	5710	5703	5681	5681
q6	218	130	131	130
q7	2228	1790	1848	1790
q8	3339	3594	3502	3502
q9	8821	8906	8769	8769
q10	3574	3478	3528	3478
q11	596	500	486	486
q12	798	574	589	574
q13	6807	3213	3130	3130
q14	295	277	274	274
q15	506	474	455	455
q16	506	441	437	437
q17	1857	1624	1609	1609
q18	8226	7897	7610	7610
q19	1661	1520	1520	1520
q20	2113	1878	1889	1878
q21	5063	4995	5035	4995
q22	1165	1041	1030	1030
Total cold run time: 65455 ms
Total hot run time: 58872 ms

@doris-robot

doris-robot commented Jul 16, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196266 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6140dc83a990c2f8232cd4372980b008c634c14b, data reload: false
query1	1278	900	902	900
query2	6241	1943	1914	1914
query3	10823	4402	4427	4402
query4	32887	23361	23306	23306
query5	3388	443	429	429
query6	264	176	186	176
query7	4002	313	310	310
query8	293	225	223	223
query9	9511	2594	2548	2548
query10	460	261	245	245
query11	17732	15197	15217	15197
query12	163	97	98	97
query13	1565	428	411	411
query14	8600	6469	7255	6469
query15	253	191	201	191
query16	8057	481	497	481
query17	1647	596	562	562
query18	2215	329	305	305
query19	199	172	163	163
query20	119	110	107	107
query21	204	107	103	103
query22	4926	4779	4776	4776
query23	35201	34161	33838	33838
query24	11644	2898	2915	2898
query25	704	436	426	426
query26	1766	171	167	167
query27	2874	346	349	346
query28	7992	2140	2132	2132
query29	1049	470	453	453
query30	267	157	167	157
query31	1069	820	814	814
query32	99	58	62	58
query33	801	317	308	308
query34	1048	516	525	516
query35	886	718	738	718
query36	1111	932	931	931
query37	200	69	70	69
query38	4075	3974	3931	3931
query39	1528	1442	1486	1442
query40	263	112	105	105
query41	53	50	48	48
query42	116	108	100	100
query43	513	484	482	482
query44	1323	821	826	821
query45	191	170	171	170
query46	1172	746	724	724
query47	2046	1925	1960	1925
query48	497	435	423	423
query49	1074	390	391	390
query50	838	409	428	409
query51	7471	7379	7339	7339
query52	106	89	86	86
query53	260	183	184	183
query54	1286	481	478	478
query55	80	75	81	75
query56	294	253	233	233
query57	1333	1192	1236	1192
query58	246	222	232	222
query59	3242	3128	3032	3032
query60	288	258	287	258
query61	156	134	114	114
query62	891	677	672	672
query63	214	186	182	182
query64	5021	683	622	622
query65	3265	3188	3172	3172
query66	1383	289	287	287
query67	15853	15752	15544	15544
query68	4753	592	571	571
query69	441	266	270	266
query70	1128	1116	1109	1109
query71	338	257	250	250
query72	6200	4079	4051	4051
query73	746	346	350	346
query74	10546	9190	8967	8967
query75	3370	2631	2626	2626
query76	2688	1000	1119	1000
query77	404	273	271	271
query78	10523	9755	9675	9675
query79	2336	600	615	600
query80	1123	426	432	426
query81	534	221	223	221
query82	656	91	87	87
query83	236	144	167	144
query84	239	76	79	76
query85	1635	341	312	312
query86	473	285	301	285
query87	4380	4251	4203	4203
query88	4081	2365	2359	2359
query89	423	290	292	290
query90	2056	185	187	185
query91	182	151	146	146
query92	59	52	51	51
query93	2262	565	556	556
query94	946	297	289	289
query95	374	251	257	251
query96	615	280	275	275
query97	3278	3102	3131	3102
query98	227	198	194	194
query99	1477	1301	1275	1275
Total cold run time: 305345 ms
Total hot run time: 196266 ms

@doris-robot

doris-robot commented Jul 16, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6140dc83a990c2f8232cd4372980b008c634c14b, data reload: false
query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.65	0.11	0.10
query5	0.54	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.03
query8	0.04	0.03	0.04
query9	0.56	0.50	0.49
query10	0.56	0.58	0.56
query11	0.14	0.09	0.10
query12	0.14	0.11	0.10
query13	0.62	0.59	0.59
query14	0.78	0.78	0.82
query15	0.84	0.82	0.81
query16	0.38	0.38	0.38
query17	1.05	1.00	1.08
query18	0.23	0.21	0.21
query19	1.89	1.89	1.82
query20	0.01	0.02	0.01
query21	15.38	0.57	0.56
query22	2.38	1.68	1.47
query23	16.89	1.13	0.85
query24	3.09	1.94	1.48
query25	0.20	0.18	0.04
query26	0.55	0.14	0.13
query27	0.04	0.04	0.04
query28	9.23	0.50	0.50
query29	12.59	3.23	3.22
query30	0.25	0.06	0.07
query31	2.87	0.39	0.39
query32	3.23	0.46	0.45
query33	3.01	3.00	3.02
query34	17.06	4.48	4.50
query35	4.52	4.49	4.51
query36	0.68	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.38 s
Total hot run time: 30.27 s

@doris-robot

doris-robot commented Jul 16, 2025

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 78.87% (56/71) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage41.22% (10954/26575)
Line Coverage31.96% (93898/293768)
Region Coverage31.12% (48443/155662)
Branch Coverage27.57% (24824/90044)

@hello-stephen

hello-stephen commented Jul 16, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 78.87% (56/71) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage74.94% (19656/26230)
Line Coverage68.25% (200222/293369)
Region Coverage66.35% (119643/180321)
Branch Coverage59.72% (60790/101798)

dataroaring
dataroaring approved these changes Jul 18, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

github-actionsBot commented Jul 18, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 18, 2025
@github-actions

github-actionsBot commented Jul 18, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hello-stephen

hello-stephen commented Jul 19, 2025

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 78.87% (56/71) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage74.94% (19656/26230)
Line Coverage68.25% (200222/293369)
Region Coverage66.35% (119643/180321)
Branch Coverage59.72% (60790/101798)

@morningman
morningman merged commit 93beba4 into apache:branch-3.0Jul 20, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hubgeter@hello-stephen@doris-robot@dataroaring@morningman