Skip to content

branch-3.0: [fix](mc)Fixed the issue that maxcompute catalog can only read part of the timestamp data #49600 - #49705

Merged
dataroaring merged 1 commit into
branch-3.0from
auto-pick-49600-branch-3.0
Apr 22, 2025
Merged

branch-3.0: [fix](mc)Fixed the issue that maxcompute catalog can only read part of the timestamp data #49600#49705
dataroaring merged 1 commit into
branch-3.0from
auto-pick-49600-branch-3.0

Conversation

@github-actions

@github-actionsgithub-actionsBot commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

Cherry-picked from #49600

@hubgeterYour Name
[fix](mc)Fixed the issue that maxcompute catalog can only read part o…
a0d8beb
…f the timestamp data (#49600)
### What problem does this PR solve?
Related pr : #48768
Problem Summary:
When using the maxcompute storage api to read the mc timestamp column,
if the timestamp precision is required to be at the nanosecond level,
only data in the range of 1970-01-01 to 2038-01-19 can be read.
Otherwise, an error will be reported: `ErrorCode=ODPS-0422214,
ErrorMessage=ODPS-0422214: Read data failed - RuntimeTimestamp data
overflow`.
This pr changes the precision to milliseconds to solve this problem.
@hello-stephen

hello-stephen commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

hello-stephen commented Mar 31, 2025

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Mar 31, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 39846 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0d8bebc3425e2fa2fd8b05148f5d403d60678a1, data reload: false
------ Round 1 ----------------------------------
q1	17593	6688	6646	6646
q2	2072	190	168	168
q3	10582	1086	1177	1086
q4	10519	717	734	717
q5	7728	2843	2834	2834
q6	228	134	129	129
q7	972	618	604	604
q8	9353	1882	2004	1882
q9	6543	6375	6380	6375
q10	7014	2228	2278	2228
q11	459	260	253	253
q12	398	210	204	204
q13	17779	2994	2944	2944
q14	236	207	207	207
q15	513	467	461	461
q16	695	579	590	579
q17	966	597	478	478
q18	7202	6670	6673	6670
q19	1413	1027	1068	1027
q20	469	206	206	206
q21	4272	3152	3259	3152
q22	1102	1001	996	996
Total cold run time: 108108 ms
Total hot run time: 39846 ms
----- Round 2, with runtime_filter_mode=off -----
q1	6593	6519	6518	6518
q2	327	232	227	227
q3	2890	2785	2890	2785
q4	2061	1806	1768	1768
q5	5723	5737	5710	5710
q6	210	126	129	126
q7	2221	1796	1817	1796
q8	3399	3657	3519	3519
q9	8832	8860	8819	8819
q10	3556	3539	3491	3491
q11	610	491	492	491
q12	834	591	600	591
q13	8482	3206	3126	3126
q14	304	274	292	274
q15	521	465	469	465
q16	687	640	647	640
q17	1837	1630	1606	1606
q18	8188	7697	7667	7667
q19	1663	1579	1567	1567
q20	2103	1812	1875	1812
q21	5445	5456	5305	5305
q22	1114	1057	1029	1029
Total cold run time: 67600 ms
Total hot run time: 59332 ms

@doris-robot

doris-robot commented Mar 31, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 196556 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0d8bebc3425e2fa2fd8b05148f5d403d60678a1, data reload: false
query1	1334	910	914	910
query2	6225	2087	2045	2045
query3	10851	4387	4147	4147
query4	65460	28709	23149	23149
query5	4977	455	469	455
query6	406	187	180	180
query7	5615	294	304	294
query8	299	209	206	206
query9	8852	2573	2587	2573
query10	439	277	255	255
query11	17306	15114	15745	15114
query12	152	108	100	100
query13	1497	438	450	438
query14	10215	7300	7027	7027
query15	209	188	191	188
query16	7188	439	471	439
query17	1060	556	565	556
query18	1874	315	310	310
query19	212	159	154	154
query20	115	111	109	109
query21	199	101	102	101
query22	4690	4672	4364	4364
query23	33992	33984	34172	33984
query24	6158	2963	2959	2959
query25	541	446	455	446
query26	659	168	172	168
query27	1837	358	372	358
query28	4355	2458	2419	2419
query29	730	468	452	452
query30	236	168	166	166
query31	1007	821	851	821
query32	67	57	60	57
query33	400	306	301	301
query34	925	510	519	510
query35	843	742	729	729
query36	1108	965	955	955
query37	114	66	66	66
query38	4075	4049	4017	4017
query39	1506	1446	1464	1446
query40	203	101	95	95
query41	51	49	49	49
query42	112	99	101	99
query43	520	497	511	497
query44	1179	817	806	806
query45	184	170	171	170
query46	1155	766	723	723
query47	2038	1992	1971	1971
query48	479	383	387	383
query49	725	393	397	393
query50	854	423	419	419
query51	7279	7149	7252	7149
query52	99	90	88	88
query53	255	183	188	183
query54	543	443	457	443
query55	78	78	76	76
query56	238	230	237	230
query57	1238	1124	1107	1107
query58	209	207	204	204
query59	3179	3054	2898	2898
query60	276	258	253	253
query61	109	107	108	107
query62	748	660	660	660
query63	219	191	188	188
query64	1371	675	633	633
query65	3218	3162	3226	3162
query66	695	288	320	288
query67	15919	15517	15601	15517
query68	3611	591	588	588
query69	434	267	251	251
query70	1181	1075	1105	1075
query71	363	256	249	249
query72	6261	4208	3952	3952
query73	755	355	345	345
query74	9750	9041	9325	9041
query75	3349	2690	2705	2690
query76	1955	1157	1097	1097
query77	497	284	283	283
query78	10546	9739	9695	9695
query79	1130	589	607	589
query80	780	438	419	419
query81	527	237	240	237
query82	181	88	93	88
query83	170	149	147	147
query84	288	81	75	75
query85	930	294	289	289
query86	331	292	303	292
query87	4530	4303	4342	4303
query88	4048	2418	2381	2381
query89	404	290	283	283
query90	2012	200	188	188
query91	187	151	151	151
query92	60	50	47	47
query93	1506	566	560	560
query94	781	282	252	252
query95	359	265	256	256
query96	622	287	288	287
query97	3328	3181	3210	3181
query98	220	198	197	197
query99	1650	1309	1331	1309
Total cold run time: 313647 ms
Total hot run time: 196556 ms

@doris-robot

doris-robot commented Mar 31, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 32.57 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0d8bebc3425e2fa2fd8b05148f5d403d60678a1, data reload: false
query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.11	0.10
query5	0.52	0.50	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.51
query10	0.54	0.54	0.56
query11	0.14	0.10	0.11
query12	0.16	0.12	0.11
query13	0.62	0.59	0.60
query14	2.87	2.73	2.83
query15	0.90	0.83	0.81
query16	0.40	0.36	0.39
query17	1.01	1.06	1.06
query18	0.24	0.23	0.22
query19	1.86	1.76	2.00
query20	0.02	0.00	0.01
query21	15.38	0.57	0.58
query22	2.72	2.07	2.06
query23	16.84	1.18	0.86
query24	2.87	1.13	1.88
query25	0.24	0.17	0.08
query26	0.39	0.16	0.14
query27	0.06	0.04	0.04
query28	9.74	0.50	0.46
query29	12.59	3.28	3.27
query30	0.24	0.06	0.06
query31	2.88	0.39	0.40
query32	3.22	0.46	0.46
query33	2.96	3.01	3.03
query34	17.25	4.49	4.49
query35	4.57	4.53	4.51
query36	0.67	0.50	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.09 s
Total hot run time: 32.57 s

@github-actions

github-actionsBot commented Apr 22, 2025

Copy link
Copy Markdown
ContributorAuthor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 22, 2025
@github-actions

github-actionsBot commented Apr 22, 2025

Copy link
Copy Markdown
ContributorAuthor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Apr 22, 2025
View reviewed changes

@dataroaringdataroaring left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
dataroaring merged commit a8fd878 into branch-3.0Apr 22, 2025
@github-actions
github-actionsBot deleted the auto-pick-49600-branch-3.0 branch April 22, 2025 06:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningmanmorningmanmorningman approved these changes
@dataroaringdataroaringdataroaring approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen@doris-robot@morningman@dataroaring@hubgeter