[bugfix](arrow) Fix time zone issues and accuracy issues by liugddx · Pull Request #38215 · apache/doris

liugddx · 2024-07-22T11:03:27Z

Proposed changes

Issue Number: close#38174

doris-robot · 2024-07-22T11:03:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

liugddx · 2024-07-22T11:05:00Z

run buildall

github-actions · 2024-07-22T11:09:56Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-22T11:30:52Z

TPC-H: Total hot run time: 39758 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e9ede229bede24c314c312422ecdcb57c89f8a6, data reload: false
------ Round 1 ----------------------------------
q1	17625	4290	4314	4290
q2	2012	198	189	189
q3	10462	1223	1122	1122
q4	10177	809	762	762
q5	7621	2662	2676	2662
q6	222	139	136	136
q7	973	600	610	600
q8	9217	2069	2097	2069
q9	8834	6539	6560	6539
q10	8839	3773	3790	3773
q11	461	233	241	233
q12	459	232	228	228
q13	17758	3015	3017	3015
q14	283	239	233	233
q15	519	496	486	486
q16	513	375	375	375
q17	969	659	677	659
q18	7969	7442	7355	7355
q19	8644	1300	1471	1300
q20	692	324	321	321
q21	4930	3128	3275	3128
q22	351	283	292	283
Total cold run time: 119530 ms
Total hot run time: 39758 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4457	4318	4293	4293
q2	376	270	259	259
q3	3089	2925	2913	2913
q4	1984	1754	1726	1726
q5	5589	5544	5464	5464
q6	220	134	140	134
q7	2229	1843	1827	1827
q8	3308	3453	3422	3422
q9	8786	8919	8802	8802
q10	4114	3768	3831	3768
q11	611	513	493	493
q12	811	648	640	640
q13	17098	3163	3181	3163
q14	308	313	285	285
q15	526	479	475	475
q16	494	440	476	440
q17	1849	1563	1504	1504
q18	8196	8252	8009	8009
q19	1787	1549	1586	1549
q20	2418	1864	1864	1864
q21	7260	4868	4807	4807
q22	582	518	496	496
Total cold run time: 76092 ms
Total hot run time: 56333 ms

doris-robot · 2024-07-22T11:42:03Z

TPC-DS: Total hot run time: 174221 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e9ede229bede24c314c312422ecdcb57c89f8a6, data reload: false
query1	919	372	365	365
query2	6451	1898	1821	1821
query3	6625	207	218	207
query4	27384	17546	17361	17361
query5	3704	494	505	494
query6	277	187	170	170
query7	4578	295	287	287
query8	242	203	191	191
query9	8399	2474	2450	2450
query10	450	304	281	281
query11	11887	10140	10077	10077
query12	128	90	86	86
query13	1635	378	368	368
query14	10042	7774	7551	7551
query15	220	168	162	162
query16	7738	518	484	484
query17	1139	531	518	518
query18	1946	285	277	277
query19	189	146	153	146
query20	91	79	82	79
query21	206	124	132	124
query22	4557	4162	4193	4162
query23	33998	33461	33643	33461
query24	9861	2947	2872	2872
query25	602	389	375	375
query26	710	152	159	152
query27	2246	275	279	275
query28	5731	2062	2077	2062
query29	895	645	659	645
query30	255	147	151	147
query31	1012	769	781	769
query32	98	55	57	55
query33	655	328	317	317
query34	890	491	514	491
query35	862	739	743	739
query36	1129	976	953	953
query37	152	88	85	85
query38	2954	2861	2913	2861
query39	912	862	843	843
query40	216	121	124	121
query41	47	44	44	44
query42	126	108	105	105
query43	525	467	483	467
query44	1063	717	725	717
query45	189	159	159	159
query46	1072	728	703	703
query47	1878	1757	1791	1757
query48	366	291	290	290
query49	839	398	399	398
query50	782	402	390	390
query51	6762	6751	6683	6683
query52	100	93	90	90
query53	352	295	291	291
query54	886	451	444	444
query55	75	70	74	70
query56	285	265	266	265
query57	1103	1094	1070	1070
query58	244	241	267	241
query59	3074	2552	2716	2552
query60	323	269	273	269
query61	98	93	92	92
query62	764	625	644	625
query63	316	287	285	285
query64	9145	2210	1667	1667
query65	3144	3120	3128	3120
query66	763	324	331	324
query67	15453	15050	14966	14966
query68	6180	532	530	530
query69	664	436	376	376
query70	1245	1135	1162	1135
query71	444	282	276	276
query72	6901	5840	5471	5471
query73	821	327	331	327
query74	6184	5723	5651	5651
query75	3717	2694	2648	2648
query76	3843	955	1025	955
query77	707	316	298	298
query78	9577	10427	9245	9245
query79	3367	517	512	512
query80	858	485	470	470
query81	560	219	218	218
query82	715	142	132	132
query83	201	169	167	167
query84	276	86	87	86
query85	1148	316	298	298
query86	450	318	316	316
query87	3225	3185	3134	3134
query88	4392	2467	2445	2445
query89	477	393	383	383
query90	1727	190	190	190
query91	124	97	102	97
query92	60	47	47	47
query93	4429	492	491	491
query94	1082	287	277	277
query95	411	310	326	310
query96	614	275	270	270
query97	3163	3053	3068	3053
query98	230	206	195	195
query99	1504	1250	1296	1250
Total cold run time: 281055 ms
Total hot run time: 174221 ms

doris-robot · 2024-07-22T11:47:36Z

ClickBench: Total hot run time: 30.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e9ede229bede24c314c312422ecdcb57c89f8a6, data reload: false
query1	0.05	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.49	0.48	0.49
query6	1.18	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.55	0.49	0.50
query10	0.54	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.58	0.57
query14	0.76	0.78	0.81
query15	0.86	0.82	0.81
query16	0.37	0.37	0.37
query17	1.04	1.03	1.05
query18	0.23	0.22	0.22
query19	1.81	1.74	1.84
query20	0.01	0.01	0.01
query21	15.39	0.79	0.67
query22	4.49	7.79	1.75
query23	18.28	1.40	1.26
query24	2.07	0.24	0.23
query25	0.16	0.09	0.09
query26	0.31	0.21	0.20
query27	0.46	0.23	0.23
query28	13.30	1.02	1.02
query29	12.61	3.29	3.28
query30	0.25	0.06	0.05
query31	2.91	0.39	0.39
query32	3.23	0.47	0.47
query33	2.92	2.85	2.89
query34	17.16	4.35	4.32
query35	4.39	4.40	4.50
query36	0.66	0.49	0.49
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 30.55 s

liugddx · 2024-07-23T02:38:50Z

run buildall

liugddx · 2024-07-23T02:39:47Z

liugddx · 2024-07-23T02:40:17Z

@xinyiZzz PTAL

github-actions · 2024-07-23T02:44:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-23T03:29:20Z

TPC-H: Total hot run time: 40048 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4dcc56cfe21badfeb9e3ca09cd89181ab325dd30, data reload: false
------ Round 1 ----------------------------------
q1	18427	4781	4368	4368
q2	2340	197	190	190
q3	11090	1145	1099	1099
q4	10646	833	814	814
q5	7619	2764	2707	2707
q6	226	142	140	140
q7	962	620	618	618
q8	9437	2147	2140	2140
q9	8808	6580	6600	6580
q10	8652	3768	3796	3768
q11	470	237	235	235
q12	391	229	220	220
q13	18755	2945	3017	2945
q14	262	235	235	235
q15	521	469	486	469
q16	515	394	373	373
q17	981	621	727	621
q18	8079	7537	7405	7405
q19	7598	1491	1424	1424
q20	675	321	325	321
q21	4881	3096	3261	3096
q22	340	280	283	280
Total cold run time: 121675 ms
Total hot run time: 40048 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4370	4199	4224	4199
q2	369	257	282	257
q3	3018	2746	2775	2746
q4	1899	1627	1577	1577
q5	5304	5317	5314	5314
q6	221	129	132	129
q7	2175	1778	1749	1749
q8	3200	3413	3347	3347
q9	8481	8448	8399	8399
q10	3937	3747	3775	3747
q11	606	478	489	478
q12	760	614	602	602
q13	17648	3004	2984	2984
q14	310	275	268	268
q15	519	489	484	484
q16	477	423	411	411
q17	1776	1502	1484	1484
q18	7673	7588	7411	7411
q19	1669	1601	1543	1543
q20	1999	1790	1797	1790
q21	4956	4776	4681	4681
q22	562	506	531	506
Total cold run time: 71929 ms
Total hot run time: 54106 ms

doris-robot · 2024-07-23T03:40:43Z

TPC-DS: Total hot run time: 173102 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4dcc56cfe21badfeb9e3ca09cd89181ab325dd30, data reload: false
query1	909	368	368	368
query2	6493	1902	1909	1902
query3	6666	200	213	200
query4	24808	17511	17295	17295
query5	4184	485	471	471
query6	289	155	158	155
query7	4587	282	282	282
query8	244	197	179	179
query9	8491	2383	2363	2363
query10	458	287	266	266
query11	10956	10044	10052	10044
query12	135	82	83	82
query13	1632	367	353	353
query14	10218	7680	7585	7585
query15	221	168	166	166
query16	7704	467	496	467
query17	1414	569	525	525
query18	1945	278	271	271
query19	188	144	143	143
query20	90	81	80	80
query21	206	126	122	122
query22	4274	4059	4026	4026
query23	33790	33158	33205	33158
query24	11436	2873	2863	2863
query25	628	358	356	356
query26	1619	148	149	148
query27	2938	263	263	263
query28	7381	1969	1944	1944
query29	1016	632	613	613
query30	284	173	149	149
query31	971	745	741	741
query32	95	51	52	51
query33	774	318	334	318
query34	894	479	475	475
query35	862	748	729	729
query36	1117	944	920	920
query37	143	80	78	78
query38	2857	2784	2700	2700
query39	864	812	800	800
query40	286	118	116	116
query41	49	47	45	45
query42	112	98	102	98
query43	520	465	477	465
query44	1203	713	708	708
query45	188	158	158	158
query46	1087	721	727	721
query47	1869	1773	1775	1773
query48	360	290	286	286
query49	1165	428	430	428
query50	797	389	379	379
query51	6774	6793	6663	6663
query52	99	96	90	90
query53	360	287	291	287
query54	944	452	447	447
query55	76	71	82	71
query56	331	282	291	282
query57	1156	1029	1065	1029
query58	267	243	260	243
query59	2852	2747	2748	2747
query60	327	291	301	291
query61	120	115	117	115
query62	851	627	645	627
query63	325	297	307	297
query64	10554	2248	7416	2248
query65	3228	3105	3134	3105
query66	1376	331	372	331
query67	15448	15037	15096	15037
query68	4629	517	527	517
query69	597	409	357	357
query70	1205	1147	1178	1147
query71	430	273	281	273
query72	7342	5665	5077	5077
query73	739	320	318	318
query74	6043	5677	5567	5567
query75	3600	2710	2737	2710
query76	3173	942	940	940
query77	624	317	309	309
query78	9852	9371	8966	8966
query79	7654	520	518	518
query80	1211	472	475	472
query81	581	227	222	222
query82	794	147	138	138
query83	193	163	165	163
query84	277	88	88	88
query85	1429	368	289	289
query86	477	295	305	295
query87	3243	3093	3115	3093
query88	4970	2396	2381	2381
query89	493	395	393	393
query90	1874	193	190	190
query91	130	103	104	103
query92	58	49	50	49
query93	6007	511	495	495
query94	1165	266	295	266
query95	399	319	314	314
query96	606	271	272	271
query97	3187	3034	3019	3019
query98	215	200	196	196
query99	1694	1287	1228	1228
Total cold run time: 291396 ms
Total hot run time: 173102 ms

doris-robot · 2024-07-23T03:46:15Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4dcc56cfe21badfeb9e3ca09cd89181ab325dd30, data reload: false
query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.51	0.49	0.52
query6	1.13	0.74	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.04
query9	0.55	0.51	0.48
query10	0.55	0.55	0.53
query11	0.14	0.11	0.12
query12	0.15	0.13	0.12
query13	0.60	0.59	0.60
query14	0.76	0.77	0.79
query15	0.85	0.80	0.81
query16	0.37	0.37	0.37
query17	1.05	0.98	0.97
query18	0.22	0.22	0.22
query19	1.90	1.81	1.74
query20	0.02	0.01	0.01
query21	15.42	0.78	0.66
query22	3.88	7.72	1.86
query23	18.25	1.33	1.19
query24	2.13	0.23	0.22
query25	0.15	0.09	0.09
query26	0.29	0.22	0.22
query27	0.45	0.24	0.23
query28	13.35	1.01	0.99
query29	12.63	3.32	3.31
query30	0.25	0.06	0.06
query31	2.85	0.38	0.37
query32	3.31	0.48	0.48
query33	2.89	2.93	2.96
query34	17.14	4.32	4.39
query35	4.41	4.38	4.50
query36	0.64	0.48	0.49
query37	0.18	0.14	0.15
query38	0.17	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.68 s
Total hot run time: 30.56 s

xinyiZzz

LGTM

github-actions · 2024-07-23T04:08:04Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-23T04:08:05Z

PR approved by anyone and no changes requested.

zclllyybb · 2024-07-23T05:29:06Z

is it possible to add some test?

liugddx · 2024-07-23T05:44:00Z

is it possible to add some test?

I will fix all test case error

liugddx · 2024-07-23T14:37:44Z

run buildall

github-actions · 2024-07-23T14:44:21Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-23T15:06:32Z

TPC-H: Total hot run time: 40560 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03499b884bdef539533d9c24747f4ee89e53a6e2, data reload: false
------ Round 1 ----------------------------------
q1	17793	4322	4266	4266
q2	2013	199	184	184
q3	10779	1152	1136	1136
q4	10192	793	765	765
q5	7530	2727	2704	2704
q6	222	136	139	136
q7	963	616	605	605
q8	9285	2075	2094	2075
q9	8752	6585	6564	6564
q10	8946	3835	3828	3828
q11	477	244	245	244
q12	455	230	229	229
q13	18684	3009	3017	3009
q14	286	248	234	234
q15	530	501	510	501
q16	531	437	405	405
q17	969	710	701	701
q18	9186	7874	8105	7874
q19	10218	1494	1294	1294
q20	697	331	346	331
q21	4963	3186	3218	3186
q22	339	289	290	289
Total cold run time: 123810 ms
Total hot run time: 40560 ms
----- Round 2, with runtime_filter_mode=off -----
q1	4426	4306	4391	4306
q2	380	267	262	262
q3	3203	2939	2923	2923
q4	1993	1656	1718	1656
q5	5562	5481	5603	5481
q6	224	134	134	134
q7	2264	1834	1847	1834
q8	3266	3412	3389	3389
q9	8813	8848	8735	8735
q10	3959	3810	3843	3810
q11	582	485	511	485
q12	810	620	649	620
q13	17224	3144	3192	3144
q14	305	282	276	276
q15	528	505	488	488
q16	486	436	439	436
q17	1830	1555	1507	1507
q18	8223	7794	7906	7794
q19	1738	1475	1646	1475
q20	3024	1887	1864	1864
q21	8991	4823	4718	4718
q22	791	501	504	501
Total cold run time: 78622 ms
Total hot run time: 55838 ms

doris-robot · 2024-07-23T15:17:33Z

TPC-DS: Total hot run time: 173892 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03499b884bdef539533d9c24747f4ee89e53a6e2, data reload: false
query1	910	371	374	371
query2	6363	1804	1811	1804
query3	6647	206	221	206
query4	27307	17478	17248	17248
query5	3495	489	492	489
query6	268	184	171	171
query7	4581	291	299	291
query8	250	201	187	187
query9	8459	2443	2428	2428
query10	445	282	310	282
query11	11608	10058	10178	10058
query12	119	91	85	85
query13	1640	381	377	377
query14	9411	7822	7821	7821
query15	236	168	171	168
query16	7642	488	474	474
query17	1556	571	522	522
query18	1753	277	268	268
query19	202	168	154	154
query20	90	83	79	79
query21	210	125	131	125
query22	4281	3972	3977	3972
query23	33990	33483	33573	33483
query24	7174	2938	3028	2938
query25	605	385	391	385
query26	698	154	156	154
query27	2334	277	304	277
query28	5465	2098	2107	2098
query29	902	646	692	646
query30	271	148	150	148
query31	971	744	767	744
query32	98	56	58	56
query33	570	338	333	333
query34	909	506	500	500
query35	885	768	803	768
query36	1091	989	966	966
query37	142	80	82	80
query38	2983	2829	2849	2829
query39	895	832	817	817
query40	202	120	123	120
query41	47	45	44	44
query42	127	129	103	103
query43	504	484	450	450
query44	1079	738	731	731
query45	211	161	162	161
query46	1088	750	728	728
query47	1870	1784	1746	1746
query48	361	293	297	293
query49	842	411	408	408
query50	781	388	389	388
query51	6838	6640	6796	6640
query52	104	95	91	91
query53	361	303	300	300
query54	585	450	455	450
query55	77	73	80	73
query56	285	269	268	268
query57	1109	1050	1083	1050
query58	259	250	255	250
query59	2825	2550	2526	2526
query60	298	279	281	279
query61	98	92	92	92
query62	784	639	658	639
query63	324	299	299	299
query64	9199	2230	1691	1691
query65	3207	3088	3104	3088
query66	755	326	333	326
query67	15509	14996	14911	14911
query68	4519	557	552	552
query69	568	443	363	363
query70	1191	1165	1138	1138
query71	381	288	280	280
query72	7093	5600	6158	5600
query73	752	327	329	327
query74	6070	5775	5639	5639
query75	3344	2707	2735	2707
query76	2166	834	1023	834
query77	515	306	298	298
query78	10419	8997	8919	8919
query79	2514	535	549	535
query80	2532	485	496	485
query81	606	223	224	223
query82	839	132	136	132
query83	308	169	170	169
query84	261	94	88	88
query85	1872	306	299	299
query86	469	311	313	311
query87	3280	3110	3085	3085
query88	4131	2391	2389	2389
query89	477	384	400	384
query90	1715	202	195	195
query91	131	100	103	100
query92	59	48	50	48
query93	3238	515	515	515
query94	785	280	298	280
query95	411	382	321	321
query96	615	278	269	269
query97	3190	3057	3034	3034
query98	227	203	192	192
query99	1532	1262	1234	1234
Total cold run time: 273395 ms
Total hot run time: 173892 ms

doris-robot · 2024-07-23T15:23:05Z

ClickBench: Total hot run time: 31.01 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03499b884bdef539533d9c24747f4ee89e53a6e2, data reload: false
query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.51	0.52
query6	1.13	0.72	0.74
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.57	0.49	0.50
query10	0.55	0.54	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.13
query13	0.60	0.58	0.58
query14	0.75	0.78	0.79
query15	0.85	0.80	0.80
query16	0.35	0.35	0.35
query17	0.98	0.98	0.96
query18	0.23	0.22	0.22
query19	1.80	1.69	1.65
query20	0.01	0.01	0.01
query21	15.38	0.79	0.64
query22	3.96	7.26	2.38
query23	18.29	1.41	1.30
query24	2.20	0.25	0.21
query25	0.16	0.09	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.22
query28	13.18	1.01	1.00
query29	12.59	3.30	3.29
query30	0.26	0.06	0.06
query31	2.86	0.40	0.40
query32	3.26	0.47	0.46
query33	2.90	2.90	2.90
query34	17.00	4.38	4.36
query35	4.45	4.35	4.38
query36	0.67	0.46	0.47
query37	0.20	0.17	0.15
query38	0.15	0.17	0.14
query39	0.04	0.04	0.04
query40	0.15	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.36 s
Total hot run time: 31.01 s

github-actions · 2024-10-08T02:30:13Z

PR approved by at least one committer and no changes requested.

dataroaring

LGTM

morningman

LGTM

Issue Number: closeapache#38174

pick #38215 --------- Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>

Issue Number: closeapache#38174

pick apache#38215 --------- Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>

Issue Number: closeapache#38174

[pick](branch-3.0) #38215 #43281 #43960 --------- Co-authored-by: Guangdong Liu <liugddx@gmail.com>

* [pick](branch-2.1) pick apache#38215 (apache#43386) pick apache#38215 --------- Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com> * pick apache#43929 * pick apache#43960

Issue Number: closeapache#38174

…676) Background: PR apache/doris#38215 added timezone support to datetime. Versions prior to this PR hardcoded the use of TimeStampMicroVector, whereas subsequent versions return TimeStampTZVector. Consequently, to ensure compatibility, the connector inferred the time unit based on the timestamp itself. Changes: Since datetime is inherently timezone-agnostic, PR apache/doris#65823 removed the timezone from datetime, causing it to return a native timestamp; timestamptz continues to return TimeStampTZVector. Connector Adaptation: No conversion is performed if there is no timezone; the existing default conversion logic is applied if a timezone is present. Impact: Reading datetime data may result in errors on versions prior to PR apache/doris#38215.

…366) Background: PR apache/doris#38215 added timezone support to datetime. Versions prior to this PR hardcoded the use of TimeStampMicroVector, whereas subsequent versions return TimeStampTZVector. Consequently, to ensure compatibility, the connector inferred the time unit based on the timestamp itself. Changes: Since datetime is inherently timezone-agnostic, PR apache/doris#65823 removed the timezone from datetime, causing it to return a native timestamp; timestamptz continues to return TimeStampTZVector. Connector Adaptation: No conversion is performed if there is no timezone; the existing default conversion logic is applied if a timezone is present. Impact: Reading datetime data may result in errors on versions prior to PR apache/doris#38215.

xinyiZzz previously approved these changes Jul 23, 2024
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024

github-actionsBot added the reviewed label Jul 23, 2024

xinyiZzz self-requested a review July 23, 2024 12:57

xinyiZzz reviewed Jul 23, 2024
View reviewed changes

Comment threadbe/src/vec/data_types/serde/data_type_datetimev2_serde.cpp Outdated

liugddx dismissed xinyiZzz’s stale review via 03499b8July 23, 2024 14:37

liugddx force-pushed the bugfix-111 branch from 4dcc56c to 03499b8Compare July 23, 2024 14:37

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Jul 23, 2024

dataroaring approved these changes Oct 8, 2024
View reviewed changes

morningman approved these changes Oct 9, 2024
View reviewed changes

morningman merged commit 4562b9a into apache:masterOct 9, 2024

eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Oct 10, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
a8e479c
Issue Number: closeapache#38174

cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
6bb847c
Issue Number: closeapache#38174

amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
d1a4d31
Issue Number: closeapache#38174

Gabriel39 added the dev/3.0.x label Nov 1, 2024

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 6, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
6fbef45
Issue Number: closeapache#38174

xinyiZzz mentioned this pull request Nov 6, 2024
[pick](branch-2.1) pick #38215 #43386
Merged

yiguolei pushed a commit that referenced this pull request Nov 9, 2024
[pick](branch-2.1) pick#38215(#43386)
9d7bc5b
pick #38215 --------- Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Nov 9, 2024

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 11, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
08182d3
Issue Number: closeapache#38174

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 14, 2024
[pick](branch-2.1) pickapache#38215(apache#43386)
93f0566
pick apache#38215 --------- Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 18, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
b005b21
Issue Number: closeapache#38174

xinyiZzz mentioned this pull request Nov 18, 2024
[pick](branch-3.0) #38215 #43281 #43960 #44136
Closed

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 19, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
b53d0e4
Issue Number: closeapache#38174

xinyiZzz mentioned this pull request Nov 19, 2024
[pick](branch-3.0) #38215 #43281 #43960 #44244
Merged

xinyiZzz pushed a commit to xinyiZzz/incubator-doris that referenced this pull request Nov 20, 2024
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
04516fb
Issue Number: closeapache#38174

yiguolei pushed a commit that referenced this pull request Nov 26, 2024
[pick](branch-3.0)#38215#43281#43960(#44244)
b64ae78
[pick](branch-3.0) #38215 #43281 #43960 --------- Co-authored-by: Guangdong Liu <liugddx@gmail.com>

yiguolei added dev/3.0.3-merged and removed dev/3.0.x labels Nov 26, 2024

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Apr 24, 2026
[bugfix](arrow) Fix time zone issues and accuracy issues (apache#38215)
825998c
Issue Number: closeapache#38174

This was referenced Jul 21, 2026
[Fix] Adapt Arrow timestamp conversion for DATETIME and TIMESTAMPTZ apache/doris-flink-connector#676
Merged
[Fix] Adapt Arrow timestamp conversion for DATETIME and TIMESTAMPTZ apache/doris-spark-connector#366
Merged

manesioz mentioned this pull request Aug 27, 2026
[Bug] Reading DATETIME from timezone-aware Arrow shifts the stored clock apache/doris-spark-connector#376
Closed
1 task

Uh oh!

Conversation

liugddx commented Jul 22, 2024

Proposed changes

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

liugddx commented Jul 22, 2024

Uh oh!

github-actionsBot commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

doris-robot commented Jul 22, 2024

Uh oh!

liugddx commented Jul 23, 2024

Uh oh!

liugddx commented Jul 23, 2024

Uh oh!

liugddx commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

xinyiZzz left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

zclllyybb commented Jul 23, 2024

Uh oh!

liugddx commented Jul 23, 2024

Uh oh!

Uh oh!

liugddx commented Jul 23, 2024

Uh oh!

github-actionsBot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

doris-robot commented Jul 23, 2024

Uh oh!

github-actionsBot commented Oct 8, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants