[fix](exec) Return Thrift DATETIME as naive Arrow timestamp by Mryange · Pull Request #67232 · apache/doris

Mryange · 2026-08-27T13:56:37Z

Thrift external scanner responses encoded Doris DATETIME values with the BE session timezone in the Arrow schema. This made clients interpret wall-clock values as instants and could introduce an eight-hour offset. The MemoryScratch sink now uses the existing naive-timestamp conversion for DATETIMEV2, while preserving timezone-aware TIMESTAMPTZ behavior.

Release note

DATETIME values returned through the Thrift external scanner are represented as timezone-naive Arrow timestamps.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Mryange · 2026-08-27T13:56:44Z

run buildall

hello-stephen · 2026-08-27T13:56:48Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hello-stephen · 2026-08-27T15:08:37Z

TPC-H: Total hot run time: 17052 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false
------ Round 1 ----------------------------------
============================================
q1	17612	3038	3002	3002
q2	2080	254	229	229
q3	10250	890	525	525
q4	4665	246	207	207
q5	7672	552	388	388
q6	134	111	92	92
q7	526	503	380	380
q8	9243	974	941	941
q9	3479	2428	2398	2398
q10	6516	841	713	713
q11	399	196	177	177
q12	607	254	197	197
q13	18134	1514	1141	1141
q14	155	149	141	141
q15	q16	432	393	369	369
q17	1348	913	843	843
q18	2994	2236	2246	2236
q19	1103	874	793	793
q20	380	281	198	198
q21	4826	1848	1880	1848
q22	326	267	234	234
Total cold run time: 92881 ms
Total hot run time: 17052 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3368	3348	3331	3331
q2	502	383	372	372
q3	2182	2293	2214	2214
q4	1185	1158	880	880
q5	2159	2129	2112	2112
q6	175	116	89	89
q7	1018	941	850	850
q8	1581	1391	1407	1391
q9	3140	3084	3060	3060
q10	1844	1792	1598	1598
q11	352	266	248	248
q12	448	427	343	343
q13	1506	1540	1164	1164
q14	184	177	152	152
q15	q16	386	401	349	349
q17	3589	3254	3244	3244
q18	4824	4424	4787	4424
q19	999	805	903	805
q20	1001	1021	818	818
q21	3837	3106	3274	3106
q22	390	340	320	320
Total cold run time: 34670 ms
Total hot run time: 30870 ms

hello-stephen · 2026-08-27T15:15:07Z

TPC-DS: Total hot run time: 81695 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false
query5	4285	420	343	343
query6	395	134	119	119
query7	4936	433	236	236
query8	292	121	116	116
query9	8670	2883	2875	2875
query10	386	217	184	184
query11	5384	1033	910	910
query12	124	72	71	71
query13	1195	447	329	329
query14	6145	2188	2059	2059
query14_1	1954	1938	1943	1938
query15	175	118	109	109
query16	921	369	353	353
query17	796	472	390	390
query18	2326	323	253	253
query19	170	142	111	111
query20	69	66	69	66
query21	198	100	90	90
query22	5302	5291	5263	5263
query23	6530	6141	5990	5990
query23_1	6210	6126	6128	6126
query24	7548	1142	810	810
query24_1	812	807	798	798
query25	456	326	273	273
query26	1269	237	132	132
query27	2935	398	292	292
query28	4894	1529	1531	1529
query29	981	450	351	351
query30	245	148	125	125
query31	1086	412	323	323
query32	121	70	69	69
query33	450	212	191	191
query34	1224	839	473	473
query35	390	410	355	355
query36	567	547	526	526
query37	122	80	68	68
query38	999	846	821	821
query39	510	517	494	494
query39_1	472	483	485	483
query40	218	88	73	73
query41	50	49	52	49
query42	72	69	69	69
query43	238	243	206	206
query44	1021	560	559	559
query45	113	105	98	98
query46	804	820	538	538
query47	764	761	698	698
query48	315	307	233	233
query49	541	223	176	176
query50	726	259	196	196
query51	7952	7954	7972	7954
query52	64	64	60	60
query53	186	195	140	140
query54	209	166	166	166
query55	75	58	52	52
query56	173	153	161	153
query57	688	623	601	601
query58	195	154	164	154
query59	1134	1125	1028	1028
query60	238	182	169	169
query61	111	112	106	106
query62	339	201	175	175
query63	171	135	137	135
query64	2591	673	530	530
query65	1569	1565	1532	1532
query66	1762	252	193	193
query67	10101	9813	9607	9607
query68	3031	1166	737	737
query69	357	225	185	185
query70	655	599	610	599
query71	240	175	171	171
query72	2329	1690	1597	1597
query73	643	565	333	333
query74	1979	1224	1119	1119
query75	1199	1114	951	951
query76	2367	745	542	542
query77	250	261	206	206
query78	3855	3630	3230	3230
query79	2785	824	589	589
query80	1620	349	285	285
query81	512	153	137	137
query82	646	121	99	99
query83	276	211	190	190
query84	294	109	87	87
query85	856	354	296	296
query86	478	169	171	169
query87	1004	973	889	889
query88	2886	2093	2085	2085
query89	283	193	171	171
query90	1967	128	125	125
query91	131	120	96	96
query92	87	69	67	67
query93	2026	1116	706	706
query94	626	251	227	227
query95	518	320	231	231
query96	816	574	274	274
query97	1041	1068	1008	1008
query98	163	133	142	133
query99	418	340	309	309
Total cold run time: 179706 ms
Total hot run time: 81695 ms

hello-stephen · 2026-08-27T15:19:28Z

ClickBench: Total hot run time: 14.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3b92a3f4b1f7d4df4cad571b82f4fcdaae66a24, data reload: false
query1	0.01	0.01	0.01
query2	0.08	0.03	0.04
query3	0.24	0.11	0.11
query4	1.60	0.10	0.10
query5	0.18	0.16	0.17
query6	1.24	0.70	0.66
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.28	0.21	0.20
query10	0.35	0.37	0.36
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.31	0.30	0.29
query14	0.44	0.44	0.44
query15	0.37	0.34	0.35
query16	0.23	0.20	0.21
query17	0.71	0.66	0.69
query18	0.18	0.17	0.17
query19	1.22	1.14	1.15
query20	0.01	0.01	0.01
query21	15.47	0.16	0.11
query22	5.09	0.05	0.05
query23	16.18	0.24	0.11
query24	3.01	0.32	0.24
query25	0.08	0.04	0.03
query26	0.78	0.17	0.13
query27	0.03	0.04	0.03
query28	3.62	0.52	0.27
query29	12.47	3.18	2.55
query30	0.25	0.10	0.11
query31	2.75	0.38	0.17
query32	3.54	0.32	0.23
query33	1.46	1.33	1.37
query34	15.38	2.12	1.77
query35	1.72	1.70	1.69
query36	0.46	0.30	0.29
query37	0.06	0.04	0.03
query38	0.04	0.02	0.02
query39	0.03	0.03	0.02
query40	0.12	0.08	0.07
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 90.51 s
Total hot run time: 14.45 s

hello-stephen · 2026-08-27T15:34:06Z

BE UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	62.70% (29321/46765)
Line Coverage	47.68% (306698/643270)
Region Coverage	43.34% (247825/571847)
Branch Coverage	44.88% (115311/256958)

hello-stephen · 2026-08-27T18:30:35Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.00% (34421/45293)
Line Coverage	60.96% (387623/635907)
Region Coverage	57.11% (325359/569664)
Branch Coverage	57.89% (148321/256225)

Mryange · 2026-08-28T07:03:36Z

run nonConcurrent

hello-stephen · 2026-08-28T09:52:09Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (3/3) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.06% (34450/45293)
Line Coverage	61.01% (387996/635907)
Region Coverage	57.19% (325786/569664)
Branch Coverage	57.96% (148504/256225)

Thrift external scanner responses encoded Doris `DATETIME` values with the BE session timezone in the Arrow schema. This made clients interpret wall-clock values as instants and could introduce an eight-hour offset. The MemoryScratch sink now uses the existing naive-timestamp conversion for `DATETIMEV2`, while preserving timezone-aware `TIMESTAMPTZ` behavior. ### Release note `DATETIME` values returned through the Thrift external scanner are represented as timezone-naive Arrow timestamps. ### Check List (For Author) - Test  - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [ ] Yes.  - Does this need documentation? - [ ] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

#67232 (#67299) Cherry-picked from #67232 Co-authored-by: Mryange <yanxuecheng@selectdb.com>

[fix](exec) Return Thrift DATETIME as naive Arrow timestamp
a3b92a3

yiguolei approved these changes Aug 28, 2026
View reviewed changes

yiguolei merged commit 2689e0d into apache:masterAug 28, 2026
30 of 31 checks passed

yiguolei added dev/4.0.x dev/4.1.x labels Aug 28, 2026

github-actionsBot mentioned this pull request Aug 28, 2026
branch-4.1: [fix](exec) Return Thrift DATETIME as naive Arrow timestamp #67232 #67299
Merged

yiguolei added dev/4.1.4-merged and removed dev/4.1.x labels Aug 29, 2026

yiguolei pushed a commit that referenced this pull request Aug 29, 2026
branch-4.1: [fix](exec) Return Thrift DATETIME as naive Arrow timestamp
790e320
#67232 (#67299) Cherry-picked from #67232 Co-authored-by: Mryange <yanxuecheng@selectdb.com>

JNSimba mentioned this pull request Aug 31, 2026
[Fix] Recover DATETIME wall clock from Arrow timezone apache/doris-spark-connector#377
Merged
5 tasks

yiguolei mentioned this pull request Sep 1, 2026
4.1.4 Release Note #67355
Open

morningman mentioned this pull request Sep 1, 2026
[Tracking] Arrow Flight SQL and ADBC Catalog: issues and improvements #65615
Open
63 tasks

Navigation Menu

Uh oh!

[fix](exec) Return Thrift DATETIME as naive Arrow timestamp - #67232

[fix](exec) Return Thrift DATETIME as naive Arrow timestamp#67232
yiguolei merged 1 commit into
apache:masterfrom
Mryange:fix-thrift-datetime-naive

Mryange commented Aug 27, 2026

Uh oh!

Mryange commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

Mryange commented Aug 28, 2026

Uh oh!

hello-stephen commented Aug 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Mryange commented Aug 27, 2026

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Mryange commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

Uh oh!

hello-stephen commented Aug 27, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Aug 27, 2026

BE Regression && UT Coverage Report

Uh oh!

Mryange commented Aug 28, 2026

Uh oh!

hello-stephen commented Aug 28, 2026

BE Regression && UT Coverage Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants