Skip to content

[fix](orc) decode timestamp through serde - #64807

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
xylaaaaa:codex/orc-timestamp-prefix
Jul 13, 2026
Merged

[fix](orc) decode timestamp through serde#64807
yiguolei merged 1 commit into
apache:branch-4.1from
xylaaaaa:codex/orc-timestamp-prefix

Conversation

@xylaaaaa

@xylaaaaaxylaaaaa commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Proposed changes

Fix ORC timestamp decoding to round nanoseconds to Doris microseconds instead of truncating them. This keeps CAST(timestamp AS VARCHAR) aligned with Hive/Trino prefix expectations for values like 2020-01-02 03:04:05.321.

The same decode path is used by nested timestamps in array/map/struct columns, so this also covers complex type projections.

Problem summary

ORC stores timestamp fractional seconds as nanoseconds, while Doris DATETIMEV2(6) keeps microseconds. The previous conversion truncated nanos with / 1000, so an ORC value such as 320999999ns became .320999 instead of .321000. Prefix predicates like:

CAST(ts ASVARCHAR) LIKE'2020-01-02 03:04:05.321%'

could therefore miss rows created by Hive/Trino ORC writers.

Solution

  • Round ORC nanoseconds to microseconds during timestamp decode.
  • Carry 999999500ns and above into the next second.
  • Apply the same helper to TIMESTAMP and TIMESTAMP_INSTANT decode paths.
  • Add a BE unit test covering rounding and second carry.

Test plan

  • ninja -j 8 doris_be_test
  • ./be/ut_build_RELEASE/test/doris_be_test --gtest_filter='OrcReaderFillDataTest.TestTimestampNanosecondsRoundToMicroseconds'
  • ./be/ut_build_RELEASE/test/doris_be_test --gtest_filter='OrcReaderFillDataTest.*'
  • git diff --check

@xylaaaaa
xylaaaaa requested a review from yiguolei as a code ownerJune 25, 2026 02:25
CopilotAI review requested due to automatic review settings June 25, 2026 02:25
@hello-stephen

hello-stephen commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

CopilotAI reviewed Jun 25, 2026

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@xylaaaaa
[fix](orc) decode timestamp through serde (apache#64854)
18dd1c4
- remove the ORC leaf-type decode switch from
`OrcReader::_fill_doris_data_column`
- add `DataTypeSerDe::read_column_from_orc` implementations for ORC leaf
types produced by the reader: numeric, decimal, datev2, string,
varbinary, datetimev2, and timestamptz
- keep ARRAY/MAP/STRUCT traversal in ORC serde/context while nested leaf
values are decoded through their nested type serde
- fix ORC timestamp and nested timestamp decoding so nanoseconds are
truncated to the target timestamp scale before string/predicate
evaluation
- fix ORC DECIMALV2 decode to rescale file values into the fixed
in-memory scale 9 representation
- preserve ORC CHAR trimming based on ORC physical CHAR type only
(cherry picked from commit fcffadc)
@xylaaaaa
xylaaaaaforce-pushed the codex/orc-timestamp-prefix branch from 3074203 to 18dd1c4CompareJuly 8, 2026 06:02
@xylaaaaaxylaaaaa changed the title [fix](orc) Round timestamp nanos to micros[fix](orc) decode timestamp through serdeJul 8, 2026
@xylaaaaa

xylaaaaa commented Jul 8, 2026

Copy link
Copy Markdown
ContributorAuthor

run buildall

@hello-stephen

hello-stephen commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 87.50% (7/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage55.82% (21340/38233)
Line Coverage38.70% (202521/523319)
Region Coverage35.13% (159043/452677)
Branch Coverage36.06% (69363/192359)

@hello-stephen

hello-stephen commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.39% (27042/37358)
Line Coverage55.36% (288323/520820)
Region Coverage52.96% (241379/455767)
Branch Coverage53.95% (103944/192651)

@xylaaaaa

xylaaaaa commented Jul 9, 2026

Copy link
Copy Markdown
ContributorAuthor

run beut

@xylaaaaa

xylaaaaa commented Jul 9, 2026

Copy link
Copy Markdown
ContributorAuthor

run p0

@xylaaaaa

xylaaaaa commented Jul 9, 2026

Copy link
Copy Markdown
ContributorAuthor

run nonConcurrent

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 75.00% (6/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage54.78% (20955/38253)
Line Coverage38.21% (199997/523353)
Region Coverage34.66% (156918/452697)
Branch Coverage35.63% (68543/192359)

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 87.50% (7/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.05% (26929/37376)
Line Coverage55.17% (287356/520852)
Region Coverage52.79% (240629/455785)
Branch Coverage53.81% (103665/192651)

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 87.50% (7/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.05% (26929/37376)
Line Coverage55.18% (287381/520852)
Region Coverage52.80% (240650/455785)
Branch Coverage53.81% (103674/192651)

@hello-stephen

hello-stephen commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 87.50% (7/8) 🎉

Increment coverage report
Complete coverage report

CategoryCoverage
Function Coverage72.10% (26948/37376)
Line Coverage55.23% (287655/520852)
Region Coverage52.83% (240787/455785)
Branch Coverage53.88% (103802/192651)

@github-actions

github-actionsBot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Jul 13, 2026
@github-actions

github-actionsBot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@yiguoleiyiguolei reopened this Jul 13, 2026
@yiguolei
yiguolei merged commit 5fc0d8e into apache:branch-4.1Jul 13, 2026
48 of 53 checks passed
@yiguoleiyiguolei mentioned this pull request Sep 1, 2026
Open
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@yiguoleiyiguoleiyiguolei approved these changes

Assignees

No one assigned

Labels

approvedIndicates a PR has been approved by one committer.reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@xylaaaaa@hello-stephen@yiguolei