Skip to content

[improvement](function) improve date_trunc function performance when timeunit is const - #25824

Merged
yiguolei merged 5 commits into
apache:masterfrom
zhangstar333:date_trunc
Oct 26, 2023
Merged

[improvement](function) improve date_trunc function performance when timeunit is const#25824
yiguolei merged 5 commits into
apache:masterfrom
zhangstar333:date_trunc

Conversation

@zhangstar333

Copy link
Copy Markdown
Contributor

this PR #22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@HappenLee

Copy link
Copy Markdown
Contributor

run buildall

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

better use function call ptr to replace strncmp? like "re"

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.59 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162165603 Bytes

@zhangstar333

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.09 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162036189 Bytes

@zhangstar333

Copy link
Copy Markdown
ContributorAuthor

run buildall

HappenLee
HappenLee previously approved these changes Oct 25, 2023

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.27 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162022534 Bytes

@zhangstar333

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162539108 Bytes

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@HappenLeeHappenLee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei
yiguolei merged commit da4de17 into apache:masterOct 26, 2023
dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
…timeunit is const (apache#25824)
this PR apache#22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.
xiaokang pushed a commit to xiaokang/doris that referenced this pull request Oct 31, 2023
…timeunit is const (apache#25824)
this PR apache#22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.
xiaokang pushed a commit to xiaokang/doris that referenced this pull request Nov 1, 2023
…timeunit is const (apache#25824)
this PR apache#22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
…timeunit is const (apache#25824)
this PR apache#22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
…timeunit is const (apache#25824)
this PR apache#22602 have check function.
only support date_trunc(column, const), so the second must be const literal
and no need to check time unit every row.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.3-mergedmerge_conflictreviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants

@zhangstar333@HappenLee@doris-robot@BiteTheDDDDt@xiaokang@yiguolei