Skip to content

[fix](stats) truncate min/max if too long (#27955) - #29196

Merged
xiaokang merged 1 commit into
apache:branch-2.0from
Jibing-Li:pickjichang
Dec 28, 2023
Merged

[fix](stats) truncate min/max if too long (#27955)#29196
xiaokang merged 1 commit into
apache:branch-2.0from
Jibing-Li:pickjichang

Conversation

@Jibing-Li

Copy link
Copy Markdown
Contributor

For some string value the max/min might be a very long string which might take too many memory of FE,
so we truncate to 1024 chars if it's too long

backport #27955

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

For some string value the max/min might be a very long string
which might take too many memory of FE,
so we truncate to 1024 chars if it's too long
@Jibing-Li
Jibing-Li marked this pull request as ready for review December 28, 2023 03:18
@Jibing-Li

Copy link
Copy Markdown
ContributorAuthor

run buildall

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.85 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 30.0 seconds inserted 10000000 Rows, about 333K ops/s
storage size: 17163024191 Bytes

@xiaokang
xiaokang merged commit 388a158 into apache:branch-2.0Dec 28, 2023
@Jibing-Li
Jibing-Li deleted the pickjichang branch December 28, 2023 14:22
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@Jibing-Li@doris-robot@xiaokang@Kikyou1997