Skip to content

[fix](Table-Valued Function) fix be core when user sepcified empty column_separator using hdfs tvf - #24369

Merged
xiaokang merged 2 commits into
apache:masterfrom
BePPPower:fixTvfColumnSeparator
Sep 14, 2023
Merged

[fix](Table-Valued Function) fix be core when user sepcified empty column_separator using hdfs tvf#24369
xiaokang merged 2 commits into
apache:masterfrom
BePPPower:fixTvfColumnSeparator

Conversation

@BePPPower

@BePPPowerBePPPower commented Sep 14, 2023

Copy link
Copy Markdown
Contributor

Proposed changes

Issue Number: close #xxx

  1. Fix be core when user sepcified empty column_separator using hdfs tvf. Now we forbid users from specifying column_separator as empty string.
  2. tvf supports to parse invisible character.
  3. Refresh the out file again

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BePPPower

Copy link
Copy Markdown
ContributorAuthor

run buildall

morningman
morningman previously approved these changes Sep 14, 2023

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2023
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@hubgeter

Copy link
Copy Markdown
Contributor

LGTM

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.84 seconds
stream load tsv: 598 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162438724 Bytes

@BePPPower

Copy link
Copy Markdown
ContributorAuthor

run buildall

@github-actionsgithub-actionsBot removed the approved Indicates a PR has been approved by one committer. label Sep 14, 2023
@BePPPower

Copy link
Copy Markdown
ContributorAuthor

run buildall

@BePPPower

Copy link
Copy Markdown
ContributorAuthor

run buildall

@morningmanmorningman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actionsgithub-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 14, 2023

@xiaokangxiaokang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot

Copy link
Copy Markdown

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.1 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162308802 Bytes

@xiaokang
xiaokang merged commit c5ef6cf into apache:masterSep 14, 2023
xiaokang pushed a commit that referenced this pull request Sep 14, 2023
@xiaokangxiaokang mentioned this pull request Sep 28, 2023
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approvedIndicates a PR has been approved by one committer.dev/2.0.2-mergedp0_breviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

@BePPPower@hubgeter@doris-robot@xiaokang@morningman