[fix](csv_reader)Fixed bug when parsing multi-character delimiters. by hubgeter · Pull Request #24572 · apache/doris

hubgeter · 2023-09-18T13:29:22Z

Proposed changes

Fixed bug when parsing multi-character delimiters.

example:
column_separator : "xx"
data.csv : data1xxxxdata2

Parse incorrectly:
data1[xx]xxdata2
data1x[xx]xdata2
data1xx[xx]data2
The string "xxxx" is parsed into three "xx" delimiters.

Parse correctly:
data1[xx]xxdata2
data1xx[xx]data2

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hubgeter · 2023-09-18T13:29:55Z

run buildall

github-actions · 2023-09-18T13:38:06Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T13:47:20Z

TeamCity be ut coverage result:
Function Coverage: 36.82% (7979/21669)
Line Coverage: 28.85% (64026/221934)
Region Coverage: 27.73% (33220/119783)
Branch Coverage: 24.36% (17031/69912)
Coverage Report: http://coverage.selectdb-in.cc/coverage/108bfb364d4b1798b8c9efdc9286c5feff563f12_108bfb364d4b1798b8c9efdc9286c5feff563f12/report/index.html

doris-robot · 2023-09-18T14:00:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.17 seconds
stream load tsv: 600 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162267551 Bytes

hubgeter · 2023-09-18T16:55:25Z

run buildall

github-actions · 2023-09-18T17:01:39Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-18T17:22:42Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (7979/21667)
Line Coverage: 28.85% (64028/221937)
Region Coverage: 27.73% (33219/119781)
Branch Coverage: 24.36% (17029/69912)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e47b96e503514421952b08b218dd7bd19512f73_5e47b96e503514421952b08b218dd7bd19512f73/report/index.html

doris-robot · 2023-09-18T17:27:27Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.52 seconds
stream load tsv: 596 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162306529 Bytes

Hastyshell

LGTM

github-actions · 2023-09-19T02:56:46Z

PR approved by anyone and no changes requested.

github-actions · 2023-09-19T13:45:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-19T14:02:02Z

TeamCity be ut coverage result:
Function Coverage: 36.75% (7979/21713)
Line Coverage: 28.82% (64034/222200)
Region Coverage: 27.71% (33237/119933)
Branch Coverage: 24.35% (17039/69988)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5e47b96e503514421952b08b218dd7bd19512f73_5e47b96e503514421952b08b218dd7bd19512f73/report/index.html

morningman · 2023-09-19T14:40:29Z

run buildall

morningman

LGTM

github-actions · 2023-09-19T14:41:44Z

PR approved by at least one committer and no changes requested.

doris-robot · 2023-09-19T14:57:02Z

TeamCity be ut coverage result:
Function Coverage: 36.76% (7981/21713)
Line Coverage: 28.83% (64058/222200)
Region Coverage: 27.72% (33251/119933)
Branch Coverage: 24.35% (17044/69988)
Coverage Report: http://coverage.selectdb-in.cc/coverage/adfae23a8df06fa7e67b0ccad308dd9382a2952f_adfae23a8df06fa7e67b0ccad308dd9382a2952f/report/index.html

hubgeter · 2023-09-20T02:21:09Z

run buildall

github-actions · 2023-09-20T02:27:16Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-20T02:38:08Z

TeamCity be ut coverage result:
Function Coverage: 36.69% (7981/21755)
Line Coverage: 28.74% (64052/222902)
Region Coverage: 27.64% (33249/120284)
Branch Coverage: 24.28% (17039/70168)
Coverage Report: http://coverage.selectdb-in.cc/coverage/942928cdb67ebded758d8e11a47a673cdc2538dd_942928cdb67ebded758d8e11a47a673cdc2538dd/report/index.html

doris-robot · 2023-09-20T02:58:47Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.2 seconds
stream load tsv: 601 seconds loaded 74807831229 Bytes, about 118 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162463195 Bytes

…pache#24572) Fixed bug when parsing multi-character delimiters.

…24572) Fixed bug when parsing multi-character delimiters.

hubgeter marked this pull request as ready for review September 18, 2023 13:29

hubgeter force-pushed the fix_csv_parse branch from 108bfb3 to 5e47b96Compare September 18, 2023 16:54

morningman added dev/2.0.2 p0_b labels Sep 19, 2023

Hastyshell approved these changes Sep 19, 2023
View reviewed changes

github-actionsBot added the reviewed label Sep 19, 2023

hubgeter force-pushed the fix_csv_parse branch from 5e47b96 to adfae23Compare September 19, 2023 13:38

morningman approved these changes Sep 19, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2023

[fix](csv_reader)Fixed bug when parsing multi-character delimiters.
942928c

hubgeter force-pushed the fix_csv_parse branch from adfae23 to 942928cCompare September 20, 2023 02:20

morningman approved these changes Sep 20, 2023
View reviewed changes

morningman merged commit c704497 into apache:masterSep 20, 2023

xuwei0912 mentioned this pull request Sep 20, 2023
[fix](stream_load)Fix multibyte separator csv file split(#24254) #24316
Closed

JingDas pushed a commit to JingDas/doris that referenced this pull request Sep 20, 2023
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (a…
e9fc326
…pache#24572) Fixed bug when parsing multi-character delimiters.

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 20, 2023

xiaokang pushed a commit that referenced this pull request Sep 20, 2023
[fix](csv_reader)Fixed bug when parsing multi-character delimiters. (#…
3965172
…24572) Fixed bug when parsing multi-character delimiters.

xiaokang mentioned this pull request Sep 28, 2023
Release Note 2.0.2 #25011
Closed

Uh oh!

Conversation

hubgeter commented Sep 18, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

hubgeter commented Sep 18, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

hubgeter commented Sep 18, 2023

Uh oh!

github-actionsBot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

doris-robot commented Sep 18, 2023

Uh oh!

Hastyshell left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

morningman commented Sep 19, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 19, 2023

Uh oh!

doris-robot commented Sep 19, 2023

Uh oh!

hubgeter commented Sep 20, 2023

Uh oh!

github-actionsBot commented Sep 20, 2023

Uh oh!

doris-robot commented Sep 20, 2023

Uh oh!

doris-robot commented Sep 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hubgeter commented Sep 18, 2023•
edited
Loading