[fix](csv_reader)fix bug that Read garbled files caused be crash. by hubgeter · Pull Request #24164 · apache/doris

hubgeter · 2023-09-11T02:55:47Z

Proposed changes

Issue Number: close #xxx
fix bug that read garbled files caused be crash.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

hubgeter · 2023-09-11T02:56:05Z

run buildall

github-actions · 2023-09-11T03:02:18Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-09-11T03:06:35Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-09-11T03:07:33Z

run buildall

doris-robot · 2023-09-11T03:25:24Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7919/21441)
Line Coverage: 28.96% (63630/219697)
Region Coverage: 27.89% (33001/118341)
Branch Coverage: 24.46% (16944/69272)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e13cc41c6ba4e925af232dd3dec58757a272651b_e13cc41c6ba4e925af232dd3dec58757a272651b/report/index.html

doris-robot · 2023-09-11T03:26:09Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 49.39 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162247020 Bytes

github-actions · 2023-09-11T07:08:41Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-09-11T07:58:24Z

run buildall

doris-robot · 2023-09-11T08:19:13Z

TeamCity be ut coverage result:
Function Coverage: 36.92% (7917/21443)
Line Coverage: 28.95% (63614/219718)
Region Coverage: 27.89% (33009/118361)
Branch Coverage: 24.46% (16945/69282)
Coverage Report: http://coverage.selectdb-in.cc/coverage/61df14a537e6029eae6b741fadfc46475386dfdc_61df14a537e6029eae6b741fadfc46475386dfdc/report/index.html

doris-robot · 2023-09-11T08:31:51Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.91 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162237045 Bytes

morningman

Please add test case

hubgeter · 2023-09-12T01:07:46Z

run buildall

github-actions · 2023-09-12T01:15:44Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-09-12T02:50:50Z

run buildall

github-actions · 2023-09-12T02:55:14Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-12T03:16:00Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7920/21444)
Line Coverage: 28.97% (63667/219760)
Region Coverage: 27.89% (33033/118444)
Branch Coverage: 24.47% (16968/69334)
Coverage Report: http://coverage.selectdb-in.cc/coverage/adc834e56310a7dc08c8d1c29fc36b34a99ed652_adc834e56310a7dc08c8d1c29fc36b34a99ed652/report/index.html

doris-robot · 2023-09-12T03:23:25Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.07 seconds
stream load tsv: 587 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162172746 Bytes

hubgeter · 2023-09-12T05:08:58Z

run buildall

github-actions · 2023-09-12T05:15:38Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-09-12T05:27:17Z

TeamCity be ut coverage result:
Function Coverage: 36.93% (7919/21443)
Line Coverage: 28.97% (63669/219745)
Region Coverage: 27.90% (33048/118448)
Branch Coverage: 24.48% (16976/69342)
Coverage Report: http://coverage.selectdb-in.cc/coverage/262454ba2173a34fdc7afcd4ef981fe7645bfb1a_262454ba2173a34fdc7afcd4ef981fe7645bfb1a/report/index.html

doris-robot · 2023-09-12T05:38:39Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.78 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162230818 Bytes

morningman

LGTM

github-actions · 2023-09-12T16:30:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-09-12T16:30:51Z

PR approved by anyone and no changes requested.

…4164) fix bug that read garbled files caused be crash.

morningman added the dev/2.0.2 label Sep 11, 2023

morningman reviewed Sep 11, 2023
View reviewed changes

hubgeter force-pushed the fix_csv_garbled_file branch from 61df14a to 7bdf10cCompare September 12, 2023 01:07

hubgeter force-pushed the fix_csv_garbled_file branch from 7bdf10c to adc834eCompare September 12, 2023 02:48

hubgeter added 6 commits September 12, 2023 13:08

[fix](csv_reader)fix bug that Reading garbled files caused be crash.
ea652ef

fix format
a1290d9

fix error
c042927

append regression test
e2c989c

fix format
d0e4d49

fix build error
262454b

hubgeter force-pushed the fix_csv_garbled_file branch from adc834e to 262454bCompare September 12, 2023 05:08

xiaokang added the p0_c label Sep 12, 2023

morningman approved these changes Sep 12, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Sep 12, 2023

github-actionsBot added the reviewed label Sep 12, 2023

zy-kkk approved these changes Sep 13, 2023
View reviewed changes

morningman merged commit e30c3f3 into apache:masterSep 13, 2023

xiaokang added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 13, 2023

xiaokang pushed a commit that referenced this pull request Sep 13, 2023
[fix](csv_reader)fix bug that Read garbled files caused be crash. (#2…
d37c92b
…4164) fix bug that read garbled files caused be crash.

xiaokang mentioned this pull request Sep 28, 2023
Release Note 2.0.2 #25011
Closed

Uh oh!

Conversation

hubgeter commented Sep 11, 2023

Proposed changes

Further comments

Uh oh!

hubgeter commented Sep 11, 2023

Uh oh!

github-actionsBot commented Sep 11, 2023

Uh oh!

github-actionsBot commented Sep 11, 2023

Uh oh!

hubgeter commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

github-actionsBot commented Sep 11, 2023

Uh oh!

hubgeter commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

doris-robot commented Sep 11, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Sep 12, 2023

Uh oh!

github-actionsBot commented Sep 12, 2023

Uh oh!

hubgeter commented Sep 12, 2023

Uh oh!

github-actionsBot commented Sep 12, 2023

Uh oh!

doris-robot commented Sep 12, 2023

Uh oh!

doris-robot commented Sep 12, 2023

Uh oh!

hubgeter commented Sep 12, 2023

Uh oh!

github-actionsBot commented Sep 12, 2023

Uh oh!

doris-robot commented Sep 12, 2023

Uh oh!

doris-robot commented Sep 12, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Sep 12, 2023

Uh oh!

github-actionsBot commented Sep 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants