[fix](Tvf) return empty set when tvf queries an empty file or an error uri by BePPPower · Pull Request #25280 · apache/doris

BePPPower · 2023-10-11T02:39:07Z

Proposed changes

Issue Number:

Before:

return errors when tvf queries an empty file or an error uri:

get parsed schema failed, empty csv file
Can not get first file, please check uri.

Now:

we just return empty set when tvf queries an empty file or an error uri.

mysql>select*from s3( "uri"="https://error_uri/exp_1.csv", "s3.access_key"="xx", "s3.secret_key"="yy", "format"="csv") limit10;
Empty set (1.29 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-10-11T02:47:13Z

clang-tidy review says "All clean, LGTM! 👍"

BePPPower · 2023-10-11T03:48:55Z

run buildall

github-actions · 2023-10-11T03:56:26Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2023-10-11T04:25:09Z

TeamCity be ut coverage result:
Function Coverage: 36.24% (8149/22486)
Line Coverage: 28.35% (65162/229840)
Region Coverage: 27.28% (33768/123775)
Branch Coverage: 23.98% (17205/71760)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9d21faf52898085daf9204c6ce1bdb8bac883c6c_9d21faf52898085daf9204c6ce1bdb8bac883c6c/report/index.html

doris-robot · 2023-10-11T04:29:08Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.47 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162315204 Bytes

morningman · 2023-10-11T14:39:31Z

 bool is_parse_name = false;
 RETURN_IF_ERROR(_prepare_parse(&read_line, &is_parse_name));

+ if (_file_reader->size() == 0) {


Need to add comment to explain why return OK when file size is 0

morningman · 2023-10-11T14:39:38Z

 RETURN_IF_ERROR(_line_reader->read_line(&ptr, &size, &_line_reader_eof, _io_ctx));
 if (size == 0) {
- return Status::InternalError("The first line is empty, can not parse column names");
+ return Status::OK();


morningman · 2023-10-11T14:43:58Z

 TNetworkAddress address = new TNetworkAddress(be.getHost(), be.getBrpcPort());
 try {
 PFetchTableSchemaRequest request = getFetchTableStructureRequest();
+ if (request == null) {


unify it to fillColumns

BePPPower · 2023-10-12T05:15:25Z

run buildall

doris-robot · 2023-10-12T05:55:22Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.86 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162376826 Bytes

BePPPower · 2023-10-12T07:02:30Z

run buildall

BePPPower · 2023-10-12T07:04:28Z

run buildall

doris-robot · 2023-10-12T07:42:06Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.9 seconds
stream load tsv: 552 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17161998075 Bytes

doris-robot · 2023-10-12T07:42:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.98 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162202182 Bytes

BePPPower · 2023-10-12T08:20:17Z

run buildall

doris-robot · 2023-10-12T09:45:32Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162410602 Bytes

BePPPower · 2023-10-12T14:05:06Z

run feut

BePPPower · 2023-10-13T03:01:07Z

run buildall

doris-robot · 2023-10-13T03:45:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.03 seconds
stream load tsv: 550 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162093248 Bytes

morningman

LGTM

github-actions · 2023-10-13T07:44:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-13T07:44:39Z

PR approved by anyone and no changes requested.

morningman · 2023-10-13T07:49:02Z

run buildall

doris-robot · 2023-10-13T08:44:17Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.32 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 22 seconds loaded 2358488459 Bytes, about 102 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162014325 Bytes

BePPPower · 2023-10-13T10:32:11Z

run p0

BePPPower · 2023-10-13T10:39:28Z

run buildall

doris-robot · 2023-10-13T11:11:41Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.97 seconds
stream load tsv: 554 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162064322 Bytes

yiguolei

lgtm

github-actions · 2023-10-14T14:40:50Z

PR approved by at least one committer and no changes requested.

morningman

LGTM

…r uri (apache#25280) ### Before: return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. ### Now: we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ```

…r uri (apache#25280) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ```

…r uri (#25280) (#31110) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ``` Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>

…r uri (apache#25280) (apache#31110) return errors when tvf queries an empty file or an error uri: 1. get parsed schema failed, empty csv file 2. Can not get first file, please check uri. we just return empty set when tvf queries an empty file or an error uri. ```sql mysql> select * from s3( "uri" = "https://error_uri/exp_1.csv", "s3.access_key"= "xx", "s3.secret_key" = "yy", "format" = "csv") limit 10; Empty set (1.29 sec) ``` Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>

xiaokang added the dev/2.0.3 label Oct 11, 2023

morningman reviewed Oct 11, 2023
View reviewed changes

BePPPower force-pushed the fixTvfEmptyFile branch from eada360 to a51b15eCompare October 13, 2023 03:00

morningman reviewed Oct 13, 2023
View reviewed changes

Comment threadfe/fe-core/src/main/java/org/apache/doris/tablefunction/LocalTableValuedFunction.java Outdated

BePPPower force-pushed the fixTvfEmptyFile branch from a51b15e to df2fd0fCompare October 13, 2023 06:16

BePPPower added 5 commits October 13, 2023 14:50

fix
b9b7c9b

add test
9a5ff4b

fix code
3e8dbed

modify docs and add tests
d612d43

add hdfs test
94d168b

BePPPower force-pushed the fixTvfEmptyFile branch from df2fd0f to 94d168bCompare October 13, 2023 06:51

add docs
abb9bbc

morningman previously approved these changes Oct 13, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 13, 2023

github-actionsBot added the reviewed label Oct 13, 2023

add en docs
52f04d1

BePPPower dismissed morningman’s stale review via 52f04d1October 13, 2023 10:39

github-actionsBot removed the approved Indicates a PR has been approved by one committer. label Oct 13, 2023

yiguolei approved these changes Oct 14, 2023
View reviewed changes

github-actionsBot added the approved Indicates a PR has been approved by one committer. label Oct 14, 2023

morningman approved these changes Oct 16, 2023
View reviewed changes

morningman removed the dev/2.0.3 label Oct 16, 2023

morningman merged commit 85b8497 into apache:masterOct 17, 2023

zxealous mentioned this pull request Feb 19, 2024
[fix](tvf) Fix tvf read empty file fail #31110
Merged

morningman added the dev/2.0.5-merged label Feb 19, 2024

Uh oh!

Conversation

BePPPower commented Oct 11, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Before:

Now:

Further comments

Uh oh!

github-actionsBot commented Oct 11, 2023

Uh oh!

BePPPower commented Oct 11, 2023

Uh oh!

github-actionsBot commented Oct 11, 2023

Uh oh!

doris-robot commented Oct 11, 2023

Uh oh!

doris-robot commented Oct 11, 2023

Uh oh!

morningmanOct 11, 2023

Choose a reason for hiding this comment

Uh oh!

morningmanOct 11, 2023

Choose a reason for hiding this comment

Uh oh!

morningmanOct 11, 2023

Choose a reason for hiding this comment

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

doris-robot commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 12, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 13, 2023

Uh oh!

github-actionsBot commented Oct 13, 2023

Uh oh!

morningman commented Oct 13, 2023

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

BePPPower commented Oct 13, 2023

Uh oh!

doris-robot commented Oct 13, 2023

Uh oh!

yiguolei left a comment

Choose a reason for hiding this comment

Uh oh!

github-actionsBot commented Oct 14, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

BePPPower commented Oct 11, 2023•
edited
Loading