Uh oh!
There was an error while loading. Please reload this page.
[Fix](parquet-reader) Fix and optimize parquet min-max filtering. - #38277
Merged
morningman merged 5 commits intoAug 12, 2024
Merged
Conversation
doris-robot
commented
Jul 24, 2024
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
kaka11chen
commented
Jul 24, 2024
ContributorAuthor
run buildall |
kaka11chenforce-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
July 24, 2024 02:20
96e0ee0 to
bd5d879Comparekaka11chen
commented
Jul 24, 2024
ContributorAuthor
run buildall |
| #pragma once | ||
| #include <gen_cpp/parquet_types.h> |
Contributor
There was a problem hiding this comment.
warning: 'gen_cpp/parquet_types.h' file not found [clang-diagnostic-error]
#include<gen_cpp/parquet_types.h>
^| ", semver=" + (version ? *version : "null") + | ||
| ", appBuildHash=" + (appBuildHash ? *appBuildHash : "null") + ")"; | ||
| } | ||
Contributor
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
Suggested change
| public: |
Additional context
be/src/vec/exec/format/parquet/parquet_common.h:166: previously declared here
public:
^| return Status::OK(); | ||
| } | ||
| int compareTo(const SemanticVersion& other) const { |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (intcmp=compareIntegers(_major, other._major); cmp!=0) returncmp; | |
| if (intcmp=compareIntegers(_major, other._major); cmp!=0) { returncmp; | |
| } |
| } | ||
| int compareTo(const SemanticVersion& other) const { | ||
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (intcmp=compareIntegers(_minor, other._minor); cmp!=0) returncmp; | |
| if (intcmp=compareIntegers(_minor, other._minor); cmp!=0) { returncmp; | |
| } |
| int compareTo(const SemanticVersion& other) const { | ||
| if (int cmp = compareIntegers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = compareIntegers(_minor, other._minor); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (intcmp=compareIntegers(_patch, other._patch); cmp!=0) returncmp; | |
| if (intcmp=compareIntegers(_patch, other._patch); cmp!=0) { returncmp; | |
| } |
| ParquetStatisticsTest() {} | ||
| }; | ||
| TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) {
^Additional context
be/test/vec/exec/parquet/parquet_statistics_test.cpp:30: 121 lines including whitespace and comments (threshold 80)
TEST_F(ParquetStatisticsTest, test_try_read_old_utf8_stats) {
^| // specific language governing permissions and limitations | ||
| // under the License. | ||
| #include <gtest/gtest.h> |
Contributor
There was a problem hiding this comment.
warning: 'gtest/gtest.h' file not found [clang-diagnostic-error]
#include<gtest/gtest.h>
^
Comment on lines
+24
to
+25
| namespace doris { | ||
| namespace vectorized { |
Contributor
There was a problem hiding this comment.
warning: nested namespaces can be concatenated [modernize-concat-nested-namespaces]
Suggested change
| namespacedoris { | |
| namespacevectorized { | |
| namespacedoris::vectorized { |
be/test/vec/exec/parquet/parquet_version_test.cpp:219:
- } // namespace vectorized- } // namespace doris+ } // namespace doris| namespace vectorized { | ||
| class ParquetVersionTest : public testing::Test { | ||
| public: | ||
| ParquetVersionTest() {} |
Contributor
There was a problem hiding this comment.
warning: use '= default' to define a trivial default constructor [modernize-use-equals-default]
Suggested change
| ParquetVersionTest() {} | |
| ParquetVersionTest() = default; |
| ParquetVersionTest() {} | ||
| }; | ||
| TEST_F(ParquetVersionTest, test_version_parser) { |
Contributor
There was a problem hiding this comment.
warning: function 'TEST_F' exceeds recommended size/complexity thresholds [readability-function-size]
TEST_F(ParquetVersionTest, test_version_parser) {
^Additional context
be/test/vec/exec/parquet/parquet_version_test.cpp:30: 91 lines including whitespace and comments (threshold 80)
TEST_F(ParquetVersionTest, test_version_parser) {
^doris-robot
commented
Jul 24, 2024
TPC-H: Total hot run time: 39895 ms |
doris-robot
commented
Jul 24, 2024
TPC-DS: Total hot run time: 174183 ms |
doris-robot
commented
Jul 24, 2024
ClickBench: Total hot run time: 31.61 s |
kaka11chenforce-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
July 24, 2024 05:00
bd5d879 to
e1d5ba2Comparekaka11chen
commented
Jul 24, 2024
ContributorAuthor
run buildall |
kaka11chenforce-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
July 24, 2024 06:27
e1d5ba2 to
1ce8cedComparekaka11chen
commented
Jul 24, 2024
ContributorAuthor
run buildall |
kaka11chenforce-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
July 24, 2024 06:40
1ce8ced to
b42b73aComparekaka11chen
commented
Jul 24, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Jul 24, 2024
TPC-H: Total hot run time: 39838 ms |
doris-robot
commented
Jul 24, 2024
TPC-DS: Total hot run time: 173837 ms |
doris-robot
commented
Jul 24, 2024
ClickBench: Total hot run time: 30.21 s |
kaka11chen
marked this pull request as ready for review
August 2, 2024 08:40
kaka11chen
commented
Aug 2, 2024
ContributorAuthor
run buildall |
| } | ||
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) { return cmp; | |
| } |
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) { return cmp; | |
| } |
| int SemanticVersion::compare_to(const SemanticVersion& other) const { | ||
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; | |
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) { return cmp; | |
| } |
| if (int cmp = _compare_integers(_major, other._major); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_minor, other._minor); cmp != 0) return cmp; | ||
| if (int cmp = _compare_integers(_patch, other._patch); cmp != 0) return cmp; | ||
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) return cmp; |
Contributor
There was a problem hiding this comment.
warning: statement should be inside braces [readability-braces-around-statements]
Suggested change
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) return cmp; | |
| if (int cmp = _compare_booleans(other._prerelease, _prerelease); cmp != 0) { return cmp; | |
| } |
| std::vector<NumberOrString> _identifiers; | ||
| }; | ||
| private: |
Contributor
There was a problem hiding this comment.
warning: redundant access specifier has the same accessibility as the previous access specifier [readability-redundant-access-specifiers]
Suggested change
| private: |
Additional context
be/src/vec/exec/format/parquet/parquet_common.h:218: previously declared here
private:
^kaka11chen
commented
Aug 2, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Aug 2, 2024
TPC-H: Total hot run time: 41555 ms |
doris-robot
commented
Aug 2, 2024
TPC-DS: Total hot run time: 168434 ms |
doris-robot
commented
Aug 2, 2024
ClickBench: Total hot run time: 29.91 s |
kaka11chenforce-pushed
the
fix_and_opt_parquet_min_max-master
branch
from
August 5, 2024 14:50
7cc53a3 to
e8f82a9Comparekaka11chen
commented
Aug 5, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Aug 5, 2024
TPC-H: Total hot run time: 42085 ms |
doris-robot
commented
Aug 5, 2024
TPC-DS: Total hot run time: 169444 ms |
doris-robot
commented
Aug 5, 2024
ClickBench: Total hot run time: 29.73 s |
kaka11chen
commented
Aug 7, 2024
ContributorAuthor
run buildall |
doris-robot
commented
Aug 7, 2024
TPC-H: Total hot run time: 41876 ms |
doris-robot
commented
Aug 7, 2024
TPC-DS: Total hot run time: 168872 ms |
doris-robot
commented
Aug 7, 2024
ClickBench: Total hot run time: 30.16 s |
Contributor
PR approved by at least one committer and no changes requested. |
Contributor
PR approved by anyone and no changes requested. |
wyxxxcat pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Aug 14, 2024
…ache#38277) ## Proposed changes Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
kaka11chen added a commit
to kaka11chen/doris
that referenced
this pull request
Aug 14, 2024
…ache#38277) ## Proposed changes Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
yiguolei pushed a commit
that referenced
this pull request
Aug 15, 2024
kaka11chen added a commit
to kaka11chen/doris
that referenced
this pull request
Oct 9, 2024
…ache#38277) Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
morningman pushed a commit
that referenced
this pull request
Oct 10, 2024
HappenLee pushed a commit
to HappenLee/incubator-doris
that referenced
this pull request
Apr 24, 2026
…ache#38277) ## Proposed changes Refer to trino's implementation - Some bugs in the historical version paquet-mr. Use `CorruptStatistics::should_ignore_statistics()` to handle. - The old version of parquet uses `min` and `max` stats, and later implements `min_value` and `max_value`. `Min`/`max` stats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values. - If it is double or float, special cases such as NaN, -0, and 0 must be handled. - If the string type only has min and max stats, but no min_value or max_value, use `ParquetPredicate::_try_read_old_utf8_stats()` to expand the range reading optimization method for optimization.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
Refer to trino's implementation
Some bugs in the historical version paquet-mr. Use
CorruptStatistics::should_ignore_statistics()to handle.The old version of parquet uses
minandmaxstats, and later implementsmin_valueandmax_value.Min/maxstats cannot be used for some types and in some cases. This is related to the comparison and sorting method of values.If it is double or float, special cases such as NaN, -0, and 0 must be handled.
If the string type only has min and max stats, but no min_value or max_value, use
ParquetPredicate::_try_read_old_utf8_stats()to expand the range reading optimization method for optimization.