Skip to content

[cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug - #41931

Merged
yiguolei merged 6 commits into
apache:branch-2.1from
suxiaogang223:parquet_hdfs_tvf_case
Oct 17, 2024
Merged

[cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug#41931
yiguolei merged 6 commits into
apache:branch-2.1from
suxiaogang223:parquet_hdfs_tvf_case

Conversation

@suxiaogang223

@suxiaogang223suxiaogang223 commented Oct 16, 2024•
edited
Loading

Copy link
Copy Markdown
Member

Proposed changes

pick pr:
#41683
#41506
#41338
#40379

suxiaogang223and others added 5 commits October 16, 2024 10:26
@suxiaogang223
[regression-test](parquet) add some parquet cases with tvf (apache#40379
34ef067
)
Add some parquet file as tvf case
@suxiaogang223
[opt](ci) start dockers in parallel (apache#41338)
fa24bf2
Start docker in parallel to reduce external pipeline time
@suxiaogang223
[fix](parquet) fix dict page offset zero incompatible behaviour (apac…
e30d8bd
…he#41506)
## Proposed changes
Reason: https://issues.apache.org/jira/browse/ARROW-5322
Java readers(parquet-mr) handles "dictionaryPageOffset = 0" to determine
if dictionary page exists where as the C readers uses
"has_dictionaryPageOffset" (_isset bit in thrift message) to determine
the same resulting in incompatible behaviours.
Therefore, we should consider that dicttionary page exists when both
`__isset.dictionary_page_offset` is true and `dictionary_page_offset` is
greater than 0.
@suxiaogang223
[enhance](parquet) support decimal256 for parquet reader (apache#41526)
3ff7697
## Proposed changes
Implemented reading parqeut files with decimal256 type
@suxiaogang223@morningman
[enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet rea…
c476529
…der (apache#41683)
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding
parquet.
relate pr: apache/arrow#42372
> Apache Parquet does not have any encodings suitable for FP data and
the available text compressors (zstd, gzip, etc) do not handle FP data
very well.
It is possible to apply a simple data transformation named "stream
splitting". Such could be "byte stream splitting" which creates K
streams of length N where K is the number of bytes in the data type (4
for floats, 8 for doubles) and N is the number of elements in the
sequence.
---------
Co-authored-by: morningman <morningman@163.com>
@doris-robot

doris-robot commented Oct 16, 2024

Copy link
Copy Markdown

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@suxiaogang223

suxiaogang223 commented Oct 16, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

github-actions[bot]
github-actionsBot reviewed Oct 16, 2024
View reviewed changes

@github-actionsgithub-actionsBot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

Comment threadbe/src/util/byte_stream_split.cpp

#include "byte_stream_split.h"

#include <glog/logging.h>

@github-actionsgithub-actionsBotOct 16, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'glog/logging.h' file not found [clang-diagnostic-error]

#include<glog/logging.h>
^

Comment threadbe/src/vec/exec/format/parquet/parquet_column_convert.h
Comment threadbe/test/util/byte_stream_split_test.cpp
@suxiaogang223

suxiaogang223 commented Oct 17, 2024

Copy link
Copy Markdown
MemberAuthor

run buildall

@doris-robot

doris-robot commented Oct 17, 2024

Copy link
Copy Markdown

TeamCity be ut coverage result:
Function Coverage: 36.02% (9346/25950)
Line Coverage: 27.60% (76788/278218)
Region Coverage: 26.41% (39441/149366)
Branch Coverage: 23.19% (20073/86568)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26132b1f6471b5e601ce31978459c093c3308360_26132b1f6471b5e601ce31978459c093c3308360/report/index.html

@yiguolei
yiguolei merged commit 1b901f6 into apache:branch-2.1Oct 17, 2024
@suxiaogang223
suxiaogang223 deleted the parquet_hdfs_tvf_case branch October 17, 2024 06:44
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actionsgithub-actions[bot]github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@suxiaogang223@doris-robot@yiguolei