Uh oh!
There was an error while loading. Please reload this page.
Enable reading StringViewArray by default from Parquet - #12092
Conversation
StringViewArray by default from Parquet
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
This comment was marked as outdated.
This comment was marked as outdated.
bc5d7f7 to
27f7d1eCompare
This comment was marked as outdated.
This comment was marked as outdated.
27f7d1e to
ce5470dCompareUh oh!
There was an error while loading. Please reload this page.
This comment was marked as outdated.
This comment was marked as outdated.
2275216 to
e8f7384Compare
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
alamb
commented
Oct 5, 2024
clickbench_extendedLooking good here |
clickbench_partitionedSlowing down also due to #12509 / #12788 |
alamb
commented
Oct 7, 2024
Current status: #11682 (comment) |
6500361 to
5b67afeCompareMy plan is to pull the changes from the following PRs into this PR and rerun the overall perf test
Hopefully that will show the performance we need. Then we just need to get the various PRs actually merged and we can do this 😅 |
5b67afe to
62a738eCompare62a738e to
dda0a9aComparedda0a9a to
4a57773Comparealamb
commented
Oct 14, 2024
Here are the current performance results: 🚀 Basically 10% faster across all queries and no slow downs 👏 @Rachelint@jayzhan211@XiangpengHao and many many others. Also for partitioned: |
alamb
commented
Oct 14, 2024
Next steps are to get the various PRs merged, get this one ready, and then write all about it. |
820b9ab to
6112175Compare6112175 to
cbdc592Comparealamb
commented
Oct 24, 2024
I made a new PR as this one has lots of now irrelevant historical context New PR here: #13101 |
Draft as it builds on:
53.1.0/ fix clippy #12724binary_as_stringparquet option, upgrade to arrow/parquet53.2.0#12816@goldmedalGroupColumnsupport forStringView/ByteView(faster grouping performance) #12809 from @Rachelinthits_partitionedWhich issue does this PR close?
Closes#11682
Rationale for this change
Reading data as
StringViewArrayis significantly faster thanStringArray. We have been testing this behind a feature flag but it is now stable enough to enable by default.See blog post #11603:
Benchmark Results
What changes are included in this PR?
schema_force_view_typesto trueAre these changes tested?
Yes, by CI tests
Are there any user-facing changes?
If you see an error related to StringView use, you can disable this feature using the schema_force_string_view option
Context
@XiangpengHao debugged these tests previously using #11862