Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) by alamb · Pull Request #11667 · apache/datafusion

alamb · 2024-07-26T10:44:16Z

~~Draft until arrow 52.2.0 is released to crates.io (expected Sat July 27)~~

Which issue does this PR close?

Note 1: this code has been previously reviewed
Note 2: The feature isn't on by default (yet) until Enable reading StringView by default from Parquet (schema_force_string_view) by default #11682

Rationale for this change

We have been integrating a set of StringView changes on the string-view2 branch as they rely on un-released code in arrow-rs . Once those changes are released and DataFusion uses them we can bring this code directly to main.

What changes are included in this PR?

Add a config to force using string view in benchmark (Add a config to force using string view in benchmark #11514) - @XiangpengHao
Add String view helper functions (Add String view helper functions #11517) - @XiangpengHao
Add ArrowBytesViewMap and ArrowBytesViewSet (Add ArrowBytesViewMap and ArrowBytesViewSet #11515) - @XiangpengHao
Enable GroupValueBytesView for aggregation with StringView types (#… - @XiangpengHao
Initial support for regex_replace on StringViewArray (Initial support for regex_replace on StringViewArray #11556) - @XiangpengHao
Add support for Utf8View for date/temporal codepaths (Add support for Utf8View for date/temporal codepaths #11518) - @a10y
GC StringViewArray in CoalesceBatchesStream (GC StringViewArray in CoalesceBatchesStream #11587) - @XiangpengHao
[Bug] fix bug in return type inference of utf8_to_int_type ([Bug] fix bug in return type inference of utf8_to_int_type #11662)
Increase ByteViewMap block size to 2MB (Increase ByteViewMap block size to 2MB #11674) - @XiangpengHao
Change --string-view to only apply to parquet formats (Change --string-view to only apply to parquet formats #11663) - @XiangpengHao
Implement native support StringView for character length (Implement native support StringView for character length #11676) - @XiangpengHao

Are these changes tested?

CI

Are there any user-facing changes?

When StringView is enabled, benchmarks run significantly faster

* add a knob to force string view in benchmark * fix sql logic test * update doc * fix ci * fix ci only test * Update benchmarks/src/util/options.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * Update datafusion/common/src/config.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> * update tests --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* add functions * add tests for hash util

* Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * add more tests * make doc happy * update new implementation * fix bug * avoid unused dep * update dep * update * fix cargo check * update doc * pick up the comments change again --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…11519) * add functions * Update `string-view` branch to arrow-rs main (#10966) * Pin to arrow main * Fix clippy with latest arrow * Uncomment test that needs new arrow-rs to work * Update datafusion-cli Cargo.lock * Update Cargo.lock * tapelo * merge * update cast * consistent dep * fix ci * avoid unused dep * update dep * update * fix cargo check * better group value view aggregation * update --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* initial support for string view regex * update tests

* Add StringView support for date_part and make_date funcs * run cargo update in datafusion-cli * cargo fmt --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* gc string view when appropriate * make clippy happy * address comments * make doc happy * update style * Add comments and tests for gc_string_view_batch * better herustic * update test * Update datafusion/physical-plan/src/coalesce_batches.rs Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org> --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2024-07-26T14:13:35Z

CI appears to be failing due to #11671

* fix bug in return type inference * update doc * add tests --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

* better default block size * fix related test

* use inferenced schema, don't load schema again * move config to parquet-only * update * update * better format * format * update

alamb · 2024-07-27T11:16:00Z

Update here is that we are on track to release arrow 52.2.0 to crates.io tomorrow Saturday July 28. (thank you to @waynexia @viirya @wjones127 for verifying / voting 🙏 ).

Then we'll need a PR to update datafusion to arrow 52.2.0 (which should be straightforward)

Then I will change this PR to ready to review and we should be able to merge it into DataFusion main

🤞

* native support for character length * Update datafusion/functions/src/unicode/character_length.rs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2024-07-29T12:40:31Z

This PR is now ready for review (mostly I am hoping another committer will approve/merge it as I can't approve my own PR)

Dandandan

🚀

alamb · 2024-07-29T20:52:34Z

Thanks @Dandandan -- so exciting!

I plan to work with @XiangpengHao and figure out what we need to do to get this feature on by default

alamb · 2024-07-29T20:52:49Z

🚀

alamband others added 9 commits July 16, 2024 15:55

Pin to pre-release version of arrow 52.2.0
987e33b

Update for deprecated method
2c808fb

Add String view helper functions (#11517)
8e0ca1a
* add functions * add tests for hash util

Initial support for regex_replace onStringViewArray(#11556)
34d42bc
* initial support for string view regex * update tests

Add support for Utf8View for date/temporal codepaths (#11518)
bb780b3
* Add StringView support for date_part and make_date funcs * run cargo update in datafusion-cli * cargo fmt --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

github-actionsBot added documentation Improvements or additions to documentation logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jul 26, 2024

Merge remote-tracking branch 'apache/main' into string-view2
2b2b8ab

alamb mentioned this pull request Jul 26, 2024
circular dependency check CI check is failing with compile error #11671
Closed

alamband others added 4 commits July 26, 2024 10:29

Merge remote-tracking branch 'apache/main' into string-view2
ea11a9d

[Bug] fix bug in return type inference ofutf8_to_int_type(#11662)
f13bb82
* fix bug in return type inference * update doc * add tests --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Merge remote-tracking branch 'apache/main' into string-view2
fb79638

Fix clippy
281fbed

XiangpengHao mentioned this pull request Jul 26, 2024
Implement native StringView support for CharacterLength #11677
Closed
4 tasks

Increase ByteViewMap block size to 2MB (#11674)
5690712
* better default block size * fix related test

This was referenced Jul 27, 2024
use StringViewArray when reading String columns from Parquet #10921
Closed
Enable reading StringView by default from Parquet (schema_force_string_view) by default #11682
Closed

Change--string-viewto only apply to parquet formats (#11663)
322c3d2
* use inferenced schema, don't load schema again * move config to parquet-only * update * update * better format * format * update

alamb changed the title ~~Merge string-view2 branch to main~~ Implement StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )Jul 27, 2024

alamb changed the title ~~Implement StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )~~StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )Jul 27, 2024

Implement native support StringView for character length (#11676)
ab8005d
* native support for character length * Update datafusion/functions/src/unicode/character_length.rs --------- Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

This was referenced Jul 27, 2024
Skipping partial aggregation when it is not helping for high cardinality aggregates #11627
Merged
Update to arrow/parquet 52.2.0#11691
Merged

alamb changed the title ~~StringView reading from parquet, up to 2x faster for some click bench queries (Merge string-view2 branch to main )~~Merge string-view branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)Jul 29, 2024

alamb added 3 commits July 28, 2024 20:13

Merge remote-tracking branch 'apache/main' into string-view2
561aee8

Remove uneeded patches
2e9c8a0

cargo fmt
f1f22fa

alamb marked this pull request as ready for review July 29, 2024 12:37

alamb mentioned this pull request Jul 29, 2024
Extract CoalesceBatchesStream to a struct #11610
Merged

alamb mentioned this pull request Jul 29, 2024
[Epic] Implement support for StringView in DataFusion #10918
Closed
16 tasks

alamb changed the title ~~Merge string-view branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)~~Merge string-view2 branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)Jul 29, 2024

alamb mentioned this pull request Jul 29, 2024
DataFusion weekly project plan (Andrew Lamb) - July 29, 2024 #11710
Closed
8 tasks

Dandandan approved these changes Jul 29, 2024
View reviewed changes

alamb merged commit a591301 into mainJul 29, 2024

alamb deleted the string-view2 branch July 29, 2024 20:52

mbutrovich mentioned this pull request Aug 14, 2025
feat: support Utf8View for more args of regexp_replace#17195
Merged

Navigation Menu

Uh oh!

Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default) - #11667

Merge `string-view2` branch: reading from parquet up to 2x faster for some ClickBench queries (not on by default)#11667
alamb merged 20 commits into
mainfrom
string-view2

alamb commented Jul 26, 2024•
edited
Loading

Uh oh!

alamb commented Jul 26, 2024

Uh oh!

alamb commented Jul 27, 2024

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

Dandandan left a comment

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

alamb commented Jul 26, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Jul 26, 2024

Uh oh!

alamb commented Jul 27, 2024

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

Dandandan left a comment

Choose a reason for hiding this comment

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

alamb commented Jul 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

alamb commented Jul 26, 2024•
edited
Loading