Uh oh!
There was an error while loading. Please reload this page.
refactor: Split Parquet BloomFilter CPU and IO into separate states - #21285
Conversation
alamb
commented
Mar 31, 2026
run benchmarks |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (ac46396) to 5ff80e4 (merge-base) diff using: tpch File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (ac46396) to 5ff80e4 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (ac46396) to 5ff80e4 (merge-base) diff using: tpcds File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Dandandan
commented
Mar 31, 2026
run benchmarks |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (bf0d992) to 6412c3a (merge-base) diff using: tpch File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (bf0d992) to 6412c3a (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing alamb/split_bloom (bf0d992) to 6412c3a (merge-base) diff using: tpcds File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
adriangbot
commented
Mar 31, 2026
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)DetailsResource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
adriangb
left a comment
There was a problem hiding this comment.
I like the direction these refactors are going in @alamb , this is the right way to organize opener.rs (I know at some point we spoke about splitting it up into more modules, this is either a good first step towards that or a better solution).
| .into_iter() | ||
| .filter_map(|column_name| { | ||
| let parquet_schema = builder.parquet_schema(); | ||
| let (column_idx, _) = parquet_column( |
There was a problem hiding this comment.
It'd be nice if we could get this to also support struct columns / struct filters cc @friendlymatthew
alamb
commented
Apr 2, 2026
Thanks again @Dandandan and @adriangb I am now working on the next steps |
Uh oh!
There was an error while loading. Please reload this page.
…pache#21285) ## Which issue does this PR close? - part of apache#20529 - Broken out of apache#20820 ## Rationale for this change We are try to be explicit about CPU and IO in the Parquet reader, however, the code for applying bloom filters still did both CPU and IO in the same future ## What changes are included in this PR? 1. Split the states for IO and CPU work when applying bloom filters ## Are these changes tested? Functionally by existing tests I also ran performance tests and they didn't show any substantial performance change ## Are there any user-facing changes? No, this is internal code reorganization
…pache#21285) ## Which issue does this PR close? - part of apache#20529 - Broken out of apache#20820 ## Rationale for this change We are try to be explicit about CPU and IO in the Parquet reader, however, the code for applying bloom filters still did both CPU and IO in the same future ## What changes are included in this PR? 1. Split the states for IO and CPU work when applying bloom filters ## Are these changes tested? Functionally by existing tests I also ran performance tests and they didn't show any substantial performance change ## Are there any user-facing changes? No, this is internal code reorganization
Which issue does this PR close?
Rationale for this change
We are try to be explicit about CPU and IO in the Parquet reader, however, the code for applying bloom filters still did both CPU and IO in the same future
What changes are included in this PR?
Are these changes tested?
Functionally by existing tests
I also ran performance tests and they didn't show any substantial performance change
Are there any user-facing changes?
No, this is internal code reorganization