Uh oh!
There was an error while loading. Please reload this page.
feat: Update Parquet row filtering to handle type coercion - #10716
Conversation
alamb
left a comment
There was a problem hiding this comment.
I think this is looking definitely on the right path
Uh oh!
There was an error while loading. Please reload this page.
alamb
left a comment
There was a problem hiding this comment.
Thank you @jeffreyssmith2nd
I went over the code and test carefully and I think it looks good. I took the liberty of merging up to resolve a conflict.
I also verified that the test covers the code change by removing the code change locally and running the test. When I did so the test fails like this:
assertion failed: filtered.is_ok()
thread 'datasource::physical_plan::parquet::row_filter::test::test_filter_type_coercion' panicked at datafusion/core/src/datasource/physical_plan/parquet/row_filter.rs:549:9:
assertion failed: filtered.is_ok()
stack backtrace:
0: rust_begin_unwind
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
1: core::panicking::panic_fmt
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
2: core::panicking::panic
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:145:5
3: datafusion::datasource::physical_plan::parquet::row_filter::test::test_filter_type_coercion
at ./src/datasource/physical_plan/parquet/row_filter.rs:549:9
4: datafusion::datasource::physical_plan::parquet::row_filter::test::test_filter_type_coercion::{{closure}}
at ./src/datasource/physical_plan/parquet/row_filter.rs:486:35
5: core::ops::function::FnOnce::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
6: core::ops::function::FnOnce::call_once
at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass `--lib`
error: 1 target failed:
`--lib`
I have some suggestions for the test, but otherwise I think this PR is ready to go.
@Ted-Jiang and @thinkharderdev I wonder if you have a moment to review this too as I believe you actively use the filter pushdown into parquet
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
alamb
left a comment
There was a problem hiding this comment.
Looks good to me -- thank you @jeffreyssmith2nd
I plan to leave this open for another day in case there are additional comments
alamb
commented
Jun 5, 2024
Thank you @jeffreyssmith2nd and @thinkharderdev |
Which issue does this PR close?
Closes#7925.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?