Uh oh!
There was an error while loading. Please reload this page.
Data: Handle null values properly in IN predicate filtering - #16697
Data: Handle null values properly in IN predicate filtering#16697hantangwangd wants to merge 1 commit into
IN predicate filtering#16697Conversation
| assertThat(filterResults) | ||
| .as("Should produce correct number of records") | ||
| .hasSameSizeAs(expected); | ||
| assertThat(filterResults).as("Random record set should match").isEqualTo(expected); |
There was a problem hiding this comment.
The description looks copy-paste leftover from the testRandomData method. This test doesn't include random test data.
There was a problem hiding this comment.
@ebyhr nice catch! Thanks for the review. I've fixed it. PTAL when you get a chance.
e801599 to
52b4d43CompareThis pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
ebyhr
commented
Jul 14, 2026
|
hantangwangd
commented
Jul 14, 2026
Sure, got it! It's great to see this issue resolved! |
When scanning table records via
IcebergGenerics.read(table)and specifying filter conditions withwhere(filter), if the filter contains anINpredicate and the corresponding target column contains null values, the query may fail directly with the following error:The root cause is: when
FilterIterator.advance()is called, it invokes theshouldKeep(item)closure method ofCloseableIterableto determine whether to keep the read item, during which thein(...)method ofEvalVisitoris executed for evaluation. In the original logic, it directly checks that the corresponding target column value is not null, and throws immediately if it is null.However, in many scenarios (such as the one constructed in the newly added test case), when a data file contains both possible valid values and null values in the target column, the records that contain null values will be read and passed to this method for evaluation, at which point an error will be thrown directly.
This PR fixes the issue by properly handling null values.