Skip to content

parquet: Add tests for pruning on Int8/Int16/Int64 columns - #9778

Merged
alamb merged 3 commits into
apache:mainfrom
progval:int-bloom-tests
Mar 31, 2024
Merged

parquet: Add tests for pruning on Int8/Int16/Int64 columns#9778
alamb merged 3 commits into
apache:mainfrom
progval:int-bloom-tests

Conversation

@progval

@progvalprogval commented Mar 24, 2024•
edited by alamb
Loading

Copy link
Copy Markdown
Contributor

[Do not merge this PR, it highlights a bug in Int8 and Int16 columns, through correct_bloom_filters: false. See #9779 for a discussion]

Which issue does this PR close?

Closes#9777.

Rationale for this change

What changes are included in this PR?

Generalizes the Int32 tests, using a macro.

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the core Core DataFusion crate label Mar 24, 2024
@progvalprogval mentioned this pull request Mar 24, 2024
Open
@progvalprogval changed the title parquet: Add tests for Bloom filters on Int8/Int16/Int64 columnsparquet: Add tests for pruning on Int8/Int16/Int64 columnsMar 24, 2024
alamb
alamb reviewed Mar 24, 2024
View reviewed changes

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @progval

[Do not merge this PR, it highlights a bug in Int8 and Int16 columns, through correct_bloom_filters: false. See https://github.com//issues/9779 for a discussion]

What would you think about merging this PR (but keeping #9779 open)?

Then the fix for #9779 could just update the tests to set correct_bloom_filters?

alamb
alamb reviewed Mar 24, 2024
View reviewed changes
Comment threaddatafusion/core/tests/parquet/row_group_pruning.rs
@progval

progval commented Mar 24, 2024

Copy link
Copy Markdown
ContributorAuthor

What would you think about merging this PR

Makes sense, I guess. I was hoping you would find how to solve #9779, because I can't :)

(but keeping #9779 open)?

Well, it should be removed when fixed, there is no point to keeping it.

@alamb

alamb commented Mar 24, 2024

Copy link
Copy Markdown
Contributor

What would you think about merging this PR

Makes sense, I guess. I was hoping you would find how to solve #9779, because I can't :)

(but keeping #9779 open)?

Well, it should be removed when fixed, there is no point to keeping it.

Agreed.

For others following along, the issue is upstream in parquet-rs: #9779 (comment)

I think what we should do with this PR is to update the comments explaining that the tests now demonstrate there is a bug in DataFusion and link to the upstream issue.

Once the upstream issue is fixed, we can then update the tests / close the false nagatives bug #9779

@alamb

alamb commented Mar 28, 2024

Copy link
Copy Markdown
Contributor

As a follow up here, I plan to add the comments explaining what is going on here (and that there are tests that show incorrect results, that are tracked by a ticket) and then merge it in.

@alambalamb mentioned this pull request Mar 30, 2024
Merged
alamb
alamb approved these changes Mar 30, 2024
View reviewed changes

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @progval -- I somehow missed that you had updated this PR. It looks really good to me.

Thanks again. Very much appreciated

@alamb

alamb commented Mar 30, 2024

Copy link
Copy Markdown
Contributor

I merged up from main to make sure we have a clean CI run and then I think we can merge this one in

@alamb
alamb merged commit 2cb6f73 into apache:mainMar 31, 2024
@alamb

alamb commented Mar 31, 2024

Copy link
Copy Markdown
Contributor

Thanks again @progval

@progval
progval deleted the int-bloom-tests branch March 31, 2024 09:12
@progvalprogval mentioned this pull request Mar 31, 2024
Closed
Lordworms pushed a commit to Lordworms/arrow-datafusion that referenced this pull request Apr 1, 2024
@progval@alamb@Lordworms
parquet: Add tests for pruning on Int8/Int16/Int64 columns (apache#9778)
c6c50b3
* parquet: Add tests for Bloom filters on Int8/Int16/Int64 columns
* Document int_tests macro
---------
Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alambalambalamb approved these changes

Assignees

No one assigned

Labels

coreCore DataFusion crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

core/tests/parquet/row_group_pruning.rs is missing tests for Int8/Int16/Int64 columns

2 participants

@progval@alamb