Skip to content

Add config option skip_physical_aggregate_schema_check - #13176

Merged
alamb merged 3 commits into
apache:mainfrom
alamb:alamb/config_option
Oct 30, 2024
Merged

Add config option skip_physical_aggregate_schema_check #13176
alamb merged 3 commits into
apache:mainfrom
alamb:alamb/config_option

Conversation

@alamb

@alambalamb commented Oct 29, 2024•
edited
Loading

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes#13065

Rationale for this change

Some plans that used to run in DataFusion 41.0.0 started erroring in DataFusion 42.0.0 due to a new check that was added.

The delta-rs 42.0.0 upgrade has it this delta-io/delta-rs#2886 (comment), and we hit the same thing in InfluxDB IOx as well.

To help unblock upgrades, I would like to permit users to optionally disable this check.

This config flag is meant to be temporary while we fix all the underlying bugs.

Background: a new check for exact schema equality added in #11989 (released in DataFusion 42.0.0). This has found quite a few bugs where the schema doesn't quite match due to nullability or metadata mismatch -- see the list on #12733. There is at least one more bug @wiedld and I are tracking down.

After upgrading to DataFusion 42 some of these plans now error (due to bugs in DataFusion, see list on #12733)

What changes are included in this PR?

Add a skip_physical_aggregate_schema_check option to disable this check so downstream users can workaround any issues they hit

Are these changes tested?

The doc is tested by CI. Since this a workaround for bugs

Are there any user-facing changes?

A new config option

@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation core Core DataFusion crate common Related to common crate labels Oct 29, 2024
@alambalamb mentioned this pull request Oct 29, 2024
Closed
@alambalamb changed the title Add option to skip physical aggregate checkAdd config option skip_physical_aggregate_schema_check Oct 29, 2024
@alamb

alamb commented Oct 29, 2024

Copy link
Copy Markdown
ContributorAuthor

I plan to backport this option to the 42 branch as well defaulted to true to unblock the delta-rs upgrade

comphead
comphead approved these changes Oct 29, 2024
View reviewed changes

@compheadcomphead left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @alamb

@alamb

alamb commented Oct 29, 2024

Copy link
Copy Markdown
ContributorAuthor

See additional context here: #13065 (comment)

@github-actionsgithub-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Oct 29, 2024
comphead
comphead reviewed Oct 29, 2024
View reviewed changes
Comment threaddatafusion/common/src/config.rs
/// When set to true, skips verifying that the schema produced by
/// planning the input of `LogicalPlan::Aggregate` exactly matches the
/// schema of the input plan.
///

@compheadcompheadOct 29, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if blank lines can break the formatting and tests eventually failed

@alambalambOct 29, 2024

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason the tests failed before is that I forgot to update the information_schema.slt test

@alamb
alamb merged commit 63e8e6a into apache:mainOct 30, 2024
@alamb

alamb commented Oct 30, 2024

Copy link
Copy Markdown
ContributorAuthor

Thanks again @comphead and @jayzhan211

@alamb
alamb deleted the alamb/config_option branch October 30, 2024 16:42
alamb added a commit to alamb/datafusion that referenced this pull request Oct 30, 2024
@alamb
Add config optionskip_physical_aggregate_schema_check(apache#13176)
54fdc5d
* Add option to skip physical aggregate check
* tweak wording
* update test
@alambalamb mentioned this pull request Oct 30, 2024
Merged
@alamb

alamb commented Oct 30, 2024

Copy link
Copy Markdown
ContributorAuthor

PR to backport #13189

alamb added a commit that referenced this pull request Oct 30, 2024
@alamb
Backport config optionskip_physical_aggregate_schema_check#13176…
e7877e2
…to 42 (#13189)
* Add config option `skip_physical_aggregate_schema_check ` (#13176)
* Add option to skip physical aggregate check
* tweak wording
* update test
* Change default value
@alambalamb mentioned this pull request Oct 30, 2024
Closed
7 tasks
@findepi

findepi commented Oct 31, 2024

Copy link
Copy Markdown
Member

Looking at #13190
It seems that schema check is too strict (It should allow fulfilling nulllable contract with non-null data, but not the other way around).

If we relax the check around this, how many problems would remain to be addressed?

@alamb

alamb commented Nov 1, 2024

Copy link
Copy Markdown
ContributorAuthor

Looking at #13190 It seems that schema check is too strict (It should allow fulfilling nulllable contract with non-null data, but not the other way around).

If we relax the check around this, how many problems would remain to be addressed?

It seems to me the check is finding real things (where the schema of the ExecutionPlan doesn't match the schema of the LogicalPlan from when it was created)

However, it also seems to me like maybe the discrepancy isn't a huge deal (so maybe it shouldn't be an error 🤔 )

@findepi

findepi commented Nov 1, 2024

Copy link
Copy Markdown
Member

Real things like type mismatch, or real things like stricter non-null guarantees (which may be considered an optimization, not an error)?

@alamb

alamb commented Nov 1, 2024

Copy link
Copy Markdown
ContributorAuthor

Real things like type mismatch, or real things like stricter non-null guarantees (which may be considered an optimization, not an error)?

What I know of was related to non-null guarantees as well as user defined metadata (on the Schema and the Field)

Dandandan pushed a commit to coralogix/arrow-datafusion that referenced this pull request Feb 11, 2025
@alamb@Dandandan
Backport config optionskip_physical_aggregate_schema_checkapache#…
e7620cf
…13176 to 42 (apache#13189)
* Add config option `skip_physical_aggregate_schema_check ` (apache#13176)
* Add option to skip physical aggregate check
* tweak wording
* update test
* Change default value
@killme2008killme2008 mentioned this pull request Mar 6, 2025
Closed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@compheadcompheadcomphead approved these changes
@jayzhan211jayzhan211jayzhan211 approved these changes

Assignees

No one assigned

Labels

commonRelated to common cratecoreCore DataFusion cratedocumentationImprovements or additions to documentationsqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Option to disable Physical input schema should be the same as the one converted from logical schema error

4 participants

@alamb@findepi@comphead@jayzhan211