Skip to content

Fix grouping set subset satisfaction - #19853

Merged
gabotechs merged 4 commits into
apache:mainfrom
freakyzoidberg:fix-grouping-set-subset-satisfaction
Jan 16, 2026
Merged

Fix grouping set subset satisfaction#19853
gabotechs merged 4 commits into
apache:mainfrom
freakyzoidberg:fix-grouping-set-subset-satisfaction

Conversation

@freakyzoidberg

@freakyzoidbergfreakyzoidberg commented Jan 16, 2026

Copy link
Copy Markdown
Member

Summary

  • Fixes incorrect results from ROLLUP/CUBE/GROUPING SETS queries when using multiple partitions
  • The subset satisfaction optimization was incorrectly allowing hash partitioning on fewer columns to satisfy requirements that include __grouping_id
  • This caused partial aggregates from different partitions to be finalized independently, producing duplicate grand totals

Closes#19849

@github-actionsgithub-actionsBot added optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt) labels Jan 16, 2026
@gabotechs

gabotechs commented Jan 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

cc @gene-bordegaray do this changes look good to you?

@gabotechs

gabotechs commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

I can confirm that this change does fix the issue, I'll let @gene-bordegaray chime in as the one assigned to the original issue.

Unless there are any relevant comments, I'll merge this in as soon as CI passes.

@gene-bordegaray

gene-bordegaray commented Jan 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Yes this looks correct, thanks @freakyzoidberg . Let me unassign myself and comment take on issue

only thing may be wanting to add plans in the sqllogictests 😄

cc: @gabotechs

gabotechs
gabotechs reviewed Jan 16, 2026
View reviewed changes
Comment threaddatafusion/sqllogictest/test_files/grouping_set_repartition.slt
Comment on lines +145 to +147
04)------AggregateExec: mode=FinalPartitioned, gby=[channel@0 as channel, brand@1 as brand, __grouping_id@2 as __grouping_id], aggr=[sum(sub.total)]
05)--------RepartitionExec: partitioning=Hash([channel@0, brand@1, __grouping_id@2], 4), input_partitions=4
06)----------AggregateExec: mode=Partial, gby=[(NULL as channel, NULL as brand), (channel@0 as channel, NULL as brand), (channel@0 as channel, brand@1 as brand)], aggr=[sum(sub.total)]

@gabotechsgabotechsJan 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 I imagine before this PR the RepartitionExec would not be there right?

@freakyzoidbergfreakyzoidbergJan 16, 2026

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

gabotechs
gabotechs approved these changes Jan 16, 2026
View reviewed changes

@gabotechsgabotechs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good! thanks @freakyzoidberg and @gene-bordegaray

@gabotechs
gabotechs added this pull request to the merge queueJan 16, 2026
Merged via the queue into apache:main with commit 1ab7e41Jan 16, 2026
32 checks passed
gabotechs pushed a commit to gabotechs/datafusion that referenced this pull request Jan 16, 2026
@freakyzoidberg@gabotechs
Fix grouping set subset satisfaction (apache#19853)
6caf352
## Summary
- Fixes incorrect results from ROLLUP/CUBE/GROUPING SETS queries when
using multiple partitions
- The subset satisfaction optimization was incorrectly allowing hash
partitioning on fewer columns to satisfy requirements that include
`__grouping_id`
- This caused partial aggregates from different partitions to be
finalized independently, producing duplicate grand totals
Closesapache#19849
This was referenced Jan 16, 2026
Merged
Closed
@alamb

alamb commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

Love it -- thank you

@alamb

alamb commented Jan 16, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Thank you @freakyzoidberg and @gabotechs

Do you we know what changes in 52 introduced this problem?

@gabotechs

gabotechs commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

You have the full conclusion here #19849

gabotechs added a commit that referenced this pull request Jan 16, 2026
@gabotechs@freakyzoidberg
[branch 52] Fix grouping set subset satisfaction (#19855)
eb00fe2
Brings #19853 into `branch-52`
Co-authored-by: Pierre Lacave <pierre.lacave@datadoghq.com>
@alamb

alamb commented Jan 16, 2026

Copy link
Copy Markdown
Contributor

Thanks @gabotechs -- I'll ask some follow up questions there

NGA-TRAN
NGA-TRAN reviewed Jan 17, 2026
View reviewed changes

@NGA-TRANNGA-TRAN left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@gene-bordegaraygene-bordegaray mentioned this pull request Jul 1, 2026
Merged
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gabotechsgabotechsgabotechs approved these changes
+1 more reviewer
@NGA-TRANNGA-TRANNGA-TRAN left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

optimizerOptimizer rulessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Different results in TPC-DS q14 depending on the number of partitions

5 participants

@freakyzoidberg@gabotechs@gene-bordegaray@alamb@NGA-TRAN