Skip to content

Fix incorrect results in BitAnd GroupsAccumulator - #6957

Merged
alamb merged 1 commit into
apache:mainfrom
alamb:alamb/fix_gby_bit_and
Jul 13, 2023
Merged

Fix incorrect results in BitAnd GroupsAccumulator#6957
alamb merged 1 commit into
apache:mainfrom
alamb:alamb/fix_gby_bit_and

Conversation

@alamb

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Resolves#6955, a bug introduced in #6904

Rationale for this change

The answer is incorrect because the starting value is always 0 and thus 0 & anything else will also be zero which is not right

What changes are included in this PR?

  1. Allow the starting value of the accumulator t be specified
  2. set the default value of the accumulator correctly for bit_and
  3. Add new tests (subset of what is in Minor: Improve aggregate test coverage more #6952)

Are these changes tested?

Yes

Are there any user-facing changes?

@github-actionsgithub-actionsBot added physical-expr Changes to the physical-expr crates core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Jul 13, 2023
@alambalamb changed the title Alamb/fix gby bit andFix incorrect results in BitAnd GroupsAccumulatorJul 13, 2023
@alamb
alambforce-pushed the alamb/fix_gby_bit_and branch from a27881f to 6459945CompareJuly 13, 2023 16:04
GROUP BY tag
ORDER BY tag
----
1 8 11 13 14 11 12 6 11 A

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before this PR, this test fails like

External error: query result mismatch:
[SQL] SELECT
bit_and(c1),
bit_and(c2),
bit_and(c3),
bit_or(c1),
bit_or(c2),
bit_or(c3),
bit_xor(c1),
bit_xor(c2),
bit_xor(c3),
tag
FROM bit_aggregate_functions
GROUP BY tag
ORDER BY tag
[Diff] (-expected|+actual)
- 1 8 11 13 14 11 12 6 11 A
- 33 11 NULL 33 11 NULL 33 11 NULL B
+ 0 0 0 13 14 11 12 6 11 A
+ 0 0 NULL 33 11 NULL 33 11 NULL B
at tests/sqllogictests/test_files/aggregate.slt:1546

DataType::Int8 => {
instantiate_primitive_accumulator!(self, Int8Type, |x, y| x
.bitand_assign(y))
instantiate_accumulator!(self, -1, Int8Type, |x, y| x.bitand_assign(y))

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the fix is passing in MAX / -1 here to get a bitpattern of all 1 initially

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good. It's better to leave a comment like above in the code.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment added in #6964

let values = values[0].as_primitive::<T>();

// update values
self.values.resize(total_num_groups, T::default_value());

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously this always started at zero, now it starts at the specified value.

I think I can now reduce the code in min/max accumulator as well

@alamb
alamb marked this pull request as ready for review July 13, 2023 16:06

@izveigorizveigor left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@alamb

Copy link
Copy Markdown
ContributorAuthor

🤔 seems clippy failure is unrelated. I will fix that

@alamb

Copy link
Copy Markdown
ContributorAuthor

Clippy fixes in #6959

@alamb
alamb merged commit c07c26c into apache:mainJul 13, 2023
@alamb

Copy link
Copy Markdown
ContributorAuthor

Thank you for the quick review @viirya and @izveigor

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratephysical-exprChanges to the physical-expr cratessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Regression: bit_and aggregate function is incorrect

3 participants

@alamb@viirya@izveigor