Skip to content

refactor: support bitmap for u8/16 and i8/16 in approx_distinct - #8462

Closed
Weijun-H wants to merge 3 commits into
apache:mainfrom
Weijun-H:update-approx-distinc
Closed

refactor: support bitmap for u8/16 and i8/16 in approx_distinct#8462
Weijun-H wants to merge 3 commits into
apache:mainfrom
Weijun-H:update-approx-distinc

Conversation

@Weijun-H

@Weijun-HWeijun-H commented Dec 7, 2023•
edited
Loading

Copy link
Copy Markdown
Member

Which issue does this PR close?

Follow #1841
Closes#1109

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@github-actionsgithub-actionsBot added the physical-expr Changes to the physical-expr crates label Dec 7, 2023
@Weijun-HWeijun-H changed the title refactor: support bitmap for u8/16 and i8/16 in approx_distinctrefactor: support bitmap for u8/16 and i8/16 in approx_distinctDec 7, 2023
Weijun-H
Weijun-H commented Dec 7, 2023
View reviewed changes
Comment threaddatafusion/physical-expr/src/aggregate/approx_distinct.rs
}

fn size(&self) -> usize {
self.bitmap.serialized_size()

@Weijun-HWeijun-HDec 7, 2023

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure is a proper way to measure roaring bitmap

@korowa

korowa commented Dec 27, 2023

Copy link
Copy Markdown
Contributor

Thank you @Weijun-H!

This PR, though, looks like the implementation of CountDistinct accumulator, and it doesn't seem that any approximation is performed in the code -- won't it be better to use these changes as default CountDistinct implementation for specified data types?

@alamb

alamb commented Jan 27, 2024

Copy link
Copy Markdown
Contributor

I wonder what we should do with this PR now we have

@korowa

korowa commented Jan 28, 2024

Copy link
Copy Markdown
Contributor

I wonder what we should do with this PR now we have

While I'm still not sure if this PR fits ApproxDistinct functionality, I suppose it might be a viable replacement for HashSets in regular CountDistinct -- so it at least worth checking / benchmarking within #1823 (paying special attention to memory consumption of bitmap-based accumulator)

@alamb

alamb commented Jan 29, 2024

Copy link
Copy Markdown
Contributor

Thanks @Weijun-H and @korowa -- I'll mark this PR as a draft now and if someone finds time to do the benchmarks we can reopen it with that in consideration

@alamb
alamb marked this pull request as draft January 29, 2024 11:57
@github-actions

github-actionsBot commented Apr 17, 2024

Copy link
Copy Markdown

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

@github-actionsgithub-actionsBot added the Stale PR has not had any activity for some time label Apr 17, 2024
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

physical-exprChanges to the physical-expr cratesStalePR has not had any activity for some time

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

approx_distinct should be leveraging bitmap for counting u8/16 and i8/16

3 participants

@Weijun-H@korowa@alamb