Uh oh!
There was an error while loading. Please reload this page.
ARROW-1572: [C++] Implement "value counts" kernels for tabulating value frequencies - #3579
Closed
emkornfield wants to merge 20 commits into
Closed
ARROW-1572: [C++] Implement "value counts" kernels for tabulating value frequencies#3579emkornfield wants to merge 20 commits into
emkornfield wants to merge 20 commits into
Conversation
andrioni
commented
Feb 7, 2019
Contributor
Thanks for picking this up from where I left off, I'll close my original PR. |
emkornfield
commented
Feb 7, 2019
ContributorAuthor
@andrioni Thanks for the contribution |
emkornfieldforce-pushed
the
count_kernel
branch
from
February 7, 2019 14:34
62d23fd to
9a5c9caComparefsaintjacques
approved these changes
Feb 8, 2019
fsaintjacques
left a comment
Contributor
There was a problem hiding this comment.
Some notes:
- May I propose to rename this to
Distribution? - We should also count Nulls.
- The current output format is 2 arrays. I think we should formalize this and return a single array of
List<Struct<Option<value>, count>>explicitly which should allow returning the count for the null value.
This can be done in another ticket/PR.
Contributor
There was a problem hiding this comment.
For completeness, you should set the datum type if we ever refactor.
ContributorAuthor
There was a problem hiding this comment.
Could you clarify? I might be missing it but I don't see an API to set the type explicitly (instead it looks like it gets derived when value that is assigned in the underlying variant)
xhochy
commented
Feb 8, 2019
Member
Rebased |
emkornfield
commented
Feb 12, 2019
ContributorAuthor
@fsaintjacques thanks for the review. My thoughts:
|
emkornfieldforce-pushed
the
count_kernel
branch
from
February 12, 2019 06:58
30dc7d5 to
8779eddCompareemkornfield
commented
Feb 12, 2019
ContributorAuthor
There was a problem hiding this comment.
this could potentially return a StructArray directly.
emkornfieldforce-pushed
the
count_kernel
branch
from
February 13, 2019 06:58
8779edd to
8c77e19Compareemkornfieldforce-pushed
the
count_kernel
branch
from
February 20, 2019 06:43
193d110 to
a6b9364Compareemkornfieldforce-pushed
the
count_kernel
branch
from
February 27, 2019 08:34
a24af0b to
123c6c0Compareemkornfield
commented
Feb 27, 2019
ContributorAuthor
@fsaintjacques I think I fixed the windows build (will check in the morning). I changed the output to be a Struct as suggested. I will file follow-up JIRAs for the rest. |
Contributor
There was a problem hiding this comment.
I'd put a DCHECK_GE(start, 0)` just to make sure it's positive.
ContributorAuthor
There was a problem hiding this comment.
Also put a ASSERT_LE in and changed to std::copy which i think fixes another bug.
emkornfieldforce-pushed
the
count_kernel
branch
2 times, most recently
from
March 1, 2019 04:11
0176819 to
ae08e40Compareemkornfield
commented
Mar 1, 2019
ContributorAuthor
Clean build 👍 |
fsaintjacques
commented
Mar 1, 2019
Contributor
I want to give a last review, but I don't think anything is blocking. |
fsaintjacques
approved these changes
Mar 1, 2019
Contributor
There was a problem hiding this comment.
Make Values and Count a public constexpr such that user can programatically retrieve the field by name at compile time, maybe expose the index that way too.
ContributorAuthor
There was a problem hiding this comment.
Good suggestion. Done.
fsaintjacques
commented
Mar 2, 2019
Contributor
Ready to merge! |
fsaintjacques
commented
Mar 2, 2019
Contributor
I think you need to rebase for the failing appveyor failure, but I'd say this is safe to merge. LGTM +1. |
emkornfieldforce-pushed
the
count_kernel
branch
3 times, most recently
from
March 4, 2019 03:59
7825282 to
2845381Compareemkornfield
commented
Mar 11, 2019
ContributorAuthor
One thing that I'll do which I'm not sure will help too much is separate out the memcpy fix into it's own PR so we can isolate performance problems to the kernel |
emkornfield
commented
Mar 11, 2019
ContributorAuthor
I think there might also be a way forward with template magic to eliminate this altogether. I'll get an update out soon |
ContributorAuthor
With this change it looks like they are fairly close on my box (rerunning causes changes in different directions in different metrics)
|
wesm
commented
Mar 14, 2019
Member
Looking at this again and then merging, thanks for your patience |
Member
I ran on a mobile Xeon 2.8 ghz with the cores pinned at 2.8ghz / no turbo boost and things looking OK here before after merging. thanks @emkornfield! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Picked up from: #1970
I mostly reused the unit tests as is and modified the rest based on feedback on that PR and adapting to new code. Some other changes I mae:
If this approach looks OK for now. I'm also going to open up a JIRA to try refactor the unittest for Hash kernels (and maybe the headers) since I think there might be a clearer more granular way of laying these out.
other things to discuss: