ARROW-16807: [C++][R] count distinct incorrectly merges state by drin · Pull Request #13583 · apache/arrow

drin · 2022-07-12T16:38:22Z

This addresses a bug where the count_distinct function simply added counts when merging state. The correct logic would be to return the number of distinct elements after both states have been merged.

State for count_distinct is backed by a MemoTable, which is then backed by a HashTable. To properly merge state, this PR adds 2 functions to each MemoTable: MaybeInsert and MergeTable. The MaybeInsert function handles simplified logic for inserting an element into the MemoTable. The MergeTable function handles iteration over elements in the MemoTable to be merged.

This PR also adds an R test and a C++ test. The R test mirrors what was provided in ARROW-16807. The C++ test, AllChunkedArrayTypesWithNulls, mirrors another C++ test, AllArrayTypesWithNulls, but uses chunked arrays for test data.

The bug described in ARROW-16807 is that when merging states for count distinct, the non_null counts are simply added (no overload that I saw). The fix was to change this into a proper merge of the data in the MemoTable. There are 3 derived classes from MemoTable. This adds a `MaybeInsert` and `MergeTable` function to each to support the merge logic.

This adds a unittest to run count_distinct on chunked arrays, which were incorrectly handled before.

This is a test to reproduce the issue seen in ARROW-16807, which calls `count_distinct` (via `n_distinct`) on a dataset with many chunks. If `summarize` is called before `collect`, then count_distinct merges distinct counts across chunks incorrectly. Calling collect before summarize does not expose the bug.

Added an extra assert to be sure that collect and summarize produce the same results when commuted.

github-actions · 2022-07-12T17:42:19Z

https://issues.apache.org/jira/browse/ARROW-16807

drin · 2022-07-12T22:39:30Z

I just realized that this introduces (or maybe just exposes) a bug when calling this function on scalar inputs. If the input is a scalar, non_nulls is incremented without changing state. To address this "correctly," the code path for scalar inputs should also update the state by using GetOrInsert as is done for the code path for vector inputs.

I am working on figuring this out, given that the compiler has type conversion issues when attempting to just call GetOrInsert with a scalar reference.

The path for count_distinct with scalar inputs didn't update state, and instead only added a single count. To fix this, we use MaybeInsert and UnboxScalar to insert the new value. The non_null count can then be set the same way for both vector and scalar inputs

The UnboxScalar implementation that has `enable_if_has_string_view` is also true for Decimal128 and Decimal256 types. Usually the implementation templated for these types would be called, but The BinaryMemoTable only has a type for BinaryTypes. This means that for count_distinct with Decimal columns, the UnboxScalar implementation that gets called is the one for string_view type

drin · 2022-07-13T23:47:58Z

@wesm @westonpace previous pair of commits makes this work for scalar inputs. I tried putting some of the conditional logic ("if type is decimal") in CountDistinctImpl but it ended up being difficult to satisfy the compiler.

the current change to UnboxScalar tries to essentially use is_decimal, but then I realized that there isn't an easy way to call the appriopriate view function unless I know which decimal type I have. Additionally, the UnboxScalar implementations for Decimal types gets the value, which I think would be difficult to downcast/change back to string_view.

all in all, this was the only approach I could get to work and it seemed semi reasonable. Feedback would be much appreciated though.

wesm · 2022-07-14T13:37:55Z

@drin I pushed a simpler solution for the UnboxScalar change and removed MaybeInsert. We should probably open a follow up Jira to improve error handling in hash table merges but I don't think it's necessary to do that in this PR

drin · 2022-07-14T16:08:14Z

I made a JIRA to track error handling improvements for hash table merge: ARROW-17074

I specifically made the priority minor, but feel free to change it (I don't know the protocol for priority assignment)

cyb70289

Looks great. Only some minor comments.
Thanks @drin !

drin · 2022-07-15T16:50:22Z

thanks for the reviews @cyb70289 and Wes!

westonpace

This looks good from my perspective. Thanks for adding the tests. I'll let others confirm before merging.

cyb70289 · 2022-07-16T01:36:09Z

Oops, why suddently so many CI errors?

R lint error and test failure
https://github.com/apache/arrow/runs/7362285260?check_suite_focus=true#step:12:51
https://github.com/apache/arrow/runs/7362285354?check_suite_focus=true#step:11:21344
https://github.com/apache/arrow/runs/7361802264?check_suite_focus=true#step:5:14778
Related to this PR? cc @nealrichardson
protobuf symbol not found in flight and opentelemetry
https://github.com/apache/arrow/runs/7361802181?check_suite_focus=true#step:10:188
https://github.com/apache/arrow/runs/7361802975?check_suite_focus=true#step:9:1306
cc @lidavidm

wesm · 2022-07-16T21:32:32Z

@cyb70289 these CI failures don't seem related — R has some lint failures and there are these otel protobuf linkage issues. I'll go ahead to merge this

nealrichardson · 2022-07-16T21:50:29Z

R issues are likely a combination of transient upstream issues and stuff that has been fixed in master. If things persist I'll follow up.

ursabot · 2022-07-17T00:10:58Z

Benchmark runs are scheduled for baseline = afc6840 and contender = af4db77. af4db77 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Failed ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Failed ⬇️1.18% ⬆️0.03%] test-mac-arm
[Failed ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️1.36% ⬆️0.14%] ursa-thinkcentre-m75q
Buildkite builds:
[Failed] af4db773 ec2-t3-xlarge-us-east-2
[Failed] af4db773 test-mac-arm
[Failed] af4db773 ursa-i9-9960x
[Finished] af4db773 ursa-thinkcentre-m75q
[Finished] afc6840c ec2-t3-xlarge-us-east-2
[Failed] afc6840c test-mac-arm
[Failed] afc6840c ursa-i9-9960x
[Finished] afc6840c ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

…Hub issue numbers (#34260) Rewrite the Jira issue numbers to the GitHub issue numbers, so that the GitHub issue numbers are automatically linked to the issues by pkgdown's auto-linking feature. Issue numbers have been rewritten based on the following correspondence. Also, the pkgdown settings have been changed and updated to link to GitHub. I generated the Changelog page using the `pkgdown::build_news()` function and verified that the links work correctly. --- ARROW-6338 #5198 ARROW-6364 #5201 ARROW-6323 #5169 ARROW-6278 #5141 ARROW-6360 #5329 ARROW-6533 #5450 ARROW-6348 #5223 ARROW-6337 #5399 ARROW-10850 #9128 ARROW-10624 #9092 ARROW-10386 #8549 ARROW-6994 #23308 ARROW-12774 #10320 ARROW-12670 #10287 ARROW-16828 #13484 ARROW-14989 #13482 ARROW-16977 #13514 ARROW-13404 #10999 ARROW-16887 #13601 ARROW-15906 #13206 ARROW-15280 #13171 ARROW-16144 #13183 ARROW-16511 #13105 ARROW-16085 #13088 ARROW-16715 #13555 ARROW-16268 #13550 ARROW-16700 #13518 ARROW-16807 #13583 ARROW-16871 #13517 ARROW-16415 #13190 ARROW-14821 #12154 ARROW-16439 #13174 ARROW-16394 #13118 ARROW-16516 #13163 ARROW-16395 #13627 ARROW-14848 #12589 ARROW-16407 #13196 ARROW-16653 #13506 ARROW-14575 #13160 ARROW-15271 #13170 ARROW-16703 #13650 ARROW-16444 #13397 ARROW-15016 #13541 ARROW-16776 #13563 ARROW-15622 #13090 ARROW-18131 #14484 ARROW-18305 #14581 ARROW-18285 #14615 * Closes: #33631 Authored-by: SHIMA Tatsuya <ts1s1andn@gmail.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

drin added 5 commits July 12, 2022 08:47

[ARROW-16807]: new tests on chunked arrays
797ad4f
This adds a unittest to run count_distinct on chunked arrays, which were incorrectly handled before.

[ARROW-16807]: minor addition to test
8452dfd
Added an extra assert to be sure that collect and summarize produce the same results when commuted.

[ARROW-16807]: fixing style
1debb4e

github-actionsBot added Component: C++ Component: R labels Jul 12, 2022

cyb70289 changed the title ~~ARROW-16807: count distinct incorrectly merges state~~ARROW-16807: [C++][R] count distinct incorrectly merges stateJul 13, 2022

cyb70289 reviewed Jul 13, 2022
View reviewed changes

Comment threadcpp/src/arrow/util/hashing.h Outdated

[ARROW-16807]: minor update to R test
63470e2

wesm reviewed Jul 13, 2022
View reviewed changes

Comment threadr/tests/testthat/test-dplyr-summarize.R Outdated
Comment threadcpp/src/arrow/util/hashing.h Outdated

drin added 2 commits July 13, 2022 16:38

cyb70289 reviewed Jul 14, 2022
View reviewed changes

Comment threadcpp/src/arrow/util/hashing.h Outdated

cyb70289 reviewed Jul 14, 2022
View reviewed changes

Comment threadcpp/src/arrow/util/hashing.h Outdated

Simpler static UnboxScalar solution, remove MaybeInsert
5d04cb0

[ARROW-16807]: retabbed R test
91d2cab

drin marked this pull request as ready for review July 14, 2022 16:15

drin requested a review from cyb70289 July 14, 2022 16:16

cyb70289 reviewed Jul 15, 2022
View reviewed changes

Comment threadcpp/src/arrow/util/hashing.h Outdated
Comment threadcpp/src/arrow/util/hashing.h Outdated
Comment threadcpp/src/arrow/util/hashing.h Outdated
Comment threadcpp/src/arrow/util/hashing.h Outdated
Comment threadcpp/src/arrow/util/hashing.h Outdated

[ARROW-16807]: added const and JIRA reference
7c4d701

drin requested a review from cyb70289 July 15, 2022 16:53

westonpace approved these changes Jul 15, 2022
View reviewed changes

cyb70289 approved these changes Jul 16, 2022
View reviewed changes

wesm merged commit af4db77 into apache:masterJul 16, 2022

drin deleted the ARROW-16807-count-distinct branch July 18, 2022 19:36

asfimport mentioned this pull request Oct 3, 2022
[C++] count_distinct aggregates incorrectly across row groups #32138
Closed

eitsupi mentioned this pull request Feb 19, 2023
GH-33631: [R] Rewrite Jira ticket numbers in pkgdown documents to GitHub issue numbers #34260
Merged

asfimport mentioned this pull request Jul 14, 2022
[C++] Improve error handling for hash table merges #32381
Closed

Uh oh!

Conversation

drin commented Jul 12, 2022

Uh oh!

github-actionsBot commented Jul 12, 2022

Uh oh!

drin commented Jul 12, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drin commented Jul 13, 2022

Uh oh!

Uh oh!

Uh oh!

wesm commented Jul 14, 2022

Uh oh!

drin commented Jul 14, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cyb70289 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

drin commented Jul 15, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

cyb70289 commented Jul 16, 2022

Uh oh!

wesm commented Jul 16, 2022

Uh oh!

nealrichardson commented Jul 16, 2022

Uh oh!

ursabot commented Jul 17, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

drin commented Jul 14, 2022•
edited
Loading

drin commented Jul 15, 2022•
edited
Loading