Uh oh!
There was an error while loading. Please reload this page.
GH-49888: [C++][Compute] Fix count for run-end encoded arrays with nulls - #49908
Conversation
fenfeng9
commented
May 2, 2026
The added tests use a simplified version of the reproducer from the issue. Reproduceimportpyarrowaspaimportpyarrow.computeaspcarray=pa.array([1, None])
encoded=pc.run_end_encode(array)
print(f"plain only_null: ", pc.count(array, mode="only_null"))
print(f"run_end_encode only_null: ", pc.count(encoded, mode="only_null"))Result |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
tadeja
left a comment
There was a problem hiding this comment.
Thank you for the fix, @fenfeng9 ! 🌞 A few additional test suggestions from my end as inline comments.
The MinGW64 CI failures look unrelated. There were some overlapping failures on main.
fenfeng9
commented
May 5, 2026
Thanks for the suggestion. I updated the C++ and Python tests. |
fenfeng9
commented
May 5, 2026
The original behavior is: Reproduceimportpyarrowaspaimportpyarrow.computeaspcarray=pa.array([1, 1, None, None, None, 2, 2, 2, None, 3])
encoded=pc.run_end_encode(array)
# Logical slice: [None, None, 2, 2, 2, None].slice_plain=array.slice(3, 6)
slice_encoded=encoded.slice(3, 6)
print("pyarrow:", pa.__version__)
print()
print(f"{'case':<12}{'only_valid':>10}{'only_null':>10}{'all':>6}")
forname, valuein [
("plain", array),
("ree", encoded),
("slice plain", slice_plain),
("slice ree", slice_encoded),
]:
print(
f"{name:<12} "f"{pc.count(value, mode='only_valid').as_py():>10} "f"{pc.count(value, mode='only_null').as_py():>10} "f"{pc.count(value, mode='all').as_py():>6}"
)Result |
fenfeng9
commented
May 5, 2026
The test failures look unrelated. |
pitrou
commented
May 6, 2026
And thanks @tadeja for the useful reviewing! |
After merging your PR, Conbench analyzed the 0 benchmarking runs that have been run so far on merge-commit 1b3f313. None of the specified runs were found on the Conbench server. The full Conbench report has more details. |
After merging your PR, Conbench analyzed the 2 benchmarking runs that have been run so far on merge-commit 1b3f313. There was 1 benchmark result with an error:
There were no benchmark performance regressions. 🎉 The full Conbench report has more details. It also includes information about 8 possible false positives for unstable benchmarks that are known to sometimes produce them. |
…ith nulls (apache#49908) ### Rationale for this change The `count` kernel used `GetNullCount()`, which reports the physical null count. For run-end encoded arrays, this ignored nulls in the encoded values child. ### What changes are included in this PR? Use `ComputeLogicalNullCount()` in the `count` kernel so run-end encoded arrays are counted correctly. Add C++ and Python tests for this case. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: apache#49888 Authored-by: fenfeng9 <fenfeng9@qq.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
Rationale for this change
The
countkernel usedGetNullCount(), which reports the physical null count. For run-end encoded arrays, this ignored nulls in the encoded values child.What changes are included in this PR?
Use
ComputeLogicalNullCount()in thecountkernel so run-end encoded arrays are counted correctly. Add C++ and Python tests for this case.Are these changes tested?
Yes.
Are there any user-facing changes?
No.
countkernel miscounts when run-end encoded array contains null #49888