Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
80d840e
GH-17211: refresh history for scalar_hash kernel
drin Jan 12, 2024
9260849
GH-17211: refresh history for updates to key_hash
drin Jan 12, 2024
e7252d0
GH-17211: refresh history for new compute fn infra
drin Jan 12, 2024
5c3db19
GH-17211: refresh history of scalar hash benchmark
drin Jan 12, 2024
49718f5
added KeyColumnArray alias
drin Oct 7, 2024
37b19dd
task: revert style and other minor changes
drin Oct 11, 2024
8f444b0
task: reducing support to some nested types
drin Oct 11, 2024
5cba979
minor: added type check for ListArray
drin Oct 11, 2024
312a3a5
Update the implementation to use column oriented hash-combine.
kszucs Dec 10, 2024
949ffed
add hypothesis test case and fix list hashing
kszucs Dec 11, 2024
d5a1ba8
ensure to keep newly allocated arrays alive until hash gets computed
kszucs Dec 11, 2024
79765d9
minor cleanup
kszucs Dec 11, 2024
d710fca
wrong include
kszucs Dec 11, 2024
3b56994
add hash32 function
kszucs Dec 13, 2024
618ec4a
add hash32 function and address review comments
kszucs Dec 13, 2024
50e33ac
update testing suite
kszucs Dec 13, 2024
1f907c2
generated struct tests
kszucs Dec 13, 2024
bf0b7a0
prepare adding hash32 tests
kszucs Dec 13, 2024
866e4b0
add tests for hash32, dictionary encoded type, extension type
kszucs Dec 16, 2024
d9f0088
ensure that NULL and 0 hashes to different numbers
kszucs Dec 16, 2024
691ccd3
fix the benchmarks
kszucs Dec 16, 2024
2eb2ba4
update docs to highlight 0 return for NULL values
kszucs Dec 16, 2024
2d5f93f
use resize instead of reserve
kszucs Dec 17, 2024
5f9614a
fix R compute function filtering test
kszucs Dec 17, 2024
9271efa
address review comments
kszucs Dec 18, 2024
fe2e206
update boolean test case
kszucs Dec 18, 2024
661d82f
update exec checks
kszucs Dec 18, 2024
40a710b
use Hasher::kHashBatchTempStackUsage
kszucs Dec 18, 2024
3bb84a5
ensure that slices properly work
kszucs Dec 18, 2024
aeacc80
add tests for fixed_size_binary()
kszucs Dec 18, 2024
0e14971
add rudimentary quality tests
kszucs Dec 18, 2024
c145283
test temporal types too
kszucs Dec 19, 2024
28a9dca
test temporal and decimal types
kszucs Dec 19, 2024
cae89ef
add type matcher to explicitly raise NotImplemented for unsupported t…
kszucs Dec 19, 2024
1293b40
update error message
kszucs Dec 19, 2024
24a7dc6
update docs
kszucs Dec 19, 2024
303d2ac
set a tolerance for the randomly generated numbers
kszucs Dec 19, 2024
ab78188
make msvc happy
kszucs Dec 20, 2024
b0d0b0a
Rename DCHECK macros to ARROW_DCHECK
kszucs May 13, 2025
46dc7b8
Missing blank lines
kszucs May 14, 2025
f07c665
chore: move key_hash benchmarks to key_hash_benchmark.cc under compute
kszucs Jun 20, 2025
b457ec1
Apply suggestions from code review
kszucs Jun 25, 2025
6528c69
use type singleton
kszucs Jun 25, 2025
59a06ec
run clang format on scalar_hash_benchmark.cc
kszucs Jun 25, 2025
53a62f9
run cmake format
kszucs Jun 25, 2025
4ba82af
Address review comments and fix missing-declarations build failure
kszucs Jul 23, 2026
39eecbe
Fix unreachable slicing check and remove unused hashing_rng in benchm…
kszucs Jul 23, 2026
6489383
Fix nested-type null and hash-corruption bugs in hash32/hash64
kszucs Jul 23, 2026
40711e8
Remove dead list-like branches from ToColumnArray
kszucs Jul 23, 2026
25f0a0b
Fix independently-null nested fields hashing as non-zero garbage
kszucs Jul 23, 2026
12fdde4
Size key_hash_benchmark's TempVectorStack from kHashBatchTempStackUsage
kszucs Jul 23, 2026
d3ff917
Hoist KeyColumnArray construction out of key_hash_benchmark's timed loop
kszucs Jul 23, 2026
c4e6a15
Fix GCC -Werror=range-loop-construct in scalar_hash_test.cc
kszucs Jul 23, 2026
121d38d
Guard against a null values buffer in ToColumnArray's binary path
kszucs Jul 23, 2026
80cb5ac
Add R test asserting hash32/hash64 are discoverable via list_compute_…
kszucs Jul 23, 2026
39497f0
Avoid hashing unreferenced child data for sliced list/map arrays, fix…
kszucs Jul 24, 2026
c2b4915
Add hypothesis tests asserting hash32/hash64 respect Array.slice()
kszucs Jul 24, 2026
f1d03bb
Revert kAddend change to the shared HashIntImp; scope the null/0 fix …
kszucs Jul 24, 2026
b7a9791
Unwrap extension types in HashableMatcher so unsupported storage type…
kszucs Jul 24, 2026
ac63803
Add test coverage for zero/null collision across byte widths and FIXE…
kszucs Jul 24, 2026
3c95c9d
Fix HashChild reusing a nested field's unshifted validity buffer
kszucs Jul 24, 2026
f9a65da
Avoid the bitmap copy in HashChild by omitting the validity buffer en…
kszucs Jul 24, 2026
8d39471
Fix double blank lines around the new Hash Functions doc section
kszucs Jul 24, 2026
85968a8
Avoid hashing unreferenced struct nested-field data, avoid gtest macr…
kszucs Jul 24, 2026
c1f30bb
Hoist the per-column vector out of KeyHashIntegers benchmark loops
kszucs Jul 24, 2026
eb9744d
Fix Hash32/Hash64 API docstrings
kszucs Jul 24, 2026
8a736f3
Fix values-child offset cancellation bug in list/map hashing
kszucs Jul 24, 2026
0f19f16
Fix valid struct rows coincidentally colliding with the null hash sen…
kszucs Jul 24, 2026
5beac3f
Loosen RandomPrimitive's hash-quality tolerance for cross-platform RN…
kszucs Jul 24, 2026
c14e1ef
Fix independently-null fields of multi-field structs not hashing to 0
kszucs Jul 27, 2026
1b550bd
Dedupe Hash64Struct{Small,Medium,Large}Strings benchmarks
kszucs Jul 27, 2026
7a0e1b7
Make hash32/hash64 null-aware instead of reserving 0 for nulls
kszucs Jul 27, 2026
f309ce6
Fix zero-width fixed_size_binary hashing, and simplify
kszucs Jul 28, 2026
e35894e
Fix NullType struct fields not invalidating their row's hash
kszucs Aug 24, 2026
59ca351
Add DoNotOptimize/ClobberMemory to key hash benchmarks
kszucs Sep 2, 2026
5eedc72
Bulk-copy hash output validity, and fix doc nits
kszucs Sep 2, 2026
1c09c5d
Reject unsupported nested child types during hash dispatch
kszucs Sep 2, 2026
abd254e
Drop unused builder_primitive.h include from hashing_benchmark
kszucs Sep 2, 2026
bd7b7ee
Route zero-width fixed_size_binary struct fields through HashArray
kszucs Sep 2, 2026
33cdf3d
Add the new scalar-hash sources to the Meson build
kszucs Sep 2, 2026
08ec6f2
Distinguish a null child element from one hashing to zero
kszucs Sep 3, 2026
6c018be
Hash map keys even when the item is null
kszucs Sep 3, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cpp/src/arrow/CMakeLists.txt
Original file line numberDiff line numberDiff line change
Expand Up@@ -869,6 +869,7 @@ if(ARROW_COMPUTE)
compute/kernels/scalar_arithmetic.cc
compute/kernels/scalar_boolean.cc
compute/kernels/scalar_compare.cc
compute/kernels/scalar_hash.cc
compute/kernels/scalar_if_else.cc
Comment thread
kszucs marked this conversation as resolved.
compute/kernels/scalar_nested.cc
compute/kernels/scalar_random.cc
Expand Down
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/CMakeLists.txt
Original file line numberDiff line numberDiff line change
Expand Up@@ -185,6 +185,7 @@ add_arrow_compute_test(row_test
arrow_compute_testing)

add_arrow_compute_benchmark(function_benchmark)
add_arrow_compute_benchmark(key_hash_benchmark)

add_subdirectory(kernels)

Expand Down
10 changes: 10 additions & 0 deletions cpp/src/arrow/compute/api_scalar.cc
Original file line numberDiff line numberDiff line change
Expand Up@@ -957,6 +957,16 @@ Result<Datum> MapLookup(const Datum& arg, MapLookupOptions options, ExecContext*
return CallFunction("map_lookup", {arg}, &options, ctx);
}

// ----------------------------------------------------------------------
// Hash functions
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash32", {input_array}, ctx);
}

Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx) {
return CallFunction("hash64", {input_array}, ctx);
}

// ----------------------------------------------------------------------

} // namespace compute
Expand Down
41 changes: 41 additions & 0 deletions cpp/src/arrow/compute/api_scalar.h
Original file line numberDiff line numberDiff line change
Expand Up@@ -1807,5 +1807,46 @@ ARROW_EXPORT Result<Datum> NanosecondsBetween(const Datum& left, const Datum& ri
/// \note API not yet finalized
ARROW_EXPORT Result<Datum> MapLookup(const Datum& map, MapLookupOptions options,
ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Array in, Array out;
/// ChunkedArray in, ChunkedArray out), but with element type UInt32. For a nested
/// input type (struct, list, map, etc.), each row's child values are combined into a
/// single hash for that row, recursively. A null input row produces a null output row;
/// within a struct, a null field makes that whole row null, while within a list or map a
/// null element does not (only the row's own validity matters there). Hash values are not
/// guaranteed to be stable across different versions of the library, and this function
/// does not currently take options, though these may be added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash32(const Datum& input_array, ExecContext* ctx = NULLPTR);

/// \brief Construct a hash value for each row of the input.
///
/// The result has the same length and shape as the input (Array in, Array out;
/// ChunkedArray in, ChunkedArray out), but with element type UInt64. For a nested
/// input type (struct, list, map, etc.), each row's child values are combined into a
/// single hash for that row, recursively. A null input row produces a null output row;
/// within a struct, a null field makes that whole row null, while within a list or map a
/// null element does not (only the row's own validity matters there). Hash values are not
/// guaranteed to be stable across different versions of the library, and this function
/// does not currently take options, though these may be added in the future.
///
/// \param[in] input_array input data to hash
/// \param[in] ctx function execution context, optional
/// \return elementwise hash values
///
/// \since 26.0.0
/// \note API not yet finalized
ARROW_EXPORT
Result<Datum> Hash64(const Datum& input_array, ExecContext* ctx = NULLPTR);

} // namespace compute
} // namespace arrow
1 change: 1 addition & 0 deletions cpp/src/arrow/compute/initialize.cc
Original file line numberDiff line numberDiff line change
Expand Up@@ -31,6 +31,7 @@ Status RegisterComputeKernels() {
internal::RegisterScalarArithmetic(registry);
internal::RegisterScalarBoolean(registry);
internal::RegisterScalarComparison(registry);
internal::RegisterScalarHash(registry);
internal::RegisterScalarIfElse(registry);
internal::RegisterScalarNested(registry);
internal::RegisterScalarRandom(registry); // Nullary
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/kernels/CMakeLists.txt
Original file line numberDiff line numberDiff line change
Expand Up@@ -77,6 +77,7 @@ add_arrow_compute_test(scalar_math_test

add_arrow_compute_test(scalar_utility_test
SOURCES
scalar_hash_test.cc
scalar_random_test.cc
scalar_set_lookup_test.cc
scalar_validity_test.cc
Expand All@@ -89,6 +90,7 @@ add_arrow_benchmark(scalar_cast_benchmark PREFIX "arrow-compute")
add_arrow_compute_benchmark(scalar_arithmetic_benchmark)
add_arrow_compute_benchmark(scalar_boolean_benchmark)
add_arrow_compute_benchmark(scalar_compare_benchmark)
add_arrow_compute_benchmark(scalar_hash_benchmark)
add_arrow_compute_benchmark(scalar_if_else_benchmark)
add_arrow_compute_benchmark(scalar_list_benchmark)
add_arrow_compute_benchmark(scalar_random_benchmark)
Expand Down
2 changes: 2 additions & 0 deletions cpp/src/arrow/compute/kernels/meson.build
Original file line numberDiff line numberDiff line change
Expand Up@@ -55,6 +55,7 @@ scalar_kernel_tests = {
},
'arrow-compute-scalar-utility-test': {
'sources': [
'scalar_hash_test.cc',
'scalar_random_test.cc',
'scalar_set_lookup_test.cc',
'scalar_validity_test.cc',
Expand All@@ -76,6 +77,7 @@ scalar_kernel_benchmarks = [
'scalar_boolean_benchmark',
'scalar_cast_benchmark',
'scalar_compare_benchmark',
'scalar_hash_benchmark',
'scalar_if_else_benchmark',
'scalar_list_benchmark',
'scalar_random_benchmark',
Expand Down
Loading
Loading