Add xxhash algorithms in SQL and expression api by Spaarsh · Pull Request #14367 · apache/datafusion

Spaarsh · 2025-01-30T04:03:02Z

Which issue does this PR close?

ClosesImplement xxhash algorithms as part of the expression API #14044.

Rationale for this change

Lack of xxhash (a quick, non-cryptographic hashing technique) functions.

What changes are included in this PR?

Are these changes tested?

Yes.

Are there any user-facing changes?

The users shall now be able to use xxhash32 and xxhash64 functions.

…f input data

Omega359 · 2025-02-03T21:59:45Z

Can you fix the clippy issue and generate the documentation (./dev/update_function_docs.sh) ?

Omega359 · 2025-02-03T22:07:26Z

+
+fn process_array<T: Hasher>(array: &dyn Array, mut hasher: T, hash_type: HashType) -> Result<Vec<String>> {
+ let mut hash_results: Vec<String> = Vec::with_capacity(array.len());
+ for i in 0..array.len() {


Instead of downcasting the array for every single element in it see if you can do it once up front.

Omega359 · 2025-02-03T22:07:46Z

+ Self {
+ signature: Signature::uniform(
+ 1,
+ vec![Utf8View, Utf8, LargeUtf8, Binary, LargeBinary],


The types you say you accept here do not match the types handled in process_array or to_string_from_scalar. You should likely handle all these and the int ones you are handling in those.

Sure I'll fix this. I had actually made several changes that had not been pushed. Due do a glitch in my device they got wiped off 😅
I removed the Int types from being accepted as input since I was thinking that allowing any other form of input other than a string wouldn't be ideal and I am anyway casting them to the Utf8 type. And all the hash functions expect their input to be of that form only. Even the crate I am using expects the same. Allowing for multiple types doesn't serve much of a purpose now that I think about it. I would love your thoughts on this!
And should I convert this issue to draft? Since I am still adding the optional seed argument.

If it's still a wip, sure, mark as draft with a WIP in the description helps a lot. Some hashes do allow binary input (See md5 for example, so that is normally reasonable.

Omega359 · 2025-02-03T22:10:07Z

Beyond the comments I've added it would be very welcome to include .slt tests for these hash functions. Search for md5 or sha256 for examples of current tests.

Spaarsh · 2025-02-04T19:44:58Z

I will add support for Null values and write the tests as well!

alamb · 2025-02-05T11:02:30Z

This PR isn't passing CI checks so I am marking it as draft to clear the review queue.

Spaarsh · 2025-02-05T17:52:07Z

Beyond the comments I've added it would be very welcome to include .slt tests for these hash functions. Search for md5 or sha256 for examples of current tests.

@Omega359 I found out that the md5 tests are found in functions.slt and expr.slt as well. While there are many other functions that have their own .slt file as well as tests in expr.slt

Since I am planning to add wywash hash functions soon, should I make a separate hash.slt and include a few tests in the expr.slt as well?

Thanks!

Omega359 · 2025-02-05T18:00:19Z

Beyond the comments I've added it would be very welcome to include .slt tests for these hash functions. Search for md5 or sha256 for examples of current tests.
@Omega359 I found out that the md5 tests are found in functions.slt and expr.slt as well. While there are many other functions that have their own .slt file as well as tests in expr.slt
Since I am planning to add wywash hash functions soon, should I make a separate hash.slt and include a few tests in the expr.slt as well?

The slt tests are known to need a bit of a cleanup in spots. A hash.slt I think is a good idea and we can have a followup PR to move the other hash functions in there for their tests.

Spaarsh · 2025-02-05T19:16:41Z

@Omega359 I had not pushed the updated cargo.lock file, hence the tests were failing. I think now this PR is ready for review. The hash.stl file has been added as well. There is a small conflict in the Cargo.lock that needs resolving.

Omega359 · 2025-02-06T18:38:05Z

Thanks for continuing to work through this feature! ❤️

I left a number of feedback items. I think the largest concern I have is the downcast_ref code not handling StringViewArray where you likely want to switch to using StringArrayType.

Spaarsh · 2025-02-07T16:16:05Z

I have added the test for Utf8View array for xxhash32 and xxhash64 as well.

Spaarsh · 2025-02-07T18:01:50Z

The tests are failing due to code in the main itself at the time of branch creation. Is this PR ready for review?

…n-API' into 14044/enhancement/add-xxhash-algorithms-in-expression-API

Omega359 · 2025-02-07T23:03:19Z

Thanks for applying the updates! I'll do one final review over the weekend then if it's good (I suspect it is!) then I'll ask a Committer to merge it in.

Omega359 · 2025-02-09T17:13:44Z

While I can see opportunities for improvements in the code I think they are relatively minor and this PR is suitable for inclusion in DF. Thanks!

@alamb

alamb

Thanks @Spaarsh and @Omega359

I don't understand how widespread the xxhash function is. If this is a feature to support delta-rs I think it would be suitable for inclusion.

If it is a function for one particular user's usecase I think we should be much more careful of taking on a new function (and the dependency and maintenance)

I started a conversation here

delta-io/delta-rs#3105 (comment)

Spaarsh · 2025-02-10T14:56:44Z

@alamb could you also get a clarification over the inclusion of wyhash functions as well? It was also requested under the same issue here.

alamb · 2025-02-10T15:15:19Z

@alamb could you also get a clarification over the inclusion of wyhash functions as well? It was also requested under the same issue here.

In my opinion there are an infinite number of potentially useful functions we could add to the DataFusion crate

However, given there are extension APIs the only benefits of adding functions to the core crates vs installing them elsewere are:

pre-integration (no extra work is required downstream)
long term maintenance (there is an active community in DataFusion)

As valuable as those properties are for users of datafusion enabled products / projects, they come at a cost to the core maintainers and our review / maintenance bandwidth is our most limited resource.

alamb · 2025-02-10T15:15:40Z

So unless there are many users (and ideally maintainers) of new functions, I am hesitant to add new ones

Omega359 · 2025-02-25T21:44:59Z

Note that this seems to also be in the spark function PR - https://github.com/apache/datafusion/pull/14392/files#diff-2bbff2d3a0ce9cec9ed9d6ec6e38ff910875af704b60855f43b47b46c96c5d44

alamb · 2025-02-26T14:24:46Z

That is all the more reason to perhaps defer to the spark function PR / focus on that

alamb · 2025-02-26T14:24:50Z

That is all the more reason to perhaps defer to the spark function PR / focus on that

github-actions · 2025-05-01T02:12:54Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

alamb · 2025-05-05T19:58:05Z

We recently added the datafusion-spark crate for spark compatible functions. @Spaarsh , would you be willing to port this function over to that crate and update it to be spark compatible?

[EPIC] Complete datafusion-spark Spark Compatible Functions #15914

Spaarsh · 2025-05-08T15:13:41Z

@alamb sure. I have my uni exams going on right now. That, along with some other commitments, might take me a while for porting these functions. If that is alright, then I'll be happy to complete this ASAP. If there is some urgency, then I'll be glad to help anyone working on this.

alamb · 2025-05-08T17:35:26Z

Thanks @Spaarsh -- good luck with your exams

github-actions · 2025-07-09T02:13:13Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

adriangb · 2025-09-18T04:15:47Z

We might want this for #17632 (comment)

Spaarsh added 4 commits January 27, 2025 02:16

Added hash module with xxhash32 and xxhash64 functions.
a682d1e

Added support for individual values and for ingesting various types o…
25ebf13
…f input data

Refactored code
5fcc67b

license fix
6353e6b

github-actionsBot added the functions Changes to functions implementation label Jan 30, 2025

Omega359 reviewed Feb 3, 2025
View reviewed changes

Spaarsh marked this pull request as draft February 4, 2025 09:55

Spaarsh added 2 commits February 5, 2025 00:11

Optional seed added
741fa18

Fixed test failings
4d24759

github-actionsBot added the documentation Improvements or additions to documentation label Feb 4, 2025

Fixed clippy failing for unnecessary cast
5f85511

Spaarsh marked this pull request as ready for review February 5, 2025 04:04

alamb marked this pull request as draft February 5, 2025 11:02

Spaarsh added 3 commits February 5, 2025 19:06

Fixed CI test fails
0d15f9c

Added support for Null inputs and corrected output for empty inputs
a729942

Fixed failing fmt checks
60e4db7

Added hash test file (xxhash32 and xxhash64)
f8e871b

github-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Feb 5, 2025

Minor fix
564af36

Omega359 reviewed Feb 6, 2025
View reviewed changes

Comment threaddatafusion/functions/src/hash/xxhash.rs

Omega359 reviewed Feb 6, 2025
View reviewed changes

Comment threaddatafusion/functions/src/hash/xxhash.rs Outdated

Omega359 reviewed Feb 6, 2025
View reviewed changes

Comment threaddatafusion/functions/src/hash/xxhash.rs Outdated

Implemented feedback items
ed2324f

Fixed fmt and .md test fails
18b1937

Merge branch '14044/enhancement/adding-xxhash-algorithms-in-expressio…
25fd5c4
…n-API' into 14044/enhancement/add-xxhash-algorithms-in-expression-API

Spaarsh marked this pull request as ready for review February 7, 2025 18:40

alamb changed the title ~~14044/enhancement/add xxhash algorithms in expression api~~Add xxhash algorithms in expression apiFeb 10, 2025

alamb changed the title ~~Add xxhash algorithms in expression api~~Add xxhash algorithms in SQL and expression apiFeb 10, 2025

alamb reviewed Feb 10, 2025
View reviewed changes

alamb marked this pull request as draft February 26, 2025 14:24

github-actionsBot added the Stale PR has not had any activity for some time label May 1, 2025

alamb mentioned this pull request May 5, 2025
[EPIC] Complete datafusion-spark Spark Compatible Functions #15914
Open
19 tasks

github-actionsBot removed the Stale PR has not had any activity for some time label May 6, 2025

github-actionsBot added the Stale PR has not had any activity for some time label Jul 9, 2025

github-actionsBot closed this Jul 16, 2025

Uh oh!

Conversation

Spaarsh commented Jan 30, 2025• edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Omega359 commented Feb 3, 2025

Uh oh!

Omega359Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

Omega359Feb 3, 2025

Choose a reason for hiding this comment

Uh oh!

SpaarshFeb 4, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Omega359Feb 4, 2025

Choose a reason for hiding this comment

Uh oh!

Omega359 commented Feb 3, 2025

Uh oh!

Spaarsh commented Feb 4, 2025

Uh oh!

alamb commented Feb 5, 2025

Uh oh!

Spaarsh commented Feb 5, 2025

Uh oh!

Omega359 commented Feb 5, 2025

Uh oh!

Spaarsh commented Feb 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Omega359 commented Feb 6, 2025

Uh oh!

Spaarsh commented Feb 7, 2025

Uh oh!

Spaarsh commented Feb 7, 2025

Uh oh!

Omega359 commented Feb 7, 2025

Uh oh!

Omega359 commented Feb 9, 2025

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Spaarsh commented Feb 10, 2025

Uh oh!

alamb commented Feb 10, 2025

Uh oh!

alamb commented Feb 10, 2025

Uh oh!

Omega359 commented Feb 25, 2025

Uh oh!

alamb commented Feb 26, 2025

Uh oh!

alamb commented Feb 26, 2025

Uh oh!

github-actionsBot commented May 1, 2025

Uh oh!

alamb commented May 5, 2025

Uh oh!

Spaarsh commented May 8, 2025

Uh oh!

alamb commented May 8, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jul 9, 2025

Uh oh!

adriangb commented Sep 18, 2025

Uh oh!

Reviewers

Assignees

Spaarsh commented Jan 30, 2025•
edited by alamb
Loading

SpaarshFeb 4, 2025•
edited
Loading

alamb commented May 8, 2025•
edited
Loading