ARROW-11188: [Rust] Support crypto functions from PostgreSQL dialect by ovr · Pull Request #9139 · apache/arrow

ovr · 2021-01-08T15:53:24Z

Implemented functions:

github-actions · 2021-01-08T17:14:19Z

https://issues.apache.org/jira/browse/ARROW-11188

alamb · 2021-01-08T19:30:27Z

This PR probably needs to be rebased to pick up the fix for #9138 FYI

ovr · 2021-01-08T19:41:14Z

Thank you @alamb for notice 👍

I've done with PR, marked it as ready for review and awaiting review from DF's team.

jorgecarleitao

@ovr , thanks a lot, looks really good. 👍 I left some minor comments. :)

jorgecarleitao · 2021-01-09T11:32:53Z

This will crash on null values, no?
Binary can also be built from an iterator, afai remember.

Yes, it will crash.
Replaced this with Ok(array.iter().map(|x| x.map(|x| $FUNC(x))).collect()) without calling as_slice() directly on SHA2DigestOutput<D> and It works. Wierd... How is it possible?)

jorgecarleitao · 2021-01-09T11:44:21Z

IMO we should have a test of one of these functions here, with and without nulls, and with an empty string.

Good catch, I added tests for nulls and empty strings by SQL execution.

…(md5, sha)

ovr · 2021-01-09T13:06:52Z

Thank you, @jorgecarleitao, for your review. I've added tests + fix handling null values.

I've compared it with PostgreSQL, and It works as expected.

select
md5('tom') AS md5_tom,
md5('') AS md5_empty_str,
md5(null) AS md5_null,
encode(sha224('tom'), 'hex') AS sha224_tom,
encode(sha224(''), 'hex') AS sha224_empty_str,
sha224(null) AS sha224_null;

[
{
"md5_tom": "34b7da764b21d298ef307d04d8152dc5",
"md5_empty_str": "d41d8cd98f00b204e9800998ecf8427e",
"md5_null": null,
"sha224_tom": "0bf6cb62649c42a9ae3876ab6f6d92ad36cb5414e495f8873292be4d",
"sha224_empty_str": "d14a028c2a3a2bc9476102bb288234c415a2b01f828ea62ac5b3e42f",
"sha224_null": null
}
]

Thanks

alamb · 2021-01-10T11:57:36Z

 pin-project-lite= "^0.2.0"
 tokio = { version = "0.2", features = ["macros", "blocking", "rt-core", "rt-threaded", "sync"] }
 log = "^0.4"
+md-5 = "^0.9.1"


It seems to me like we might want to start offering a way to keep the number of required dependencies of DataFusion down. For example, in this case we could potentially put the use of crypto functions behind a feature flag.
I am not proposing to add the feature flag as part of this PR, but more like trying to set the general direction of allowing users to pick features that they need and not have to pay compilation time (or binary size) cost for those they don't
What do you think @jorgecarleitao and @andygrove

I generally agree with you, @alamb. In this case, we want to support posgres dialect, so it makes sense to support these functions (and not implement these ourselves, as they are even security related).
In general, as long as the crates are small, I do not see a major issue. Our expensive dependencies are Tokio, crossbeam, etc, specially because they really increase the compile time (e.g. compared to the arrow crate).
We already offer a scalar UDF that has the same performance as our own expressions. So, I think that this is the most we can do here.

alamb · 2021-01-10T12:02:38Z

- Signature::Uniform(1, vec![DataType::Utf8, DataType::LargeUtf8])
- }
- BuiltinScalarFunction::Rtrim => {
+ BuiltinScalarFunction::Upper


👍 nice cleanup

codecov-io · 2021-01-10T13:01:57Z

Codecov Report

Merging #9139 (871e18c) into master (08cccd6) will decrease coverage by 0.04%.
The diff coverage is 66.66%.

@@ Coverage Diff @@## master #9139 +/- ##
==========================================
- Coverage 81.81% 81.77% -0.05% 
==========================================
Files 214 215 +1 Lines 51373 51461 +88 ==========================================
+ Hits 42033 42083 +50 - Misses 9340 9378 +38

Impacted Files	Coverage Δ
rust/datafusion/src/logical_plan/expr.rs	`76.92% <ø> (ø)`
rust/datafusion/src/physical_plan/mod.rs	`86.00% <ø> (ø)`
rust/arrow/src/util/display.rs	`48.33% <33.33%> (-0.79%)`	⬇️
rust/datafusion/src/physical_plan/functions.rs	`74.82% <60.00%> (-3.69%)`	⬇️
...datafusion/src/physical_plan/crypto_expressions.rs	`75.00% <75.00%> (ø)`
rust/datafusion/tests/sql.rs	`99.82% <100.00%> (+<0.01%)`	⬆️
rust/datafusion/src/datasource/csv.rs	`65.00% <0.00%> (-16.25%)`	⬇️
rust/datafusion/src/datasource/parquet.rs	`94.77% <0.00%> (-1.44%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08cccd6...871e18c. Read the comment docs.

mqy · 2021-01-11T15:10:16Z

Perhaps the PR description should be rewritten before merge.

alamb · 2021-01-11T21:41:19Z

I filed https://issues.apache.org/jira/browse/ARROW-11214 to track the feature flag idea

alamb · 2021-01-11T21:41:56Z

Thanks again for the contribution @ovr !

ovr force-pushed the crypto-functions branch from 4a06629 to 7f262d2Compare January 8, 2021 17:03

github-actionsBot added Component: Rust - DataFusion Component: Rust labels Jan 8, 2021

ovr force-pushed the crypto-functions branch 3 times, most recently from 22d0ec5 to acf8fe0Compare January 8, 2021 19:38

ovr marked this pull request as ready for review January 8, 2021 19:38

ovr force-pushed the crypto-functions branch from acf8fe0 to 14d759aCompare January 8, 2021 19:41

jorgecarleitao reviewed Jan 9, 2021
View reviewed changes

ovr added 2 commits January 9, 2021 15:58

ARROW-11188: [Rust] Support crypto functions from PostgreSQL dialect …
014811e
…(md5, sha)

ARROW-11188: [Rust] Add tests, handle null values
3baf93c

ovr force-pushed the crypto-functions branch from 0cb4653 to 3baf93cCompare January 9, 2021 12:58

alamb approved these changes Jan 10, 2021
View reviewed changes

ARROW-11188: [Rust] Export crypto functions in physical plan
871e18c

mqy mentioned this pull request Jan 11, 2021
ARROW-8676: [Rust] IPC RecordBatch body compression #9137
Closed

ovr changed the title ~~ARROW-11188: [Rust] Support crypto functions from PostgreSQL dialect …~~ARROW-11188: [Rust] Support crypto functions from PostgreSQL dialectJan 11, 2021

alamb closed this in 6da7718Jan 11, 2021

alamb mentioned this pull request Apr 26, 2021
Add optional rust features for functions in library to keep dependencies down apache/datafusion#146
Closed

asfimport mentioned this pull request Jan 11, 2021
[Rust] Implement crypto functions from PostgreSQL dialect #27093
Closed

Uh oh!

Conversation

ovr commented Jan 8, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actionsBot commented Jan 8, 2021

Uh oh!

alamb commented Jan 8, 2021

Uh oh!

ovr commented Jan 8, 2021

Uh oh!

jorgecarleitao left a comment

Choose a reason for hiding this comment

Uh oh!

jorgecarleitaoJan 9, 2021

Choose a reason for hiding this comment

Uh oh!

ovrJan 9, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jorgecarleitaoJan 9, 2021

Choose a reason for hiding this comment

Uh oh!

ovrJan 9, 2021

Choose a reason for hiding this comment

Uh oh!

ovr commented Jan 9, 2021

Uh oh!

alambJan 10, 2021

Choose a reason for hiding this comment

Uh oh!

jorgecarleitaoJan 10, 2021

Choose a reason for hiding this comment

Uh oh!

alambJan 10, 2021

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Jan 10, 2021

Codecov Report

Uh oh!

mqy commented Jan 11, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jan 11, 2021

Uh oh!

alamb commented Jan 11, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ovr commented Jan 8, 2021•
edited
Loading

ovrJan 9, 2021•
edited
Loading

mqy commented Jan 11, 2021•
edited
Loading