feat(function): add greatest function by rluvaton · Pull Request #12474 · apache/datafusion

rluvaton · 2024-09-15T15:39:46Z

Which issue does this PR close?

Rationale for this change

Support more operators

What changes are included in this PR?

Implement the Spark implementation for greatest:
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.greatest.html

Are these changes tested?

I was not able to find where you put the functions test, the coalesce for example has only one test for the return type, I must missing something

datafusion/datafusion/functions/src/core/coalesce.rs

Lines 142 to 157 in 7bd7747

mod test {
use arrow::datatypes::DataType;
use datafusion_expr::ScalarUDFImpl;
usecrate::core;
#[test]
fntest_coalesce_return_types(){
let coalesce = core::coalesce::CoalesceFunc::new();
let return_type = coalesce 
.return_type(&[DataType::Date32,DataType::Date32])
.unwrap();
assert_eq!(return_type,DataType::Date32);
}
}

Are there any user-facing changes?

Yes, I need to add documentation, but first check if this feature is desired

This match the Spark implementation for greatest: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.greatest.html

Weijun-H · 2024-09-18T14:40:35Z

FYI #12357 (comment)

comphead · 2024-09-19T15:02:13Z

Related to #6531

rluvaton · 2024-09-23T20:32:11Z

I was not able to find where you put the functions test, the coalesce for example has only one test for the return type, I must missing something

Hey @alamb, we talked about the test location at the beginning of the CMU talk you gave, where the tests are located?

alamb · 2024-09-25T16:39:07Z

I think the tests I was talking about are described here:

https://github.com/apache/datafusion/tree/4a3c09a9316fb8940aeb1c5b9b48bc3b7259d5d4/datafusion/sqllogictest#readme

If you want to test the type of a function you can do it like

select arrow_typeif(my_func(foo)))

rluvaton · 2024-09-25T18:00:41Z

I see, thank you, how is your experience with debugging this kind of test?

alamb · 2024-09-25T19:42:40Z

It is great!

findepi

nice!

findepi · 2024-11-14T11:00:59Z

+
+ let cmp = make_comparator(lhs, rhs, SORT_OPTIONS)?;
+
+ let len = lhs.len().min(rhs.len());


they array lengths should be equal (otherwise we would be losing data)

findepi · 2024-11-14T11:01:21Z

+ // If both arrays are not nested, have the same length and no nulls, we can use the faster vectorised kernel
+ // - If both arrays are not nested: Nested types, such as lists, are not supported as the null semantics are not well-defined.
+ // - both array does not have any nulls: cmp::gt_eq will return null if any of the input is null while we want to return false in that case
+ if !lhs.data_type().is_nested() && lhs.null_count() == 0 && rhs.null_count() == 0 {


this probably should use logical null count

findepi · 2024-11-14T11:02:18Z

+ // - If both arrays are not nested: Nested types, such as lists, are not supported as the null semantics are not well-defined.
+ // - both array does not have any nulls: cmp::gt_eq will return null if any of the input is null while we want to return false in that case
+ if !lhs.data_type().is_nested() && lhs.null_count() == 0 && rhs.null_count() == 0 {
+ return cmp::gt_eq(&lhs, &rhs).map_err(|e| e.into());


please add a test with float NaN values.

…eatest # Conflicts: # datafusion/sqllogictest/test_files/functions.slt

Weijun-H · 2024-11-17T04:08:47Z

Please add a document function for it

jayzhan211

👍

Weijun-H

LGTM! Thanks @rluvaton

waynexia · 2024-11-22T07:22:14Z

+ for array in arrays_iter {
+ largest = keep_larger(Arc::clone(array), largest)?;
+ }


keep_larger constructs an intermediate array for each call. I'd prefer to reduce the materialization to just the last one. It's okay to do this in the follow-up PR

I'll do it in a separate PR, this PR is large enough

Tried using interleave like we talked but than I would not be able to use the kernels functions so I did not do it

waynexia

We also need to add proto-related supports for this new function. But it is not a blocker for merging this PR from my perspective. Thank you @rluvaton

rluvaton · 2024-11-22T07:28:41Z

What is proto related support?

waynexia · 2024-11-22T07:33:17Z

What is proto related support?

Sorry, forget it 🙈 They are wiped out in #10173

rluvaton · 2024-11-22T08:41:51Z

Thank you @waynexia see you on Monday at your CMU talk, I'll join 10 minutes earlier so we can talk about data fusion if interested 😊

rluvaton · 2024-11-23T15:28:28Z

Can we merge this?

waynexia · 2024-11-23T15:51:47Z

Thank you again for working on this!

rluvaton · 2024-11-23T17:24:02Z

Thank you, I'll try to contribute again!

rluvaton added 2 commits September 15, 2024 18:28

feat(function): add greatest function
69761df
This match the Spark implementation for greatest: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.greatest.html

Merge branch 'apache:main' into add-greatest
7d63bd2

github-actionsBot added the functions Changes to functions implementation label Sep 15, 2024

remove unused
4904440

rluvaton closed this Sep 23, 2024

rluvaton deleted the add-greatest branch September 23, 2024 12:22

waynexia mentioned this pull request Nov 14, 2024
Add greatest(T,...) and least(T,...) SQL functions #6531
Closed

rluvaton restored the add-greatest branch November 14, 2024 08:23

rluvaton reopened this Nov 14, 2024

findepi reviewed Nov 14, 2024
View reviewed changes

rluvaton added 6 commits November 16, 2024 17:46

fix finding common supertype in greatest
5e3a260

allow single argument for greatest
b929239

assert that both array have the same length
966116b

use logical null count
d065d4a

remove unused import
ccbea44

add docs
ed2dbad

github-actionsBot added the documentation Improvements or additions to documentation label Nov 16, 2024

rluvaton added 2 commits November 16, 2024 20:52

add greatest slt tests
4df3c39

add greatest slt tests
1c51c87

github-actionsBot added the sqllogictest SQL Logic Tests (.slt) label Nov 16, 2024

rluvaton added 2 commits November 16, 2024 21:12

Merge remote-tracking branch 'refs/remotes/upstream/main' into add-gr…
e6c5841
…eatest # Conflicts: # datafusion/sqllogictest/test_files/functions.slt

fix merge conflicts
655419a

rluvaton marked this pull request as ready for review November 16, 2024 19:14

Weijun-H reviewed Nov 17, 2024
View reviewed changes

Comment threaddocs/source/user-guide/sql/scalar_functions.md Outdated

rluvaton added 3 commits November 18, 2024 23:09

add docs
47652e6

revert manual docs changes
cfa6185

Merge remote-tracking branch 'upstream/main' into add-greatest
fda197f

github-actionsBot removed the documentation Improvements or additions to documentation label Nov 18, 2024

rluvaton requested a review from Weijun-H November 18, 2024 21:38

jayzhan211 reviewed Nov 19, 2024
View reviewed changes

Comment threaddatafusion/functions/src/core/greatest.rs Outdated

Update based on cr
03fbd8f

waynexia self-requested a review November 21, 2024 16:31

rluvaton added 4 commits November 21, 2024 20:47

fix lint
df5ad68

run fmt
2d83abf

run clippy
0db1fd0

Uppdated docs using ./dev/update_function_docs.sh
d985681

github-actionsBot added the documentation Improvements or additions to documentation label Nov 21, 2024

jayzhan211 approved these changes Nov 22, 2024
View reviewed changes

Weijun-H approved these changes Nov 22, 2024
View reviewed changes

waynexia reviewed Nov 22, 2024
View reviewed changes

waynexia approved these changes Nov 22, 2024
View reviewed changes

waynexia merged commit fe2da2b into apache:mainNov 23, 2024

rluvaton deleted the add-greatest branch November 23, 2024 17:24

alamb mentioned this pull request Dec 3, 2024
Dec 3. 2024: This week in DataFusion #13630
Closed
3 tasks

rluvaton mentioned this pull request Dec 14, 2024
feat(function): add least function #13786
Merged

	mod test {
	use arrow::datatypes::DataType;

	use datafusion_expr::ScalarUDFImpl;

	usecrate::core;

	#[test]
	fntest_coalesce_return_types(){
	let coalesce = core::coalesce::CoalesceFunc::new();
	let return_type = coalesce
	.return_type(&[DataType::Date32,DataType::Date32])
	.unwrap();
	assert_eq!(return_type,DataType::Date32);
	}
	}


		let cmp = make_comparator(lhs, rhs, SORT_OPTIONS)?;

		let len = lhs.len().min(rhs.len());

Uh oh!

Conversation

rluvaton commented Sep 15, 2024

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Weijun-H commented Sep 18, 2024

Uh oh!

comphead commented Sep 19, 2024

Uh oh!

rluvaton commented Sep 23, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Sep 25, 2024

Uh oh!

rluvaton commented Sep 25, 2024

Uh oh!

alamb commented Sep 25, 2024

Uh oh!

findepi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Weijun-H commented Nov 17, 2024

Uh oh!

Uh oh!

Uh oh!

jayzhan211 left a comment

Choose a reason for hiding this comment

Uh oh!

Weijun-H left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

waynexia left a comment

Choose a reason for hiding this comment

Uh oh!

rluvaton commented Nov 22, 2024

Uh oh!

waynexia commented Nov 22, 2024

Uh oh!

rluvaton commented Nov 22, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rluvaton commented Nov 23, 2024

Uh oh!

waynexia commented Nov 23, 2024

Uh oh!

rluvaton commented Nov 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

rluvaton commented Sep 23, 2024•
edited
Loading

rluvaton commented Nov 22, 2024•
edited
Loading