Skip to content

Change array_agg to return null on no input rather than empty list - #11299

Merged
jayzhan211 merged 13 commits into
apache:mainfrom
jayzhan211:array-agg-no-row
Jul 10, 2024
Merged

Change array_agg to return null on no input rather than empty list#11299
jayzhan211 merged 13 commits into
apache:mainfrom
jayzhan211:array-agg-no-row

Conversation

@jayzhan211

@jayzhan211jayzhan211 commented Jul 6, 2024

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

As @findepi pointed out in #11274 (comment) that most of the aggregate function does not return non-null result if no row qualified. I double check the result in Postgres and Duckdb and find out they does not return empty list for array_agg. I think we can follow the behaviour as they did.

I also hope this can make dealing with nullability simpler

  1. Check the nullability of each (aggregate) function
  2. Make sure nullable is helpful for logical optimizer.

Closes #.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

The result of array agg is changed

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@github-actionsgithub-actionsBot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Jul 6, 2024
@jayzhan211jayzhan211 changed the title Change array agg semantic for empty result from empty list to empty rowChange array agg result from empty list to empty row if no row qualifedJul 6, 2024
@jayzhan211jayzhan211 changed the title Change array agg result from empty list to empty row if no row qualifedChange array agg result from empty list to null if no row qualifedJul 6, 2024
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Comment threaddatafusion/sqllogictest/test_files/aggregate.slt
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@github-actionsgithub-actionsBot added the logical-expr Logical plan and expressions label Jul 6, 2024
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@github-actionsgithub-actionsBot added the core Core DataFusion crate label Jul 6, 2024
@jayzhan211
jayzhan211 marked this pull request as ready for review July 6, 2024 07:49
Comment on lines -1761 to -1780
query III
WITH indices AS (
SELECT 1 AS idx UNION ALL
SELECT 2 AS idx UNION ALL
SELECT 3 AS idx UNION ALL
SELECT 4 AS idx UNION ALL
SELECT 5 AS idx
)
SELECT data.arr[indices.idx] as element, array_length(data.arr) as array_len, dummy
FROM (
SELECT array_agg(distinct c2) as arr, count(1) as dummy FROM aggregate_test_100
) data
CROSS JOIN indices
ORDER BY 1
----
1 5 100
2 5 100
3 5 100
4 5 100
5 5 100

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why removed?

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rewrite it to the simpler one!

Comment threaddatafusion/sqllogictest/test_files/aggregate.slt
statement ok
drop table t;

# test with no values

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add array_agg(distinct case on empty table

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@jayzhan211jayzhan211 mentioned this pull request Jul 7, 2024
@jayzhan211
jayzhan211 marked this pull request as draft July 8, 2024 23:51
@github-actionsgithub-actionsBot removed the logical-expr Logical plan and expressions label Jul 9, 2024
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@github-actionsgithub-actionsBot added the logical-expr Logical plan and expressions label Jul 9, 2024
@jayzhan211
jayzhan211 marked this pull request as ready for review July 9, 2024 08:56
@jayzhan211
jayzhan211 marked this pull request as draft July 9, 2024 09:54
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
@jayzhan211
jayzhan211 marked this pull request as ready for review July 9, 2024 15:29

@alambalamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me @jayzhan211 . Thank you for this PR

Also, thank you @findepi for the reviews 🙏

I double checked the answers in postgres and I believe after this PR DataFusion has consistent behavior

postgres=# create table t(a int, b float, c bigint);
ERROR: relation "t" already exists
postgres=# drop table t;DROPTABLE
postgres=# create table t(a int, b float, c bigint);
CREATE TABLE
postgres=# insert into t values (1, 1.2, 2);
INSERT 01
postgres=# select array_agg(a) from t where a > 2;
array_agg
-----------
(1 row)
postgres=# select array_agg(b) from t where b > 3.1;
array_agg
-----------
(1 row)
postgres=# select array_agg(c), count(1) from t where c > 3;
array_agg | count
-----------+-------
| 0
(1 row)
postgres=# select array_agg(distinct a) from t where a > 3;
array_agg
-----------
(1 row)

Comment threaddatafusion/sqllogictest/test_files/aggregate.slt
@alambalamb changed the title Change array agg result from empty list to null if no row qualifedChange array_agg to return null on no input rather than empty listJul 10, 2024
@jayzhan211
jayzhan211 merged commit d3f6372 into apache:mainJul 10, 2024
@jayzhan211
jayzhan211 deleted the array-agg-no-row branch July 10, 2024 23:32
@jayzhan211

Copy link
Copy Markdown
ContributorAuthor

Thanks @alamb@findepi

Lordworms pushed a commit to Lordworms/arrow-datafusion that referenced this pull request Jul 12, 2024
apache#11299)
* change array agg semantic for empty result
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* return null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix order sensitive
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add more test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix multi-phase case
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add comment
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix clone
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
---------
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
findepi pushed a commit to findepi/datafusion that referenced this pull request Jul 16, 2024
apache#11299)
* change array agg semantic for empty result
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* return null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix order sensitive
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add more test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix multi-phase case
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add comment
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix clone
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
---------
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
xinlifoobar pushed a commit to xinlifoobar/datafusion that referenced this pull request Jul 17, 2024
apache#11299)
* change array agg semantic for empty result
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* return null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix order sensitive
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add more test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix multi-phase case
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add comment
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix clone
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
---------
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
xinlifoobar pushed a commit to xinlifoobar/datafusion that referenced this pull request Jul 18, 2024
apache#11299)
* change array agg semantic for empty result
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* return null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix order sensitive
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add more test
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix null
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix multi-phase case
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* add comment
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* cleanup
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
* fix clone
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
---------
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

coreCore DataFusion cratelogical-exprLogical plan and expressionsphysical-exprChanges to the physical-expr cratessqllogictestSQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jayzhan211@findepi@alamb