Skip to content

Support Avg distinct for float64 type - #15413

Closed
qazxcdswe123 wants to merge 6 commits into
apache:mainfrom
qazxcdswe123:avg-distinct-float
Closed

Support Avg distinct for float64 type#15413
qazxcdswe123 wants to merge 6 commits into
apache:mainfrom
qazxcdswe123:avg-distinct-float

Conversation

@qazxcdswe123

@qazxcdswe123qazxcdswe123 commented Mar 25, 2025

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The original PR is splited into 2 parts, one for float64 type and one for float64+decimal type

What changes are included in this PR?

  1. mv DistinctSumAccumulator to common so that it can be used in Float64DistinctAvgAccumulator
  2. implement Float64DistinctAvgAccumulator using DistinctSumAccumulator
  3. tested in aggregate.slt

Are these changes tested?

  • tested in aggregate.slt

Are there any user-facing changes?

No

@github-actionsgithub-actionsBot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Mar 25, 2025
Comment threaddatafusion/functions-aggregate-common/src/aggregate/avg_distinct/numeric.rs Outdated
@Omega359

Copy link
Copy Markdown
Contributor

the sqlite tests need updating prior to this issue being pushed to main. Confirmed test failures, here are a few examples:

External error: query is expected to fail with error:
(regex) DataFusion error: Execution error: avg\(DISTINCT\) aggregations are not available
but got error:
DataFusion error: Arrow error: Invalid argument error: number of columns(2) must match number of fields(3) in schema
[SQL] SELECT - COUNT ( * ) * AVG ( DISTINCT + col3 ) AS col3 FROM tab0 WHERE NOT NULL <= + + col3
at ../../datafusion-testing/data/sqlite/index/random/100/slt_good_1.slt:11104
External error: query is expected to fail, but actually succeed:
[SQL] SELECT DISTINCT - COUNT ( * ) * AVG ( DISTINCT + 89 ) col4 FROM tab0 AS cor0 WHERE NULL NOT IN ( CAST ( NULL AS INTEGER ) + - 15 )
at ../../datafusion-testing/data/sqlite/index/random/1000/slt_good_8.slt:17760
External error: query is expected to fail, but actually succeed:
[SQL] SELECT - AVG ( DISTINCT 27 ) + - MAX ( DISTINCT col1 ) AS col4 FROM tab0 WHERE NOT NULL <> - + 90
at ../../datafusion-testing/data/sqlite/index/random/1000/slt_good_0.slt:6040
External error: query is expected to fail with error:
(regex) DataFusion error: Execution error: avg\(DISTINCT\) aggregations are not available
but got error:
DataFusion error: Arrow error: Invalid argument error: number of columns(1) must match number of fields(2) in schema
[SQL] SELECT ALL - AVG ( DISTINCT 48 ) AS col4 FROM tab0 WHERE NOT - col4 BETWEEN ( NULL ) AND CAST ( 2 AS INTEGER )
at ../../datafusion-testing/data/sqlite/index/random/1000/slt_good_7.slt:38250

@alamb

Copy link
Copy Markdown
Contributor

I merged up from main and will use the new github action to run extended tests

@alamb

Copy link
Copy Markdown
Contributor

Run extended tests

@Omega359

Copy link
Copy Markdown
Contributor

Run extended tests

I see it did trigger but I somehow was expecting feedback in the comments

@Omega359

Copy link
Copy Markdown
Contributor

jayzhan211
jayzhan211 previously approved these changes Mar 29, 2025
@jayzhan211
jayzhan211 dismissed their stale reviewMarch 29, 2025 01:41

extended test

@Omega359

Copy link
Copy Markdown
Contributor

I regenerated the sqlite tests again and I think an issue I'm seeing with them is actually caused or triggered by this PR.

Here is an example:

query error DataFusion error: Arrow error: Invalid argument error: number of columns\(1\) must match number of fields\(2\) in schema
SELECT84*+-61*+AVG ( DISTINCT ( 70 ) ) AS col2 FROM tab4 WHERE ( NULL ) BETWEEN NULLAND col3

If this is in fact a bug not caused by this PR a new issue should be filed. I myself do not have the time to diagnosis the cause of this unfortunately.

@qazxcdswe123

Copy link
Copy Markdown
ContributorAuthor

I regenerated the sqlite tests again and I think an issue I'm seeing with them is actually caused or triggered by this PR.

Here is an example:

query error DataFusion error: Arrow error: Invalid argument error: number of columns\(1\) must match number of fields\(2\) in schema
SELECT84*+-61*+AVG ( DISTINCT ( 70 ) ) AS col2 FROM tab4 WHERE ( NULL ) BETWEEN NULLAND col3

If this is in fact a bug not caused by this PR a new issue should be filed. I myself do not have the time to diagnosis the cause of this unfortunately.

sadly I'm working on my undergrad thesis project at this time and do not have time to investigate this either 😢 , might be back around mid april

@alamb

alamb commented Apr 6, 2025

Copy link
Copy Markdown
Contributor

sadly I'm working on my undergrad thesis project at this time and do not have time to investigate this either 😢 , might be back around mid april

Good luck with your project / thesis!

@qazxcdswe123
qazxcdswe123 marked this pull request as draft April 24, 2025 13:42
@github-actionsgithub-actionsBot added documentation Improvements or additions to documentation sql SQL Planner development-process Related to development process of DataFusion logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate substrait Changes to the substrait crate labels Apr 25, 2025
@github-actionsgithub-actionsBot added catalog Related to the catalog crate common Related to common crate execution Related to the execution crate proto Related to proto crate datasource Changes to the datasource crate ffi Changes to the ffi crate labels Apr 25, 2025
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

catalogRelated to the catalog cratecommonRelated to common cratecoreCore DataFusion cratedatasourceChanges to the datasource cratedevelopment-processRelated to development process of DataFusiondocumentationImprovements or additions to documentationexecutionRelated to the execution crateffiChanges to the ffi cratefunctionsChanges to functions implementationlogical-exprLogical plan and expressionsoptimizerOptimizer rulesphysical-exprChanges to the physical-expr cratesprotoRelated to proto cratesqlSQL PlannersqllogictestSQL Logic Tests (.slt)substraitChanges to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@qazxcdswe123@Omega359@alamb@jayzhan211