Move `Count` to `functions-aggregate`, update MSRV to rust 1.75 by jayzhan211 · Pull Request #10484 · apache/datafusion

jayzhan211 · 2024-05-13T12:54:58Z

Which issue does this PR close?

Part of #8708

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

There are 3 todos after this PR,

remove builtin aggregate function Count:
remove expr_fn::count and count distinct
rename count's name lowercase

I will file an issue before merging this one

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-06-05T09:45:31Z

error: package `clap_builder v4.5.2` cannot be built because it requires rustc 1.74 or newer, while the currently active rustc version is 1.73.0
Either upgrade to rustc 1.74 or newer, or use
cargo update -p clap_builder@4.5.2 --precise ver
where `ver` is the latest version of `clap_builder` supporting rustc 1.73.0

clap_builder needs 1.74, which used in criterion v0.5.1 (in datafusion-physical-expr)

│ │ ├── petgraph v0.6.5
│ │ │ ├── fixedbitset v0.4.2
│ │ │ └── indexmap v2.2.6 (*)
│ │ └── regex v1.10.4 (*)
│ │ [dev-dependencies]
│ │ ├── arrow v51.0.0 (*)
│ │ ├── criterion v0.5.1
│ │ │ ├── anes v0.1.6
│ │ │ ├── cast v0.3.0
│ │ │ ├── ciborium v0.2.2
│ │ │ │ ├── ciborium-io v0.2.2
│ │ │ │ ├── ciborium-ll v0.2.2
│ │ │ │ │ ├── ciborium-io v0.2.2
│ │ │ │ │ └── half v2.4.1 (*)
│ │ │ │ └── serde v1.0.203 (*)
│ │ │ ├── clap v4.5.4
│ │ │ │ ├── clap_builder v4.5.2
│ │ │ │ │ ├── anstream v0.6.14 (*)
│ │ │ │ │ ├── anstyle v1.0.7
│ │ │ │ │ ├── clap_lex v0.7.0
│ │ │ │ │ └── strsim v0.11.1
│ │ │ │ └── clap_derive v4.5.4 (proc-macro)
│ │ │ │ ├── heck v0.5.0
│ │ │ │ ├── proc-macro2 v1.0.85 (*)
│ │ │ │ ├── quote v1.0.36 (*)
│ │ │ │ └── syn v2.0.66 (*)

I upgrade to 1.75 to the latest available version.

I'm not sure why we have this issue in this PR 🤔

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

alamb · 2024-06-11T15:29:09Z

This PR has several conflicts -- other than those, do you think this PR is ready for another review @jayzhan211 ?

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 · 2024-06-12T00:47:00Z

This PR has several conflicts -- other than those, do you think this PR is ready for another review @jayzhan211 ?

Yes, it is ready for review.

Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

alamb

Awesome -- thank you @jayzhan211 -- let's do it.

alamb · 2024-06-12T22:27:35Z

 readme = "README.md"
 repository = "https://github.com/apache/datafusion"
-rust-version = "1.73"
+rust-version = "1.75"


By my reading of https://github.com/apache/datafusion?tab=readme-ov-file#rust-version-compatibility-policy
is that since 1.74 was released in Nov 2023, we can safely update in this PR

jayzhan211 · 2024-06-12T23:54:47Z

Thanks @alamb !

…he#10484) * mv accumulate indices Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * complete udaf Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * register Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix expr Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * filter distinct count Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * todo: need to move count distinct too Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * move code around Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * move distinct to aggr-crate Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * replace Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * backup Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix function name and physical expr Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix physical optimizer Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix all slt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix with args Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * add label Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * revert builtin related code back Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix test Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix substrait Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix doc Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmy Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix udaf macro for distinct but not apply Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix count distinct and use workspace Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * add reverse Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * remove old code Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * backup Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * use macro Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * expr builder Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * introduce expr builder Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * add example Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * clean agg sta Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * combine agg Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * limit distinct and fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup name Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix ci Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix window Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix ci Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix merged Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix rebase Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * cleanup Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * use std Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * update mrsv Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * upd msrv Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * revert test Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * downgrade to 1.75 Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * 1.76 Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * ahas Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * revert to 1.75 Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * rm count Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix merge Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fmt Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * clippy Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * rm sum in test_no_duplicate_name Signed-off-by: jayzhan211 <jayzhan211@gmail.com> * fix Signed-off-by: jayzhan211 <jayzhan211@gmail.com> --------- Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 added 15 commits May 11, 2024 10:59

mv accumulate indices
755215d
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

complete udaf
12366e6
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

register
9f43638
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix expr
7dd2aae
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

filter distinct count
2fb0c2a
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

todo: need to move count distinct too
9dd044c
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move code around
ab9eaaa
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

move distinct to aggr-crate
1506453
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

replace
6a554ee
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

Merge branch 'count-udaf' into count-for-all
dae3061

backup
a44c967
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix function name and physical expr
dd5a90f
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix physical optimizer
c9eebb3
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix all slt
6375d47
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

cleanup
37d8974
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

github-actionsBot added sql SQL Planner logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels May 13, 2024

jayzhan211 added 7 commits May 13, 2024 20:58

Merge remote-tracking branch 'upstream/main' into count-for-all
643ffe9

cleanup
9917b37
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix with args
c6ba32b
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

add label
2d80a8b
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

revert builtin related code back
2bcb10b
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix test
12f27c4
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

fix substrait
a391407
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

github-actionsBot added the substrait Changes to the substrait crate label May 14, 2024

fix doc
3ace4bf
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rm count
ea864e0
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 marked this pull request as ready for review June 5, 2024 06:34

jayzhan211 requested a review from alamb June 5, 2024 06:35

jayzhan211 added 6 commits June 7, 2024 07:35

Merge remote-tracking branch 'upstream/main' into count-for-all
d621fb6

Merge remote-tracking branch 'upstream/main' into count-for-all
b3f8f8c

Merge remote-tracking branch 'upstream/main' into count-for-all
121e4cb

fix merge
d89424d
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

Merge remote-tracking branch 'upstream/main' into count-for-all
a88916d

fmt
0924248
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 added 2 commits June 12, 2024 08:42

Merge remote-tracking branch 'upstream/main' into count-for-all
33a02f9

clippy
4b13fc0
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

rm sum in test_no_duplicate_name
4bd680d
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 marked this pull request as draft June 12, 2024 07:53

jayzhan211 force-pushed the count-for-all branch 2 times, most recently from 45249fb to 4bd680dCompare June 12, 2024 13:48

jayzhan211 added 2 commits June 12, 2024 21:49

Merge remote-tracking branch 'upstream/main' into count-for-all
cf6f9ac

fix
97a1227
Signed-off-by: jayzhan211 <jayzhan211@gmail.com>

jayzhan211 marked this pull request as ready for review June 12, 2024 14:21

alamb changed the title ~~Move Count to functions-aggregate~~Move Count to functions-aggregate, update MSRVJun 12, 2024

alamb changed the title ~~Move Count to functions-aggregate, update MSRV~~Move Count to functions-aggregate, update MSRV to rust 1.75Jun 12, 2024

alamb approved these changes Jun 12, 2024
View reviewed changes

jayzhan211 merged commit 8f718dd into apache:mainJun 12, 2024

jayzhan211 deleted the count-for-all branch June 12, 2024 23:54

findepi mentioned this pull request Jul 2, 2024
Add example of dataframe API aggregations #11219
Closed

huaxingao mentioned this pull request Jul 8, 2024
feat: Upgrade to DataFusion 40.0.0-rc1 apache/datafusion-comet#644
Merged

Navigation Menu

Uh oh!

Move `Count` to `functions-aggregate`, update MSRV to rust 1.75 - #10484

Move `Count` to `functions-aggregate`, update MSRV to rust 1.75#10484
jayzhan211 merged 81 commits into
apache:mainfrom
jayzhan211:count-for-all

jayzhan211 commented May 13, 2024•
edited
Loading

Uh oh!

jayzhan211 commented Jun 5, 2024•
edited
Loading

Uh oh!

alamb commented Jun 11, 2024

Uh oh!

jayzhan211 commented Jun 12, 2024

Uh oh!

alamb left a comment

Uh oh!

alambJun 12, 2024

Uh oh!

jayzhan211 commented Jun 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

jayzhan211 commented May 13, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

jayzhan211 commented Jun 5, 2024• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alamb commented Jun 11, 2024

Uh oh!

jayzhan211 commented Jun 12, 2024

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alambJun 12, 2024

Choose a reason for hiding this comment

Uh oh!

jayzhan211 commented Jun 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jayzhan211 commented May 13, 2024•
edited
Loading

jayzhan211 commented Jun 5, 2024•
edited
Loading