Uh oh!
There was an error while loading. Please reload this page.
Convert approx_distinct to UDAF - #10851
Merged
Merged
Conversation
Lordworms
marked this pull request as ready for review
June 10, 2024 04:34
goldmedal
left a comment
Contributor
There was a problem hiding this comment.
Thanks @Lordworms
I think overall good but some suggestions.
| pub struct ApproxDistinct { | ||
| signature: Signature, | ||
| alias: Vec<String>, |
Contributor
There was a problem hiding this comment.
ApproxDistinct has no any alias. I think we can just remove this field and use default AggregateUDFImpl::alias
| /// Get the register histogram (each value in register index into | ||
| /// the histogram; u32 is enough because we only have 2**14=16384 registers | ||
| #[allow(dead_code)] |
Contributor
There was a problem hiding this comment.
After moving to function-aggregate, I think we can remove the original file datafusion/physical-expr/src/aggregate/hyperloglog.rs.
| pub use crate::window::rank::{dense_rank, percent_rank, rank, Rank, RankType}; | ||
| pub use crate::window::row_number::RowNumber; | ||
| pub use crate::PhysicalSortExpr; | ||
| pub use datafusion_functions_aggregate::approx_distinct::ApproxDistinct; |
Contributor
There was a problem hiding this comment.
I guess we don't need to public use it.
proto style fix prost fix clippy fix doc fix clippy
Lordworms
commented
Jun 10, 2024
ContributorAuthor
I have resolved all the reviews, plz merge this first to avoid an extra conflicts solve, Thanks for your review @goldmedal |
jayzhan211
approved these changes
Jun 11, 2024
jayzhan211
commented
Jun 11, 2024
Contributor
Thanks @Lordworms and @goldmedal |
5 tasks
findepi pushed a commit
to findepi/datafusion
that referenced
this pull request
Jul 16, 2024
* Convert approx_distinct to UDAF proto style fix prost fix clippy fix doc fix clippy * refactor code
Michael-J-Ward added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 24, 2024
… UDAF Ref: approx_distinct apache/datafusion#10851 Ref: approx_median apache/datafusion#10840 Ref: approx_percentile_cont and _with_weight apache/datafusion#10917
Michael-J-Ward added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 24, 2024
… UDAF Ref: approx_distinct apache/datafusion#10851 Ref: approx_median apache/datafusion#10840 Ref: approx_percentile_cont and _with_weight apache/datafusion#10917
Michael-J-Ward added a commit
to Michael-J-Ward/datafusion-python
that referenced
this pull request
Jul 25, 2024
… UDAF Ref: approx_distinct apache/datafusion#10851 Ref: approx_median apache/datafusion#10840 Ref: approx_percentile_cont and _with_weight apache/datafusion#10917
andygrove pushed a commit
to apache/datafusion-python
that referenced
this pull request
Jul 31, 2024
* chore: update datafusion deps * feat: impl ExecutionPlan::static_name() for DatasetExec This required trait method was added upstream [0] and recommends to simply forward to `static_name`. [0]: apache/datafusion#10266 * feat: update first_value and last_value wrappers. Upstream signatures were changed for the new new `AggregateBuilder` api [0]. This simply gets the code to work. We should better incorporate that API into `datafusion-python`. [0] apache/datafusion#10560 * migrate count to UDAF Builtin Count was removed upstream. TBD whether we want to re-implement `count_star` with new API. Ref: apache/datafusion#10893 * migrate approx_percentile_cont, approx_distinct, and approx_median to UDAF Ref: approx_distinct apache/datafusion#10851 Ref: approx_median apache/datafusion#10840 Ref: approx_percentile_cont and _with_weight apache/datafusion#10917 * migrate avg to UDAF Ref: apache/datafusion#10964 * migrage corr to UDAF Ref: apache/datafusion#10884 * migrate grouping to UDAF Ref: apache/datafusion#10906 * add alias `mean` for UDAF `avg` * migrate stddev to UDAF Ref: apache/datafusion#10827 * remove rust alias for stddev The python wrapper now provides stddev_samp alias. * migrage var_pop to UDAF Ref: apache/datafusion#10836 * migrate regr_* functions to UDAF Ref: apache/datafusion#10898 * migrate bitwise functions to UDAF The functions now take a single expression instead of a Vec<_>. Ref: apache/datafusion#10930 * add missing variants for ScalarValue with todo * fix typo in approx_percentile_cont * add distinct arg to count * comment out failing test `approx_percentile_cont` is now returning a DoubleArray instead of an IntArray. This may be a bug upstream; it requires further investigation. * update tests to expect lowercase `sum` in query plans This was changed upstream. Ref: apache/datafusion#10831 * update ScalarType data_type map * add docs dependency pickleshare * re-implement count_star * lint: ruff python lint * lint: rust cargo fmt * include name of window function in error for find_window_fn * refactor `find_window_fn` for debug clarity * search default aggregate functions by both name and aliases The alias list no longer includes the name of the function. Ref: apache/datafusion#10658 * fix markdown in find_window_fn docs * parameterize test_window_functions `first_value` and `last_value` are currently failing and marked as xfail. * add test ids to test_simple_select tests marked xfail * update find_window_fn to search built-ins first The behavior of `first_value` and `last_value` UDAFs currently does not match the built-in behavior. This allowed me to remove `marks=pytest.xfail` from the window tests. * improve first_call and last_call use of the builder API * remove trailing todos * fix examples/substrait.py * chore: remove explicit aliases from functions.rs Ref: #779 * remove `array_fn!` aliases * remove alias rules for `expr_fn_vec!` * remove alias rules from `expr_fn!` macro * remove unnecessary pyo3 var-arg signatures in functions.rs * remove pyo3 signatures that provided defaults for first_value and last_value * parametrize test_string_functions * test regr_ function wrappers Closes#778
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes#10837
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?