Uh oh!
There was an error while loading. Please reload this page.
ARROW-11606: [Rust] [DataFusion] Add input schema to HashAggregateExec - #9481
ARROW-11606: [Rust] [DataFusion] Add input schema to HashAggregateExec#9481andygrove wants to merge 2 commits into
Conversation
andygrove
commented
Feb 12, 2021
@jorgecarleitao Does this make sense? |
jorgecarleitao
commented
Feb 12, 2021
Makes a lot of sense to me. 👍 Does it address the issue on Ballista? |
andygrove
commented
Feb 12, 2021
Yes, I think so. See https://github.com/ballista-compute/ballista/pull/505 ... I am going to finish testing this over the weekend. |
jorgecarleitao
left a comment
There was a problem hiding this comment.
LGTM. Thanks @andygrove , and great work on Ballista so far . Putting DataFusion up to the test :)
alamb
commented
Feb 12, 2021
FWIW @jorgecarleitao and @andygrove -- I have been using DataFusion quite heavily in IOx (e.g. https://github.com/influxdata/influxdb_iox/pull/795) and it is doing great. DataFusion is a really cool piece of software |
andygrove
commented
Feb 12, 2021
Thanks for the quick review and merge @jorgecarleitao and @alamb. I think distributed query execution will be working well enough in Ballista to support some TPC-H queries this weekend! The great news is that Ballista is now a tiny codebase. DataFusion is working out really well thanks to all the great work that went into 3.0.0 ❤️ |
To make it easier to implement serde for `HashAggregateExec` we need access to the schema that the aggregate expressions are compiled against. For `Partial` aggregates this is the same as the schema of the input. However, for `Final` aggregates it is not, which is why we need to add this additional schema information. Closesapache#9481 from andygrove/hash-agg-input-schema Authored-by: Andy Grove <andygrove73@gmail.com> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>
To make it easier to implement serde for
HashAggregateExecwe need access to the schema that the aggregate expressions are compiled against. ForPartialaggregates this is the same as the schema of the input. However, forFinalaggregates it is not, which is why we need to add this additional schema information.