[SPARK-19851] Add support for EVERY and ANY (SOME) aggregates

[ptkool]

What changes were proposed in this pull request?

This pull request implements the EVERY and ANY aggregates.

How was this patch tested?

Testing was performed using unit tests, integration tests, and manual tests.

[ptkool]

Moved this PR to a feature branch and lost comments. The original PR is here: #17194

[rxin]

Can we just do a logical rewrite to turn them into "condA + condB + condC > 0" (for Some/Any) and "condA + condB + condC = 3" for Every?

[ptkool]

@rxin I'm not sure where you're going with your proposal. These are aggregate functions, not scalar functions.

[rxin]

I was saying rather than implementing them, just rewrite them into an aggregate on the conditions and compare them against the value.

[ptkool]

@rxin Ok. So you're proposing rewrites for these aggregates that look something like this?

some(cond) => sum(cond) > 0
every(cond) => sum(not(cond)) = 0

[ptkool]

@rxin Actually, @hvanhovell proposed the following rewrites which I think are better:

some(cond) => max(cond) = true
every(cond) => min(cond) = true

[rxin]

sgtm

[gatorsmile]

@ptkool Could you address the comments and we can review it? Thanks!

[HyukjinKwon]

ok to test

[HyukjinKwon]

@ptkool mind updating the PR please?

[SparkQA]

Test build #93044 has finished for PR 17648 at commit e0b5cc6.

• This patch fails to generate documentation.
• This patch does not merge cleanly.
• This patch adds no public classes.

[gatorsmile]

@dilipbiswal Could you take this over?

[dilipbiswal]

@gatorsmile Sure.

[gatorsmile]

@ptkool Thanks for your contribution! This feature will be available in the next release. Spark 3.0