[SPARK-42664][CONNECT] Support `bloomFilter` function for `DataFrameStatFunctions` by LuciferYang · Pull Request #40352 · apache/spark

LuciferYang · 2023-03-09T11:41:23Z

What changes were proposed in this pull request?

This is pr using BloomFilterAggregate to implement bloomFilter function for DataFrameStatFunctions.

Why are the changes needed?

Add Spark connect jvm client api coverage.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Add new test
Manually check Scala 2.13

LuciferYang · 2023-03-10T10:16:53Z

+ numBits: Long,
+ fpp: Double): BloomFilter = {
+
+ val dataType = sparkSession


Add this due to:
check col support type in server side
Add Cast for IntegerType/ShortType/ByteType

Can we use the TypeOf expression for this instead? Alternatively we can try to figure this out in the planner.

LuciferYang · 2023-04-07T10:09:15Z

In the last commit, make BloomFilterAggregate explicitly supported IntegerType/ShortType/ByteType and added corresponding updaters, then removed pass dataType and adding cast nodes

LuciferYang · 2023-04-07T10:10:45Z

GA failure is not related to the current PR

LuciferYang · 2023-04-07T10:46:32Z

+ numBits: Long,
+ fpp: Double): BloomFilter = {
+
+ val agg = if (!fpp.isNaN) {


Before chanage to always pass 3 parameters, always pass all 4 parameters.
Now change to pass (col, expectedNumItems, fpp) if !fpp. isNaN , otherwise pass (col, expectedNumItems, numBits).

hvanhovell · 2023-08-09T14:01:24Z

@LuciferYang can we restart this effort.

I promise I will look at the proto PR :)

LuciferYang · 2023-08-09T14:04:18Z

This pr has been rebased many times, should I submit a new one...

LuciferYang · 2023-08-09T14:18:47Z

I make a clean one #42414

LuciferYang marked this pull request as draft March 9, 2023 11:41

github-actionsBot added CONNECT SQL labels Mar 9, 2023

LuciferYang commented Mar 9, 2023
View reviewed changes

Comment thread...connect/server/src/main/scala/org/apache/spark/sql/connect/service/SparkConnectService.scala Outdated

LuciferYang changed the title ~~[SPARK-42664][CONNECT] Support bloomFilter function for DataFrameStatFunctions~~[WIP][SPARK-42664][CONNECT] Support bloomFilter function for DataFrameStatFunctionsMar 9, 2023

support
af12b55

LuciferYang force-pushed the SPARK-42664 branch from f42da43 to af12b55Compare March 9, 2023 14:19

LuciferYang added 2 commits March 9, 2023 22:21

format
6f2e736

remove Alias
92da589

hvanhovell reviewed Mar 9, 2023
View reviewed changes

Comment thread...connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala Outdated

hvanhovell reviewed Mar 9, 2023
View reviewed changes

Comment threadconnector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala Outdated

hvanhovell reviewed Mar 9, 2023
View reviewed changes

Comment threadconnector/connect/client/jvm/src/main/scala/org/apache/spark/sql/DataFrameStatFunctions.scala Outdated

hvanhovell reviewed Mar 9, 2023
View reviewed changes

Comment threadconnector/connect/client/jvm/src/test/scala/org/apache/spark/sql/DataFrameStatSuite.scala Outdated

LuciferYang added 7 commits March 10, 2023 14:18

support Int/Byte/Short
71fba34

support string
ab78bae

support string
ebcec0b

change to check numBits
6ee5528

tmp fix
dfffd53

refactor
8d0e517

move check to server side
d193de0

LuciferYang commented Mar 10, 2023
View reviewed changes

Comment thread...connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala Outdated

LuciferYang changed the title ~~[WIP][SPARK-42664][CONNECT] Support bloomFilter function for DataFrameStatFunctions~~[SPARK-42664][CONNECT] Support bloomFilter function for DataFrameStatFunctionsMar 10, 2023

LuciferYang marked this pull request as ready for review March 10, 2023 15:38

LuciferYang added 6 commits March 13, 2023 10:35

Merge branch 'apache:master' into SPARK-42664
943fcdb

try remove lazy
dc62996

make lazy
df56a05

Merge branch 'apache:master' into SPARK-42664
d225405

Merge branch 'apache:master' into SPARK-42664
f588a6d

Merge branch 'apache:master' into SPARK-42664
e7bb98a

hvanhovell reviewed Apr 7, 2023
View reviewed changes

Comment thread...rc/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/BloomFilterAggregate.scala Outdated

LuciferYang added 2 commits April 7, 2023 17:11

merge case
176ce09

refactor to remove pass dataType
ea2234c

chanage to always pass 3 parameters
5bd6169

LuciferYang commented Apr 7, 2023
View reviewed changes

LuciferYang added 18 commits April 7, 2023 18:52

add NaN check in Planner
43d6305

bridge optimalNumOfBits
dbbb61a

Merge branch 'apache:master' into SPARK-42664
9520355

Merge branch 'apache:master' into SPARK-42664
3a24371

Merge branch 'apache:master' into SPARK-42664
c9c9f79

Merge branch 'apache:master' into SPARK-42664
a244420

Merge branch 'upmaster' into SPARK-42664
d004993

Merge branch 'apache:master' into SPARK-42664
62fdf91

Merge branch 'apache:master' into SPARK-42664
60217da

Merge branch 'apache:master' into SPARK-42664
79d7d89

Merge branch 'apache:master' into SPARK-42664
61bea56

Merge branch 'apache:master' into SPARK-42664
80d6e6d

Merge branch 'apache:master' into SPARK-42664
caab9bf

Merge branch 'upmaster' into SPARK-42664
e3aeb86

Merge branch 'apache:master' into SPARK-42664
1fdc5c0

Merge branch 'apache:master' into SPARK-42664
8861190

Merge branch 'upmaster' into SPARK-42664
efcc1b5

Merge branch 'upmaster' into SPARK-42664
5134264

LuciferYang closed this Aug 9, 2023

LuciferYang deleted the SPARK-42664 branch March 30, 2026 08:09

Navigation Menu

Uh oh!

[SPARK-42664][CONNECT] Support `bloomFilter` function for `DataFrameStatFunctions` - #40352

[SPARK-42664][CONNECT] Support `bloomFilter` function for `DataFrameStatFunctions`#40352
LuciferYang wants to merge 47 commits into
apache:masterfrom
LuciferYang:SPARK-42664

LuciferYang commented Mar 9, 2023•
edited
Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LuciferYangMar 10, 2023•
edited
Loading

Uh oh!

hvanhovellApr 7, 2023

Uh oh!

Uh oh!

Uh oh!

LuciferYang commented Apr 7, 2023

Uh oh!

LuciferYang commented Apr 7, 2023

Uh oh!

LuciferYangApr 7, 2023•
edited
Loading

Uh oh!

hvanhovell commented Aug 9, 2023

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

LuciferYang commented Mar 9, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LuciferYangMar 10, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvanhovellApr 7, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

LuciferYang commented Apr 7, 2023

Uh oh!

LuciferYang commented Apr 7, 2023

Uh oh!

LuciferYangApr 7, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hvanhovell commented Aug 9, 2023

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

LuciferYang commented Aug 9, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

LuciferYang commented Mar 9, 2023•
edited
Loading

LuciferYangMar 10, 2023•
edited
Loading

LuciferYangApr 7, 2023•
edited
Loading