Skip to content

Implement Grouped Map UDFs - #143

Merged
imback82 merged 12 commits into
dotnet:masterfrom
eerhardt:GroupedMapUdfs
Jul 9, 2019
Merged

Implement Grouped Map UDFs#143
imback82 merged 12 commits into
dotnet:masterfrom
eerhardt:GroupedMapUdfs

Conversation

@eerhardt

@eerhardteerhardt commented Jun 14, 2019

Copy link
Copy Markdown
Member

Implementing "Grouped map UDFs" functionality, similar to Pandas UDF functionality - https://docs.databricks.com/spark/latest/spark-sql/udf-python-pandas.html#grouped-map-udfs.

Grouped Map UDFs take an Arrow RecordBatch and return an Arrow RecordBatch. The shape and the number of rows returned are defined by the user. This allows for reducing and expanding rows in a UDF.

Implements part of #45.

@imback82imback82 added this to the June 2019 milestone Jun 15, 2019
@imback82imback82 added the enhancement New feature or request label Jun 15, 2019
imback82
imback82 reviewed Jun 18, 2019
View reviewed changes
Comment threadexamples/Microsoft.Spark.CSharp.Examples/Sql/Basic.cs Outdated
Comment threadexamples/Microsoft.Spark.CSharp.Examples/Sql/Basic.cs Outdated
Comment threadexamples/Microsoft.Spark.CSharp.Examples/Sql/Basic.cs Outdated
Comment threadexamples/Microsoft.Spark.CSharp.Examples/Sql/Basic.cs Outdated
new IArrowArray[]
{
records.Column(groupFieldIndex),
CreateArrowArray(characterCount)

@imback82imback82Jun 18, 2019

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we not call this if returnLength is 0? (I know this works but just to be explicit)

@eerhardteerhardtJun 28, 2019

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would we use instead? null? I think that can lead to other errors down the line.

Comment threadsrc/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/DataFrameTests.cs Outdated
Comment threadsrc/csharp/Microsoft.Spark/Sql/Column.cs Outdated
Comment threadsrc/csharp/Microsoft.Spark/Sql/RelationalGroupedDataset.cs
Comment threadsrc/csharp/Microsoft.Spark/Sql/RelationalGroupedDataset.cs Outdated
Comment threadsrc/csharp/Microsoft.Spark/Sql/RelationalGroupedDataset.cs Outdated
Comment threadsrc/csharp/Microsoft.Spark/Sql/WorkerFunction.cs Outdated
@eerhardt

eerhardt commented Jul 2, 2019

Copy link
Copy Markdown
MemberAuthor

I've responded to all feedback. Please take a look and let me know your thoughts.

suhsteve
suhsteve reviewed Jul 3, 2019
View reviewed changes
Comment threadsrc/csharp/Microsoft.Spark.E2ETest/IpcTests/Sql/DataFrameTests.cs
@eerhardt

eerhardt commented Jul 9, 2019

Copy link
Copy Markdown
MemberAuthor

This PR should be ready to be merged. Let me know if you have any other thoughts/comments.

imback82
imback82 approved these changes Jul 9, 2019
View reviewed changes

@imback82imback82 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@imback82
imback82 merged commit 815c2a9 into dotnet:masterJul 9, 2019
@eerhardt
eerhardt deleted the GroupedMapUdfs branch July 9, 2019 22:53
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@suhstevesuhstevesuhsteve left review comments
@stephentoubstephentoubAwaiting requested review from stephentoub
@rapothrapothAwaiting requested review from rapoth
+1 more reviewer
@imback82imback82imback82 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@eerhardteerhardt

Labels

enhancementNew feature or request

Projects

None yet

Milestone

June 2019

Development

Successfully merging this pull request may close these issues.

3 participants

@eerhardt@suhsteve@imback82