Skip to content

Spark 3.4: Fix system function pushdown in CoW row-level commands - #10119

Merged
aokolnychyi merged 1 commit into
apache:mainfrom
aokolnychyi:system-pushdown-3.4
Apr 11, 2024
Merged

Spark 3.4: Fix system function pushdown in CoW row-level commands#10119
aokolnychyi merged 1 commit into
apache:mainfrom
aokolnychyi:system-pushdown-3.4

Conversation

@aokolnychyi

@aokolnychyiaokolnychyi commented Apr 10, 2024

Copy link
Copy Markdown
Contributor

This PR cherry-picks PR #9873 to Spark 3.4.

@aokolnychyiaokolnychyi mentioned this pull request Apr 10, 2024
Closed
@aokolnychyi

aokolnychyi commented Apr 10, 2024

Copy link
Copy Markdown
ContributorAuthor

I forgot 3.4 is a bit different, I may need to make some changes. Let me see.

@aokolnychyi

aokolnychyi commented Apr 10, 2024

Copy link
Copy Markdown
ContributorAuthor

I excluded the unnecessary branch in ReplaceStaticInvoke and adapted the tests.

szehon-ho
szehon-ho approved these changes Apr 11, 2024
View reviewed changes

@szehon-hoszehon-ho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, ReplaceData is not available in Spark 3.4?

@tmnd1991

tmnd1991 commented Apr 11, 2024

Copy link
Copy Markdown

lgtm, ReplaceData is not available in Spark 3.4?

ReplaceData is there in 3.4 but has 1 less argument (groupFilterCondition: Option[Expression]) but I see it's not used at all in ReplaceStaticInvoke logic, so I would say that we could add that branch too in 3.4 but definitely @aokolnychyi had a good reason to remove it 😄

@aokolnychyi

aokolnychyi commented Apr 11, 2024

Copy link
Copy Markdown
ContributorAuthor

In 3.4, we use custom ReplaceIcebergData and rewrite the operations differently. I think that logic is only needed in 3.5.

@aokolnychyi
aokolnychyi merged commit 290a6a0 into apache:mainApr 11, 2024
@aokolnychyi

aokolnychyi commented Apr 11, 2024

Copy link
Copy Markdown
ContributorAuthor

Thanks for reviewing, @szehon-ho@tmnd1991@nastra!

@aokolnychyiaokolnychyi added this to the Iceberg 1.5.1 milestone Apr 17, 2024
amogh-jahagirdar pushed a commit to amogh-jahagirdar/iceberg that referenced this pull request Apr 18, 2024
@aokolnychyi@amogh-jahagirdar
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
a2376a7
…ache#10119)
@amogh-jahagirdaramogh-jahagirdar mentioned this pull request Apr 18, 2024
Merged
nastra pushed a commit that referenced this pull request Apr 18, 2024
@amogh-jahagirdar@aokolnychyi
Spark 3.4: Fix system function pushdown in CoW row-level commands (#1…
cbb8530
…0119) (#10171)
Co-authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
@lurnagao-dahua

lurnagao-dahua commented Aug 26, 2024

Copy link
Copy Markdown
Contributor

Hi,Is there a plan to port to Spark 3.3?

szehon-ho pushed a commit to szehon-ho/iceberg that referenced this pull request Sep 16, 2024
@amogh-jahagirdar@aokolnychyi
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
63551f7
…ache#10119) (apache#10171)
Co-authored-by: Anton Okolnychyi <aokolnychyi@apple.com>
sasankpagolu pushed a commit to sasankpagolu/iceberg that referenced this pull request Oct 27, 2024
@aokolnychyiSasank Pagolu
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
99ee0a9
…ache#10119)
zachdisc pushed a commit to zachdisc/iceberg that referenced this pull request Dec 23, 2024
@aokolnychyi@zachdisc
Spark 3.4: Fix system function pushdown in CoW row-level commands (ap…
3f69fa0
…ache#10119)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nastranastranastra approved these changes
@szehon-hoszehon-hoszehon-ho approved these changes

Assignees

No one assigned

Labels

spark

Projects

None yet

Milestone

Iceberg 1.5.1

Development

Successfully merging this pull request may close these issues.

5 participants

@aokolnychyi@tmnd1991@lurnagao-dahua@nastra@szehon-ho