spark 3.3 read by snapshot ref schema by namrathamyske · Pull Request #6717 · apache/iceberg

namrathamyske · 2023-02-01T06:30:20Z

In PR #5150 we introduced read from branch or tag. There is a bug where we cannot read the snapshot specific schema changes when doing

spark.read().option("branch", <branch_name>). The fix in this PR is applied similar to #3722

Still yet to add support for tag. If given a heads up for branch, will go ahead with tag
cc @jackieo168

@rdblue @jackye1995 @amogh-jahagirdar @wypoon Let me know what you think

amogh-jahagirdar · 2023-02-01T07:21:04Z

Thanks for the fix @namrathamyske ! left some comments

namrathamyske · 2023-02-02T00:48:17Z

Thanks @jackieo168 for coming up with this!

jackye1995 · 2023-02-07T16:48:43Z

@aokolnychyi you might be also interested in this because #6651 depends on this

namrathamyske · 2023-02-10T22:59:56Z

@aokolnychyi can you take a look at this ?

amogh-jahagirdar

Looking close just style and one bug in the SparkCachedTableCatalog implementation. Looking at CI, also like some spotless issues . Thanks @namrathamyske !

jackye1995

looks good to me!

amogh-jahagirdar

Just some nits overall LGTM

aokolnychyi · 2023-02-22T17:11:34Z

I should be able to review this one today as well.

rdblue

I saw a couple of nits, but overall I think this is good to go.

jackye1995 · 2023-02-22T18:57:46Z

I just asked for permission to push to this branch, addressing the comments

amogh-jahagirdar

Thanks for addressing the nits @jackye1995 , overall this looks good to be merged from my side

aokolnychyi · 2023-02-22T21:53:51Z

A few minor nits and should be good to go. Thanks everyone!

aokolnychyi · 2023-02-22T23:26:12Z

Thanks, @namrathamyske and @jackye1995! Thanks for reviewing, @amogh-jahagirdar @rdblue @jackieo168!

spark 3.3 read by snapshot ref bug fix
5e34e23

github-actionsBot added the spark label Feb 1, 2023

namrathamyske changed the title ~~spark 3.3 read by snapshot ref bug fix for schema change~~spark 3.3 read by snapshot ref schemaFeb 1, 2023

namrathamyske marked this pull request as ready for review February 1, 2023 06:44

amogh-jahagirdar reviewed Feb 1, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java Outdated
Comment threadspark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java Outdated

namrathamyske mentioned this pull request Feb 1, 2023
Spark 3.3 write to branch snapshot #6651
Closed

jackieo168 reviewed Feb 4, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java Outdated
Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

jackye1995 self-requested a review February 6, 2023 17:45

jackye1995 requested a review from aokolnychyi February 7, 2023 16:48

rdblue reviewed Feb 9, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java Outdated

rdblue reviewed Feb 9, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java

namrathamyske added 3 commits February 10, 2023 13:58

spotless
7f88298

adding tags
1afd51b

tests fix
0d9e059

namrathamyske requested review from amogh-jahagirdar and rdblue and removed request for amogh-jahagirdar, aokolnychyi, jackye1995 and rdblue February 10, 2023 22:55

spotless
78ef8f2

tag fixes
8252200

jackye1995 reviewed Feb 11, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java

jackye1995 reviewed Feb 11, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java Outdated

jackye1995 reviewed Feb 11, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java Outdated

amogh-jahagirdar reviewed Feb 12, 2023
View reviewed changes

namrathamyske added 2 commits February 11, 2023 23:19

spotless
77ef6c4

spotless
cb3dac9

jackye1995 reviewed Feb 12, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/test/java/org/apache/iceberg/spark/source/TestSnapshotSelection.java Outdated

jackye1995 reviewed Feb 12, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java Outdated

jackye1995 reviewed Feb 12, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java Outdated

review comments changes
5d1d7a9

jackye1995 approved these changes Feb 12, 2023
View reviewed changes

amogh-jahagirdar approved these changes Feb 14, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java
Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java Outdated

jackye1995 added this to the Iceberg 1.2.0 milestone Feb 21, 2023

rdblue reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java Outdated

rdblue reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java Outdated

rdblue reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

rdblue reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/IcebergSource.java Outdated

rdblue approved these changes Feb 22, 2023
View reviewed changes

address comments
a5fb911

amogh-jahagirdar approved these changes Feb 22, 2023
View reviewed changes

aokolnychyi reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java

aokolnychyi reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCachedTableCatalog.java

aokolnychyi reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java

aokolnychyi reviewed Feb 22, 2023
View reviewed changes

Comment threadspark/v3.3/spark/src/main/java/org/apache/iceberg/spark/source/SparkTable.java Outdated

address comments
94528ce

aokolnychyi approved these changes Feb 22, 2023
View reviewed changes

aokolnychyi merged commit ce0d075 into apache:masterFeb 22, 2023

haizhou-zhao pushed a commit to haizhou-zhao/iceberg that referenced this pull request Feb 23, 2023
Spark 3.3: Support reading tag or branch via identifiers (apache#6717)
5daa320

krvikash pushed a commit to krvikash/iceberg that referenced this pull request Mar 16, 2023
Spark 3.3: Support reading tag or branch via identifiers (apache#6717)
51fbffc

Uh oh!

Conversation

namrathamyske commented Feb 1, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amogh-jahagirdar commented Feb 1, 2023

Uh oh!

namrathamyske commented Feb 2, 2023

Uh oh!

Uh oh!

Uh oh!

jackye1995 commented Feb 7, 2023

Uh oh!

Uh oh!

Uh oh!

namrathamyske commented Feb 10, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aokolnychyi commented Feb 22, 2023

Uh oh!

rdblue left a comment

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Feb 22, 2023

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aokolnychyi commented Feb 22, 2023

Uh oh!

aokolnychyi commented Feb 22, 2023• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

namrathamyske commented Feb 1, 2023•
edited
Loading

aokolnychyi commented Feb 22, 2023•
edited
Loading