Skip to content

Spark 3.1: Use snapshot schema for time travel queries - #3768

Merged
rdblue merged 1 commit into
apache:masterfrom
wypoon:schema-for-snapshot-spark31
Dec 18, 2021
Merged

Spark 3.1: Use snapshot schema for time travel queries#3768
rdblue merged 1 commit into
apache:masterfrom
wypoon:schema-for-snapshot-spark31

Conversation

@wypoon

@wypoonwypoon commented Dec 17, 2021

Copy link
Copy Markdown
Contributor

Port of #3722 to v3.1.

Notes:
I took the diff from #3722, edited it to change v3.2 to v3.1 in the paths, and tried to git apply 3722.diff. That did not apply, so I did git am -3 < 3722.diff, which left conflicts in SparkTable.java and TestDeleteFrom.java.

The conflicts in SparkTable.java were straightforward to resolve.

TestDeleteFrom in v3.2 has testDeleteFromUnpartitionedTable and testDeleteFromPartitionedTable which are note present in v3.1; I had based the new test, testDeleteFromTableAtSnapshot on testDeleteFromUnpartitionedTable, and it is still usable because we don't actually delete anything, just check that an exception is thrown.

Also, for the negative test cases added in TestDeleteFrom and TestUnpartitionedWrites, I had to exclude the SparkSessionCatalog configuration from the parameterized testing, as otherwise I get

org.apache.spark.sql.AnalysisException: The namespace in session catalog must have exactly one name part: default.table.snapshot_id_9024049109842482017
at org.apache.spark.sql.connector.catalog.LookupCatalog$SessionCatalogAndIdentifier$.unapply(LookupCatalog.scala:65)

instead of the IllegalArgumentException I was looking for.

@rdblue

Copy link
Copy Markdown
Contributor

Thanks, @wypoon! Ready to merge when tests are passing.

@rdblue
rdblue merged commit e62f3b7 into apache:masterDec 18, 2021
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@wypoon@rdblue