Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spark 3.1: Use snapshot schema for time travel queries #3768

Merged
merged 1 commit into from
Dec 18, 2021

Conversation

wypoon
Copy link
Contributor

@wypoon wypoon commented Dec 17, 2021

Port of #3722 to v3.1.

Notes:
I took the diff from #3722, edited it to change v3.2 to v3.1 in the paths, and tried to git apply 3722.diff. That did not apply, so I did git am -3 < 3722.diff, which left conflicts in SparkTable.java and TestDeleteFrom.java.

The conflicts in SparkTable.java were straightforward to resolve.

TestDeleteFrom in v3.2 has testDeleteFromUnpartitionedTable and testDeleteFromPartitionedTable which are note present in v3.1; I had based the new test, testDeleteFromTableAtSnapshot on testDeleteFromUnpartitionedTable, and it is still usable because we don't actually delete anything, just check that an exception is thrown.

Also, for the negative test cases added in TestDeleteFrom and TestUnpartitionedWrites, I had to exclude the SparkSessionCatalog configuration from the parameterized testing, as otherwise I get

org.apache.spark.sql.AnalysisException: The namespace in session catalog must have exactly one name part: default.table.snapshot_id_9024049109842482017
	at org.apache.spark.sql.connector.catalog.LookupCatalog$SessionCatalogAndIdentifier$.unapply(LookupCatalog.scala:65)

instead of the IllegalArgumentException I was looking for.

@github-actions github-actions bot added the spark label Dec 17, 2021
@rdblue
Copy link
Contributor

rdblue commented Dec 18, 2021

Thanks, @wypoon! Ready to merge when tests are passing.

@rdblue rdblue merged commit e62f3b7 into apache:master Dec 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants