-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Spark: Support Spark 3.3 #5056
Spark: Support Spark 3.3 #5056
Conversation
"copy 3.2 files from 3.3" should be in another PR, otherwise after squash, the git will not keep commit history. |
makes sense, thanks @pan3793, will do it in a separate PR. |
spark/v3.3/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
Outdated
Show resolved
Hide resolved
UPDATE : I am investigating the RC for the two remaining UT's failure in |
spark/v3.3/spark/src/test/java/org/apache/iceberg/spark/sql/TestDeleteFrom.java
Outdated
Show resolved
Hide resolved
0788a47
to
38eb20e
Compare
ACK, opened a new PR for "copy 3.2 files from 3.3" |
38eb20e
to
dfffccc
Compare
updated this pr also with 3.3 only changes just in case we decide to merge 3.3 only first. |
superceded by #5094 Thank you everyone for your awesome reviews :) !! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Closes #4713
About Change
ReplaceIcebergData
to be used in place ofReplaceData
, as it extends command and commands are eagerly executed, which was causing problems withUpdate
(in case we allow allowScanDuplication) &RowLevelCommandDynamicPruning
(filters not being injected)RewriteDeleteFromTable
->RewriteDeleteFromIcebergTable
,OptimizeMetadataOnlyDeleteFromTable
->OptimizeMetadataOnlyDeleteFromIcebergTable
to avoid conflicting rule in upstream which doesn't support DeltaWrites and uses ReplaceDataOther Notes :
Have marked some UT's ignored for complete suite to run:
[1]
TestAddFilesProcedure
failed due to bug in spark upstream, have fixed it in upstream but since the snapshot is last based of 7th May it still failing hence ignored it.JIRA : https://issues.apache.org/jira/browse/SPARK-39417
[2] Investigating the RC for the below failures :
TestMerge#testMergeWithInvalidAssignments
TestDeleteFrom#testDeleteFromUnpartitionedTable, TestDeleteFrom#testDeleteFromPartitionedTable
cc @rdblue @aokolnychyi @jackye1995 @RussellSpitzer @kbendick @amogh-jahagirdar @rajarshisarkar @flyrain