feat: Enable Spark SQL tests for Spark 3.5.1 #603
Conversation
| results as that version of Spark. To do this, we need to make some changes to the Apache Spark source code so that | ||
| it enables Comet. |
There was a problem hiding this comment.
we need to make some changes
Technically we do not need to make changes in Spark to enable Comet, but for testing purpose, we implemented some convenient features.
| ## Generating The Diff File | ||
|
|
||
| ```shell | ||
| git diff v3.5.1 58c7ce4407d2c4685a1feaf3e60cefac32de0d39 > ../datafusion-comet/dev/diffs/3.5.1.diff |
There was a problem hiding this comment.
Can be simply git diff v3.5.1 > ..
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #603 +/- ##
=============================================
- Coverage 54.66% 34.06% -20.61%
- Complexity 792 796 +4
=============================================
Files 103 108 +5
Lines 9713 38731 +29018
Branches 1851 8568 +6717
=============================================
+ Hits 5310 13195 +7885
- Misses 3451 22793 +19342
- Partials 952 2743 +1791 ☔ View full report in Codecov by Sentry. |
|
@kazuyukitanimura Spark SQL tests are now passing in CI, but I had to ignore quite a few. I will file a follow on issue for resolving those. |
There was a problem hiding this comment.
Looks good.
What would be the best way to find ignored tests that are specific to 3.5?
For example, https://github.com/apache/datafusion-comet/blob/main/dev/diffs/4.0.0-preview1.diff#L445
Spark 4.0.0-preview1.diff has #551 mentioned for all additionally ignored tests compared 3.4.3.diff
Good point. I will go ahead and update the diff with links for all newly ignored tests compared to 3.4.3. |
|
Thanks for the review @kazuyukitanimura |
* Enable Spark SQL tests for Spark 3.5.1 * fix conflicts * finish creating diff * improve docs, update slf4j version * improve docs * fix diff * fix diff * address feedback * add IgnoreComet to patch * update test * remove unused imports * ignore 2 failing tests * scalastyle * ignore failing parquet metadata tests * add links to tracking issue
Which issue does this PR close?
Closes #589
Rationale for this change
What changes are included in this PR?
How are these changes tested?