Add spark sql integration test for Hudi #3194
Conversation
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
There was a problem hiding this comment.
Nit: we put versions in the file pluginlibs.versions.toml, refer it as line 35 does.
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
| // TODO: extract a polaris-rest module as a thin layer for | ||
| // client to depends on. | ||
| implementation(project(":polaris-core")) { isTransitive = false } | ||
| testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:1.1.0") |
There was a problem hiding this comment.
for the actual spark project, we don't really intend to introduce any table format specific dependency, even for testing. i didn't see any change in the actual spark project, is there a reason that we need this?
There was a problem hiding this comment.
let me try removing and see what occurs
| exclude("org.slf4j", "jul-to-slf4j") | ||
| } | ||
|
|
||
| // Add spark-hive for Hudi integration - provides HiveExternalCatalog that Hudi needs |
There was a problem hiding this comment.
@rahil-c could you also update the readme to include the support for hudi?
It would be great if we could also have a notebook in the get-started to help people to onboard for hudi, we could do that in follow up, we should also extend the regress test to include actual end to end test for hudi to avoid any potential break of the feature
There was a problem hiding this comment.
Will try to follow up in seperate pr for this if possible?
|
This PR is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days. |
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
github-project-automation
bot
moved this from Done
to PRs In Progress
in Basic Kanban Board
rahil-c
force-pushed
the
rahil/polaris-hudi-it-main
branch
from
1d5e8b2 to
d349939
Compare
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
5 participants
Description
SparkDeltaITChecklist
CHANGELOG.md(if needed)site/content/in-dev/unreleased(if needed)