Cluster crash when using dataflint on Databricks runtime 17.3 LTS

[CommanderWahid]

Hello,

Using dataflint with DBR 17.3 LTS in Databricks causes the cluster to crash. Below are the logs:

26/02/03 12:29:07 ERROR DriverDaemon$: XXX Fatal uncaught exception. Terminating driver.
java.lang.NoClassDefFoundError: jakarta/servlet/Servlet
at org.apache.spark.dataflint.api.DataflintJettyUtils$.createStaticHandler(DataflintJettyUtils.scala:39)
at org.apache.spark.dataflint.api.DataflintJettyUtils$.addStaticHandler(DataflintJettyUtils.scala:11)
at org.apache.spark.dataflint.api.Spark4PageFactory.addStaticHandler(Spark4PageFactory.scala:45)
at org.apache.spark.dataflint.DataflintSparkUICommonInstaller.loadUI(DataflintSparkUICommonLoader.scala:115)
at org.apache.spark.dataflint.DataflintSparkUICommonInstaller.install(DataflintSparkUICommonLoader.scala:106)
at org.apache.spark.dataflint.DataflintSparkUILoader$.install(DataflintSparkUILoader.scala:16)
at io.dataflint.spark.SparkDataflintDriverPlugin.registerMetrics(SparkDataflintPlugin.scala:26)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$registerMetrics$1(PluginContainer.scala:76)
at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$registerMetrics$1$adapted(PluginContainer.scala:75)
at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619)
at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617)
at scala.collection.AbstractIterable.foreach(Iterable.scala:935)
at org.apache.spark.internal.plugin.DriverPluginContainer.registerMetrics(PluginContainer.scala:75)
at org.apache.spark.SparkContext.$anonfun$new$60(SparkContext.scala:1104)
at org.apache.spark.SparkContext.$anonfun$new$60$adapted(SparkContext.scala:1104)
at scala.Option.foreach(Option.scala:437)
at org.apache.spark.SparkContext.(SparkContext.scala:1104)
at com.databricks.backend.daemon.driver.DatabricksILoop$.$anonfun$initializeSharedDriverContext$4(DatabricksILoop.scala:429)
at com.databricks.backend.daemon.driver.ClassLoaders$.withContextClassLoader(ClassLoaders.scala:29)
at com.databricks.backend.daemon.driver.DatabricksILoop$.initializeSharedDriverContext(DatabricksILoop.scala:429)
at com.databricks.backend.daemon.driver.DatabricksILoop$.getOrCreateSharedDriverContext(DatabricksILoop.scala:310)
at com.databricks.backend.daemon.driver.DriverCorral.sharedEnvContext(DriverCorral.scala:444)
at com.databricks.backend.daemon.driver.DriverCorral.(DriverCorral.scala:264)
at com.databricks.backend.daemon.driver.DriverDaemon.(DriverDaemon.scala:120)
at com.databricks.backend.daemon.driver.DriverDaemon$.create(DriverDaemon.scala:795)
at com.databricks.backend.daemon.driver.DriverDaemon$.initialize(DriverDaemon.scala:1128)
at com.databricks.backend.daemon.driver.DriverDaemon$.wrappedMain(DriverDaemon.scala:1068)
at com.databricks.DatabricksMain.$anonfun$main$5(DatabricksMain.scala:251)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at com.databricks.DatabricksMain.$anonfun$withStartupProfilingData$2(DatabricksMain.scala:743)
at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:510)
at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:616)
at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:643)
at com.databricks.logging.AttributionContextTracing.$anonfun$withAttributionContext$1(AttributionContextTracing.scala:80)
at com.databricks.logging.AttributionContext$.$anonfun$withValue$1(AttributionContext.scala:348)
at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
at com.databricks.logging.AttributionContext$.withValue(AttributionContext.scala:344)
at com.databricks.logging.AttributionContextTracing.withAttributionContext(AttributionContextTracing.scala:78)
at com.databricks.logging.AttributionContextTracing.withAttributionContext$(AttributionContextTracing.scala:75)
at com.databricks.DatabricksMain.withAttributionContext(DatabricksMain.scala:116)
at com.databricks.logging.AttributionContextTracing.withAttributionTags(AttributionContextTracing.scala:127)
at com.databricks.logging.AttributionContextTracing.withAttributionTags$(AttributionContextTracing.scala:108)
at com.databricks.DatabricksMain.withAttributionTags(DatabricksMain.scala:116)
at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:611)
at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:519)
at com.databricks.DatabricksMain.recordOperationWithResultTags(DatabricksMain.scala:116)
at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:511)
at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:475)
at com.databricks.DatabricksMain.recordOperation(DatabricksMain.scala:116)
at com.databricks.DatabricksMain.withStartupProfilingData(DatabricksMain.scala:742)
at com.databricks.DatabricksMain.$anonfun$main$4(DatabricksMain.scala:250)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at com.databricks.context.integrity.IntegrityCheckContext$ThreadLocalStorage$.withValue(IntegrityCheckContext.scala:73)
at com.databricks.DatabricksMain.$anonfun$main$1(DatabricksMain.scala:250)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at com.databricks.context.integrity.IntegrityCheckContext$ThreadLocalStorage$.withValue(IntegrityCheckContext.scala:73)
at com.databricks.DatabricksMain.main(DatabricksMain.scala:222)
at com.databricks.backend.daemon.driver.DriverDaemon.main(DriverDaemon.scala)
Caused by: java.lang.ClassNotFoundException: jakarta.servlet.Servlet
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
... 58 more
26/02/03 12:29:07 INFO ShutdownHookManager$: Running shutdown hook defined at com.databricks.DatabricksMain.initDatabricks
26/02/03 12:29:07 INFO ShutdownHookManager$: Completed shutdown hook defined at com.databricks.DatabricksMain.initDatabricks
26/02/03 12:29:07 INFO ShutdownHookManager$: Running shutdown hook defined at com.databricks.DatabricksMain.initDatabricks

Environment

• spark version: 4.0
• platform: Databricks
• dataflint version: 0.8.3

Installation method: using init script described here

Here is the content of the init-script :

DATAFLINT_VERSION="0.8.3"
SPARK_DEFAULTS_FILE="/databricks/driver/conf/00-custom-spark-driver-defaults.conf"

if [[ $DB_IS_DRIVER = "TRUE" ]]; then
  mkdir -p /mnt/driver-daemon/jars/
  cp /Workspace/Shared/init-script/dataflint/$DATAFLINT_VERSION/dataflint-spark4_2.13-$DATAFLINT_VERSION.jar /mnt/driver-daemon/jars/dataflint-spark4_2.13-$DATAFLINT_VERSION.jar
  echo "[driver] {" >> $SPARK_DEFAULTS_FILE
  echo "  spark.plugins = io.dataflint.spark.SparkDataflintPlugin" >> $SPARK_DEFAULTS_FILE
  echo "}" >> $SPARK_DEFAULTS_FILE
fi

The required JAR file has already been downloaded and placed in the following folder:
/Workspace/Shared/init-script/dataflint/$DATAFLINT_VERSION/