Configure Kyuubi to support Delta on s3

[nishanth5028]

Hi Everyone,
I am exploring Kyuubi and am trying to setup Kyuubi to help query the delta table stored on S3(on prem). I am trying to setup Kyuubi on the local dev cluster and I need help with some questions.

1. From the documentation here https://kyuubi.readthedocs.io/en/master/connector/spark/delta_lake.html , it requires kyuubi-spark-sql-engine-1.9.0-SNAPSHOT_2.12.jar , however I do not see this jar with the kyuubi installation i.e. in the /jars or anywhere else in the kyuubi distribution.
2. Secondly I see the latest available release is 1.8.0 , however documentation indicates working with version 1.9, does kyuubi support delta lake integration with 1.8.0 release or should wait till the release 1.9.0 ?

[pan3793]

The 1.9.0-SNAPSHOT is a variable to reflect the current project version, since you access master branch docs, it's 1.9.0-SNAPSHOT.
For latest released version https://kyuubi.readthedocs.io/en/v1.8.0/connector/spark/delta_lake.html

The jar is under externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0.jar, this jar is kind of Kyuubi internal stuff, briefly speaking, it will be used by Kyuubi to bootstrap a Spark application, something like

$SPARK_HOME/spark-submit \
  --conf ... \
  $KYUUBI_HOME/externals/engines/spark/kyuubi-spark-sql-engine_2.12-1.8.0.jar

so you don't need to care about it.

[pan3793]

BTW, there is a docker compose based playground, you may want to play with it before installing in your ECS or bare metal machine