[Bug] Can't use a HDFS dir for `kyuubi.operation.result.saveToFile.dir`

[camper42]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

we use JuiceFS as our HDFS. And have set fs.jfs.impl & fs.AbstractFileSystem.jfs.impl properly.

we set spark.sql.warehouse.dir="jfs://datalake/hive"

without kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp, everything works fine.

with kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp, Spark engine failed with error java.io.IOException: JuiceFS initialized failed for jfs:///.

Our friends at JuiceFS have discovered that Paths.get in getEngineResultSavePath may cause the problem.

scala> import java.nio.file.Paths
import java.nio.file.Paths

scala> Paths.get("jfs://datalake/tmp", "test_engine_id").toString
res0: String = jfs:/datalake/tmp/test_engine_id

scala> Paths.get("hdfs://datalake/tmp", "test_engine_id").toString
res1: String = hdfs:/datalake/tmp/test_engine_id

Should this be fixed or saveToFile only support local dir ?

Affects Version(s)

1.9.0

Kyuubi Server Log Output

No response

Kyuubi Engine Log Output

24/05/30 10:51:41 WARN FileSystem: Failed to initialize fileystem jfs:/datalake/tmp/spark-f47b20fb20d44845898adcaa3ae3cb68/8d6fc3b6-eee9-4a66-885f-8cbd8a5365e7: java.io.IOException: JuiceFS initialized failed for jfs:///

Kyuubi Server Configurations

kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp

Kyuubi Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @camper42,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[wForget]

Spark engine failed with error java.io.IOException: JuiceFS initialized failed for jfs:///.

@camper42 Can you provide a full error stack?

[camper42]

@wForget

24/05/31 17:14:53 ERROR SparkSQLEngine:
    Spark application name: kyuubi_CONNECTION_SPARK_SQL_camper42_43ac361b-3028-416f-ad3a-3a2438f4b39b
          application ID:  spark-5e63469a46dd4360bfc1e02187dbdfe0
          application tags:
          application web UI: http://spark-73dd2d8fcdedc2e0-driver-svc.kyuubi.svc:4040
          master: k8s://https://10.104.0.1:443
          version: 3.5.1
          driver: [cpu: 2, mem: 8g]
          executor: [cpu: 4, mem: 10g, maxNum: 25]
    Start time: Fri May 31 17:14:25 CST 2024
    End time: Fri May 31 17:14:53 CST 2024
    User: camper42 (shared mode: CONNECTION)
    State: STOPPED
    Diagnostic: Failed to start SparkSQLEngine: JuiceFS initialized failed for jfs:///
org.apache.kyuubi.KyuubiException: Failed to start SparkSQLEngine: JuiceFS initialized failed for jfs:///
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:371)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1$adapted(SparkSQLEngine.scala:340)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.startEngine(SparkSQLEngine.scala:340)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:405)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
	at org.apache.spark.deploy.SparkSubmit.$anonfun$submit$2(SparkSubmit.scala:169)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:62)
	at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:61)
	at java.base/java.security.AccessController.doPrivileged(Unknown Source)
	at java.base/javax.security.auth.Subject.doAs(Unknown Source)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:61)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:169)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: JuiceFS initialized failed for jfs:///
	at com.juicefs.JuiceFileSystemImpl.initialize(JuiceFileSystemImpl.java:544)
	at org.apache.hadoop.fs.FilterFileSystem.initialize(FilterFileSystem.java:98)
	at com.juicefs.JuiceFileSystem.initialize(JuiceFileSystem.java:241)
	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.start(SparkSQLEngine.scala:96)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.$anonfun$startEngine$1(SparkSQLEngine.scala:358)
	... 24 more```

[wForget]

It looks like juicefs.name is not configured correctly.

https://github.com/juicedata/juicefs/blob/4c168477ca4b03cd2261a4e24c034643dc1d5d36/sdk/java/src/main/java/io/juicefs/JuiceFileSystemImpl.java#L337

[camper42]

@wForget as I mentioned above, getEngineResultSavePath/getSessionResultSavePath/getOperationResultSavePath in SparkSQLSessionManager misuse java.nio.file.Paths to process OPERATION_RESULT_SAVE_TO_FILE_DIR

https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala#L214-L226

so, even if I configured our jfs correctly, getEngineResultSavePath return a wrong path.

scala> import java.nio.file.Paths
import java.nio.file.Paths

scala> Paths.get("jfs://datalake/tmp", "test_engine_id").toString
res0: String = jfs:/datalake/tmp/test_engine_id

scala> Paths.get("hdfs://datalake/tmp", "test_engine_id").toString
res1: String = hdfs:/datalake/tmp/test_engine_id

[wForget]

Oh, I see, that means saveToFile.dir is a path without scheme, should we use org.apache.hadoop.fs.Path to handle the path?

[camper42]

actually, SparkSQLEngine and SparkSQLSessionManager use org.apache.hadoop.fs.Path after retrive path from getEngineResultSavePath and getSessionResultSavePath. 😅

https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala#L95
https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala#L191

[camper42]

1. kyuubi.operation.result.saveToFile.dir=jfs://datalake/tmp
2. getEngineResultSavePath/getSessionResultSavePath/getOperationResultSavePath use java.nio.file.Paths and return a Path like jfs:/datalake/tmp (jfs:// -> jfs:/)
3. SparkSQLEngine.start failed at val fs = path.getFileSystem(spark.sparkContext.hadoopConfiguration) https://github.com/apache/kyuubi/blob/master/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala#L96

[wForget]

Thanks @camper42 , do you mind sending a PR to fix this?

[camper42]

yes, will send a PR this afternoon