Skip to content

Commit b21fb41

Browse files
pan3793LuciferYang
authored andcommitted
[SPARK-54565][CORE] SparkBuildInfo should load spark-version-info.properties from its own classloader
### What changes were proposed in this pull request? Change SparkBuildInfo to use its own classloader instead of thread context classloader to load `spark-version-info.properties`. ### Why are the changes needed? I hit an issue during the Connect JDBC driver & JetBrains DataGrip integration. ``` 2025-11-25 18:48:09,475 [ 55114] WARN - #c.i.d.d.BaseDatabaseErrorHandler$MissingDriverClassErrorInfo - Exception org.apache.spark.SparkException: Could not find spark-version-info.properties [in thread "RMI TCP Connection(3)-127.0.0.1"] java.lang.ExceptionInInitializerError: Exception org.apache.spark.SparkException: Could not find spark-version-info.properties [in thread "RMI TCP Connection(3)-127.0.0.1"] at org.apache.spark.SparkBuildInfo$.<clinit>(SparkBuildInfo.scala:35) at org.apache.spark.sql.connect.client.SparkConnectClient$.org$apache$spark$sql$connect$client$SparkConnectClient$$genUserAgent(SparkConnectClient.scala:978) at org.apache.spark.sql.connect.client.SparkConnectClient$Configuration$.apply$default$8(SparkConnectClient.scala:999) at org.apache.spark.sql.connect.client.SparkConnectClient$Builder.<init>(SparkConnectClient.scala:683) at org.apache.spark.sql.connect.client.SparkConnectClient$.builder(SparkConnectClient.scala:676) at org.apache.spark.sql.connect.client.jdbc.SparkConnectConnection.<init>(SparkConnectConnection.scala:31) at org.apache.spark.sql.connect.client.jdbc.NonRegisteringSparkConnectDriver.connect(NonRegisteringSparkConnectDriver.scala:36) at com.intellij.database.remote.jdbc.helpers.JdbcHelperImpl.connect(JdbcHelperImpl.java:786) at com.intellij.database.remote.jdbc.impl.RemoteDriverImpl.connect(RemoteDriverImpl.java:47) ``` After adding some debug messages, I found it was caused by using wrong classloader. ``` c.i.e.r.RemoteProcessSupport - ContextClassLoader: com.intellij.database.remote.jdbc.impl.JdbcClassLoader$1559cc356 c.i.e.r.RemoteProcessSupport - SparkBuildInfo ClassLoader: com.intellij.database.remote.jdbc.impl.JdbcClassLoader$JdbcClassLoaderImpl62e93ea8 ``` Similar issue that affects Hive JDBC driver and Spark's Isolated Classloader (see SPARK-32256) was fixed by [HADOOP-14067](https://issues.apache.org/jira/browse/HADOOP-14067) ### Does this PR introduce _any_ user-facing change? This fixes corner issues that the application uses multiple classloaders with Spark libs. ### How was this patch tested? Pass GHA to ensure the change breaks nothing, also manually verified the Connect JDBC driver & JetBrains DataGrip integration. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53279 from pan3793/SPARK-54565. Lead-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Signed-off-by: yangjie01 <yangjie01@baidu.com> (cherry picked from commit 095c2c3) Signed-off-by: yangjie01 <yangjie01@baidu.com>
1 parent ddc020a commit b21fb41

File tree

1 file changed

+2
-2
lines changed

1 file changed

+2
-2
lines changed

common/utils/src/main/scala/org/apache/spark/SparkBuildInfo.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ private[spark] object SparkBuildInfo {
2929
spark_build_date: String,
3030
spark_doc_root: String) = {
3131

32-
val resourceStream = Thread.currentThread().getContextClassLoader.
33-
getResourceAsStream("spark-version-info.properties")
32+
val resourceStream = getClass.getClassLoader
33+
.getResourceAsStream("spark-version-info.properties")
3434
if (resourceStream == null) {
3535
throw new SparkException("Could not find spark-version-info.properties")
3636
}

0 commit comments

Comments
 (0)