[SPARK-52010] Do not generate API docs for internal classes #50797

cloud-fan · 2025-05-06T01:52:12Z

What changes were proposed in this pull request?

While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them.

Why are the changes needed?

API doc should not include internal classes

Does this PR introduce any user-facing change?

no

How was this patch tested?

N/A

Was this patch authored or co-authored using generative AI tooling?

no

cloud-fan · 2025-05-06T01:52:36Z

common/utils/src/main/scala/org/apache/spark/util/SparkStringUtils.scala

@@ -16,11 +16,11 @@
 */
 package org.apache.spark.util

-trait SparkStringUtils {
+private[spark] trait SparkStringUtils {


added in #49062

cloud-fan · 2025-05-06T01:52:50Z

common/utils/src/main/scala/org/apache/spark/util/SparkTestUtils.scala

@@ -25,7 +25,7 @@ import javax.tools.{JavaFileObject, SimpleJavaFileObject, ToolProvider}

 import scala.jdk.CollectionConverters._

-trait SparkTestUtils {
+private[spark] trait SparkTestUtils {


added in #43354

cloud-fan · 2025-05-06T01:55:08Z

connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala

@@ -30,7 +30,7 @@ import org.apache.spark.sql.catalyst.util.{FailFastMode, ParseMode, PermissiveMo
 import org.apache.spark.sql.errors.QueryCompilationErrors
 import org.apache.spark.sql.types._

-private[sql] case class AvroDataToCatalyst(


4.0 added two new internal classes in this package: AvroExpressionEvalUtils, AvroCompressionCodec. Given that this package contains only internal classes and deprecated APIs since 3.0, I marked the entire org.apache.spark.sql.avro package as private, so that we no longer need private[sql] in these existing classes.

cloud-fan · 2025-05-06T01:55:42Z

project/SparkBuild.scala

-        "org.apache.spark.sql.scripting"
+        "org.apache.spark.kafka010",
+        "org.apache.spark.network",
+        "org.apache.spark.sql.avro",


See https://github.com/apache/spark/pull/50797/files#r2074591645

cloud-fan · 2025-05-06T01:56:55Z

.../kafka-0-10-token-provider/src/main/scala/org/apache/spark/kafka010/KafkaConfigUpdater.scala

@@ -30,7 +30,7 @@ import org.apache.spark.internal.Logging
 /**
 * Class to conveniently update Kafka config params, while logging the changes
 */
-private[spark] case class KafkaConfigUpdater(module: String, kafkaParams: Map[String, Object])


4.0 added one new internal class in this package: KafkaTokenProviderExceptions. Given that this package contains only internal classes, I marked the entire org.apache.spark.kafka010 package as private, so that we no longer need private[spark] in these existing classes.

cloud-fan · 2025-05-06T01:57:09Z

project/SparkBuild.scala

        "org.apache.spark.deploy",
-        "org.apache.spark.util.collection",
-        "org.apache.spark.sql.scripting"
+        "org.apache.spark.kafka010",


See https://github.com/apache/spark/pull/50797/files#r2074596294

cloud-fan · 2025-05-06T01:57:30Z

project/SparkBuild.scala

+        "org.apache.spark.network",
+        "org.apache.spark.sql.avro",
+        "org.apache.spark.sql.scripting",
+        "org.apache.spark.types.variant",


The new variant project adds this new package, but it only contain internal classes.

cloud-fan · 2025-05-06T01:57:59Z

project/SparkBuild.scala

+        "org.apache.spark.sql.avro",
+        "org.apache.spark.sql.scripting",
+        "org.apache.spark.types.variant",
+        "org.apache.spark.ui.flamegraph",


added by #42988 , should be intrenal

cloud-fan · 2025-05-06T01:58:34Z

sql/catalyst/src/main/scala/org/apache/spark/sql/exceptions/SqlScriptingException.scala

@@ -23,7 +23,7 @@ import org.apache.spark.{SparkThrowable, SparkThrowableHelper}
 import org.apache.spark.sql.catalyst.trees.Origin
 import org.apache.spark.sql.exceptions.SqlScriptingException.errorMessageWithLineNumber

-class SqlScriptingException (
+private[sql] class SqlScriptingException (


added by the new scripting feature.

cloud-fan · 2025-05-06T01:58:52Z

sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala

@@ -871,7 +871,7 @@ abstract class JdbcDialect extends Serializable with Logging {
 /**
 * Make the `classifyException` method throw out the original exception
 */
-trait NoLegacyJDBCError extends JdbcDialect {
+private[sql] trait NoLegacyJDBCError extends JdbcDialect {


added by #46937

cloud-fan · 2025-05-06T01:59:34Z

cc @HyukjinKwon @dongjoon-hyun @yaooqinn @hvanhovell

dongjoon-hyun

+1, it looks good to me. Thank you, @cloud-fan .

### What changes were proposed in this pull request? While reviewing the API doc of 4.0 RC4, I found that there are some newly added internal classes appear in the API doc. This PR fixes them. ### Why are the changes needed? API doc should not include internal classes ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #50797 from cloud-fan/api. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Kent Yao <yao@apache.org> (cherry picked from commit 9f5ae88) Signed-off-by: Kent Yao <yao@apache.org>

yaooqinn · 2025-05-06T03:51:07Z

Merged to master/4.0, thank you @cloud-fan @dongjoon-hyun

HyukjinKwon

LGTM2

### What changes were proposed in this pull request? This is a followup of #50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #51038 from cloud-fan/mima. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

### What changes were proposed in this pull request? This is a followup of #50797 , to exclde these private packages from mima check. ### Why are the changes needed? mima shouldn't check private APIs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A ### Was this patch authored or co-authored using generative AI tooling? no Closes #51038 from cloud-fan/mima. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com> (cherry picked from commit f634311) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Do not generate API docs for private classes

9452d5d

github-actions bot added SQL BUILD DSTREAM AVRO labels May 6, 2025

cloud-fan commented May 6, 2025

View reviewed changes

cloud-fan changed the title ~~[SPARK-52010] Do not generate API docs for private classes~~ [SPARK-52010] Do not generate API docs for internal classes May 6, 2025

dongjoon-hyun approved these changes May 6, 2025

View reviewed changes

yaooqinn approved these changes May 6, 2025

View reviewed changes

yaooqinn closed this in 9f5ae88 May 6, 2025

HyukjinKwon reviewed May 6, 2025

View reviewed changes

cloud-fan mentioned this pull request May 28, 2025

[INFRA][MINOR] Update mima exclude #51038

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-52010] Do not generate API docs for internal classes #50797

[SPARK-52010] Do not generate API docs for internal classes #50797

Uh oh!

cloud-fan commented May 6, 2025

Uh oh!

cloud-fan May 6, 2025 •

edited

Loading

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan May 6, 2025

Uh oh!

cloud-fan commented May 6, 2025

Uh oh!

dongjoon-hyun left a comment

Uh oh!

yaooqinn commented May 6, 2025

Uh oh!

HyukjinKwon left a comment

Uh oh!

Uh oh!

[SPARK-52010] Do not generate API docs for internal classes #50797

[SPARK-52010] Do not generate API docs for internal classes #50797

Uh oh!

Conversation

cloud-fan commented May 6, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

cloud-fan May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented May 6, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented May 6, 2025

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cloud-fan May 6, 2025 •

edited

Loading