Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-49937][INFRA] Ban call the method SparkThrowable#getErrorClass #48426

Closed
wants to merge 2 commits into from

Conversation

panbingkun
Copy link
Contributor

@panbingkun panbingkun commented Oct 12, 2024

What changes were proposed in this pull request?

The pr aims to ban call the method SparkThrowable#getErrorClass.

Why are the changes needed?

After PR #48196, SparkThrowable#getErrorClass has been marked as Deprecated. In order to prevent future developers from calling SparkThrowable#getErrorClass again, which may require continuous fix and migration, calling SparkThrowable#getErrorClass is strictly prohibited as it will fail at the compilation level.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

  • Pass GA.
  • Manually test.

Was this patch authored or co-authored using generative AI tooling?

No.

@panbingkun
Copy link
Contributor Author

panbingkun commented Oct 12, 2024

Manual verification is as follows:

  • To verify if there will be errors during compilation, in the class CheckAnalysis, change the call to SparkThrowable#getCondition to call SparkThrowable#getErrorClass
    image

  • Run the following compilation commands

./build/sbt -Phadoop-3 -Pkinesis-asl -Pdocker-integration-tests -Phive -Phadoop-cloud -Pspark-ganglia-lgpl -Phive-thriftserver -Pyarn -Pkubernetes -Pvolcano Test/package streaming-kinesis-asl-assembly/assembly connect/assembly
  • Compilation will ultimately fail
    image

@@ -1621,7 +1621,7 @@ class PreemptedError() {
// errors have the lowest priority.
def set(error: Exception with SparkThrowable, priority: Option[Int] = None): Unit = {
val calculatedPriority = priority.getOrElse {
error.getCondition match {
error.getErrorClass match {
Copy link
Contributor Author

@panbingkun panbingkun Oct 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Further validation will be double-checked on GA as follows:
https://github.com/panbingkun/spark/actions/runs/11303132501/job/31439885127
image

@panbingkun
Copy link
Contributor Author

cc @MaxGekk @LuciferYang

@panbingkun panbingkun marked this pull request as ready for review October 12, 2024 05:57
@github-actions github-actions bot removed the SQL label Oct 12, 2024
@LuciferYang
Copy link
Contributor

Whether we can only ban calls to SparkThrowable#getErrorClass in Scala code. If so, we should clarify it in the PR title and description.

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants