[SPARK-53946][BUILD] Upgrade SBT to 1.11.7

[sarutak]

What changes were proposed in this pull request?

This PR aims to upgrade SBT to 1.11.7

Release Notes:

• 1.9.4: https://github.com/sbt/sbt/releases/tag/v1.9.4
• 1.9.5: https://github.com/sbt/sbt/releases/tag/v1.9.5
• 1.9.6: https://github.com/sbt/sbt/releases/tag/v1.9.6
• 1.9.7: https://github.com/sbt/sbt/releases/tag/v1.9.7
• 1.9.8: https://github.com/sbt/sbt/releases/tag/v1.9.8
• 1.9.9: https://github.com/sbt/sbt/releases/tag/v1.9.9
• 1.10.0: https://github.com/sbt/sbt/releases/tag/v1.10.0
• 1.10.1: https://github.com/sbt/sbt/releases/tag/v1.10.1
• 1.10.2: https://github.com/sbt/sbt/releases/tag/v1.10.2
• 1.10.3: https://github.com/sbt/sbt/releases/tag/v1.10.3
• 1.10.4: https://github.com/sbt/sbt/releases/tag/v1.10.4
• 1.10.5: https://github.com/sbt/sbt/releases/tag/v1.10.5
• 1.10.6: https://github.com/sbt/sbt/releases/tag/v1.10.6
• 1.10.7: https://github.com/sbt/sbt/releases/tag/v1.10.7
• 1.10.8: https://github.com/sbt/sbt/releases/tag/v1.10.8
• 1.10.9: https://github.com/sbt/sbt/releases/tag/v1.10.9
• 1.10.10: https://github.com/sbt/sbt/releases/tag/v1.10.10
• 1.10.11: https://github.com/sbt/sbt/releases/tag/v1.10.11
• 1.11.0: https://github.com/sbt/sbt/releases/tag/v1.11.0
• 1.11.1: https://github.com/sbt/sbt/releases/tag/v1.11.1
• 1.11.2: https://github.com/sbt/sbt/releases/tag/v1.11.2
• 1.11.3: https://github.com/sbt/sbt/releases/tag/v1.11.3
• 1.11.4: https://github.com/sbt/sbt/releases/tag/v1.11.4
• 1.11.5: https://github.com/sbt/sbt/releases/tag/v1.11.5
• 1.11.6: https://github.com/sbt/sbt/releases/tag/v1.11.6
• 1.11.7: https://github.com/sbt/sbt/releases/tag/v1.11.7

Difference between 1.9.3 and 1.11.7 is as follows.
sbt/sbt@v1.9.3...v1.11.7

To upgrade SBT, this PR also changes SparkBuild.scala to exclude slf4j-api from mllib-local.
Without this change, running ./dev/test-dependencies.sh and ./dev/mima in this order will result in failure.

./dev/test-dependencies.sh

...

./dev/mima

[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error] 
[error]         at lmcoursier.internal.shaded.coursier.Artifacts$.$anonfun$fetchArtifacts$9(Artifacts.scala:365)
[error]         at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1(Task.scala:14)
[error]         at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$extension$1$adapted(Task.scala:14)
[error]         at lmcoursier.internal.shaded.coursier.util.Task$.wrap(Task.scala:82)
[error]         at lmcoursier.internal.shaded.coursier.util.Task$.$anonfun$flatMap$2(Task.scala:14)
[error]         at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
[error]         at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:51)
[error]         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error]         at java.base/java.lang.Thread.run(Thread.java:840)
[error] Caused by: lmcoursier.internal.shaded.coursier.cache.ArtifactError$NotFound: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar
[error]         at lmcoursier.internal.shaded.coursier.cache.internal.Downloader.$anonfun$checkFileExists$1(Downloader.scala:603)
[error]         at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error]         at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error]         at scala.util.Success.map(Try.scala:213)
[error]         at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error]         at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:42)
[error]         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:74)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[error]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[error]         at java.base/java.lang.Thread.run(Thread.java:840)
[error] (mllib-local / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] file:/root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar: not found: /root/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar

slf4j-api-1.7.5 is a transitive dependency of mllib-local.
At runtime, slf4j-api-2.0.17 will be used so dev/test-dependencies.sh downloads slf4j-api-1.7.5.pom but does not download slf4j-api-1.7.5.jar.
On the other hand, dev/mima will fetch slf4j-api-1.7.5.jar from .m2 even though it's not used at runtime but it's absent in .m2.

Why are the changes needed?

We last upgraded SBT two years ago.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

GA.

Was this patch authored or co-authored using generative AI tooling?

No.

[sarutak]

Waiting for #52652 merged.

[HyukjinKwon]

LGTM if it works

[pan3793]

@sarutak, the root cause of failure should be

coursier has always been assuming that the repository that contains metadata (like POM files) also contains the JARs.

see coursier/coursier#2942 for more details.

So, excluding a single dep does not solve the real issue.

I also experienced such failure internally even with SBT before 1.9.3, due to incomplete downloaded artifacts under .m2/repositories, I'm not sure why this fails constantly since SBT 1.9.4

[pan3793]

Instead of excluding, does it work if overriding slf4j-api version globally in

object DependencyOverrides {
   ...
}

?

[sarutak]

@pan3793

see coursier/coursier#2942 for more details.

So, excluding a single dep does not solve the real issue.

I also experienced such failure internally even with SBT before 1.9.3, due to incomplete downloaded artifacts under .m2/repositories, I'm not sure why this fails constantly since SBT 1.9.4

I had already seen the thread and I understand this solution is just a workaround but I think it's better to keep up with the latest SBT if workaround exists.
Anyway, mllib-local transitively depends on slf4j-2.0.17 when building with Maven so it's better to override or exclude slf4j-1.7.5 when building with SBT for the consistency.

[pan3793]

I had already seen the thread and I understand this solution is just a workaround but I think it's better to keep up with the latest SBT if workaround exists.

@sarutak, I agree to move forward.

Anyway, mllib-local transitively depends on slf4j-2.0.17 when building with Maven so it's better to override or exclude slf4j-1.7.5 when building with SBT for the consistency.

Excluding slf4j-api from mllib-local looks dangerous to me, because it's likely that the invoked code from deps calls slf4j-api underlying. I have tested to only do overriding, dev/mima works. Can you revert the slf4j-api exclusion from mllib-local?

[sarutak]

@pan3793

Excluding slf4j-api from mllib-local looks dangerous to me, because it's likely that the invoked code from deps calls slf4j-api underlying. I have tested to only do overriding, dev/mima works. Can you revert the slf4j-api exclusion from mllib-local?

Yes, I think overriding is safer too and I just forgot to remove excludeDependencies.
Thank you for letting me know.

[pan3793]

LGTM if CI passes.