Drop support for Java 8

[andygrove]

What is the problem the feature request solves?

The version of arrow-java that we are using has a pretty serious bug that is fixed in version 18.3.0, but that version does not support Java 8, so we cannot upgrade.

This issue is to discuss dropping support for Java 8 in Comet.

Spark drops support for Java 8 starting with version 4.0.0
Iceberg drops support for Java 8 starting with version 1.7.0

The downside is that we will no longer be able to support users who are using Java 8 with Spark 3.4/3.5 or users with versions of Iceberg before 1.7.0, but if we cannot fix the arrow-java bug then it is probably not safe to run in production anyway.

Describe the potential solution

No response

Additional context

No response

[mbutrovich]

I'm a +1 for dropping Java 8 to bump java-arrow dependency. In addition to picking up important bug fixes, we need a newer release to benefit from Utf8View/StringView Arrow types as discussed in #1403.

[andygrove]

I see this comment in the github actions:

    - name: Install JDK ${{inputs.jdk-version}}
      uses: actions/setup-java@v4
      with:
        # distribution is chosen to be zulu as it still offers JDK 8 with Silicon support, which
        # is not available in the adopt distribution
        distribution: 'zulu'
        java-version: ${{inputs.jdk-version}}

I don't know if we want to change this to use adopt rather than zulu?

[Kontinuation]

Another Spark extension projection I am working on is also upgrading to Java 11: apache/sedona#1942. The main consideration is that major cloud vendors all provide hosted Spark service with Java 11 in at least 1 LTS version, and on-premise users can upgrade their JDK version on their own, so upgrading to Java 11 won't pose too much friction to the users.

I think it is OK to stick to zulu, as it was the default distribution in setup-java V1 and today lots of projects are still using it, including Apache Iceberg.