Switch to Kyuubi Shaded Zookeeper

[pan3793]

Why are the changes needed?

This PR aims to migrate the vanilla Zookeeper and Curator to the Kyuubi Shaded Zookeeper. It's the first step to adapting JDK 17.

There is a known issue ZOOKEEPER-3779 that Zookeeper 3.4 client can not run on JDK 14 and above, in apache/kyuubi-shaded#5, we fixed this issue by a surgical.

With the above fixing, zk-3.4 and zk-3.6 clients both work well on JDK 17, we just randomly pick some cases to make sure zk-3.6 is tested

zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).

We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[codecov-commenter]

Codecov Report

Merging #4852 (1b36220) into master (86e5b7a) will decrease coverage by 0.09%.
The diff coverage is n/a.

❗ Current head 1b36220 differs from pull request most recent head d960cc9. Consider uploading reports for the commit d960cc9 to get more accurate results

@@             Coverage Diff              @@
##             master    #4852      +/-   ##
============================================
- Coverage     58.13%   58.04%   -0.09%     
  Complexity       13       13              
============================================
  Files           584      584              
  Lines         32593    32593              
  Branches       4321     4321              
============================================
- Hits          18949    18920      -29     
- Misses        11828    11852      +24     
- Partials       1816     1821       +5     

Impacted Files	Coverage Δ
...a/org/apache/kyuubi/ha/client/DiscoveryPaths.scala	100.00% <ø> (ø)
...ubi/ha/client/zookeeper/ZookeeperACLProvider.scala	85.00% <ø> (ø)
.../ha/client/zookeeper/ZookeeperClientProvider.scala	73.91% <ø> (ø)
...ha/client/zookeeper/ZookeeperDiscoveryClient.scala	73.57% <ø> (ø)
...he/kyuubi/jdbc/hive/ZooKeeperHiveClientHelper.java	35.05% <ø> (ø)
...rg/apache/kyuubi/server/api/v1/AdminResource.scala	85.09% <ø> (ø)
...rg/apache/kyuubi/zookeeper/EmbeddedZookeeper.scala	86.66% <ø> (ø)

... and 11 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[pan3793]

The dependency check fails because of the known issue, this PR should be good to go once kyuubi shaded 0.1.0 is released

[bowenliang123]

I would suggest that activate profile zookeeper-3.6 along with JDK 17 be default.

<id>zookeeper-3.6</id>
<activation>
      <jdk>17</jdk>
</activation>

Or create a new profile java-17 and activate this profile for zookeeper.

[pan3793]

zk-3.6 does not depend on jdk-17. we just randomly pick some cases to make sure zk-3.6 is tested

zk-3.4 client supports zk-3.4+ server, but zk-3.6 client only supports zk-3.5+ server; in the meanwhile, zk-3.4 is adopted widely, (CDH 5/6, HDP, EMR created before 2023).

We are sticky to zk-3.4 to ensure that Kyuubi can be out-of-box in the most existing Hadoop cluster but also provide zk-3.6 as an alternative(simply replace the kyuubi-shaded-zk-3.4 jar w/ kyuubi-shaded-zk-3.6, or build w/ -Pzookeeper-3.6) for users who concerns that zk-3.4 is EOL.

[pan3793]

Thanks, merged to master