Skip to content

HBASE-28153 Upgrade zookeeper to a newer version #5475

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 1, 2023
Merged

HBASE-28153 Upgrade zookeeper to a newer version #5475

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 0 additions & 7 deletions hbase-it/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,13 +132,6 @@
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
<!-- Hadoop needs Netty 3.x at test scope for the minicluster -->
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>${netty.hadoop.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
Expand Down
7 changes: 0 additions & 7 deletions hbase-server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -699,13 +699,6 @@
</exclusion>
</exclusions>
</dependency>
<!-- Hadoop needs Netty 3.x at test scope for the minicluster>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>${netty.hadoop.version}</version>
<scope>test</scope>
</dependency-->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minikdc</artifactId>
Expand Down
9 changes: 9 additions & 0 deletions hbase-zookeeper/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,15 @@
<artifactId>log4j-slf4j-impl</artifactId>
<scope>test</scope>
</dependency>
<!-- ZooKeeperServer needs the below dependencies, thus MiniZooKeeperCluster also needs them -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This undeclared dependency of recent zookeeper versions is annoying. We will CNFE without it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I think a better way is to have a zookeeper-client module and a zookeeper-server module. Now they only have a zookeeper module but they only declare the dependencies for zookeeper-client at compile scope so we need to manually add the zookeeper server dependencies...

</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one can be at test scope.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We expose MiniZooKeeperCluster as IA.Public and it is under src/main, not src/test, so we must declare the dependencies which are only used by zookeeper server at compile/runtime scope...

IIRC there is a issue for removing the class but there are objections from our users...

</dependency>
</dependencies>

<build>
Expand Down
133 changes: 35 additions & 98 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -808,9 +808,18 @@
-->
<hadoop.version>${hadoop-three.version}</hadoop.version>
<assembly.file>src/main/assembly/hadoop-three-compat.xml</assembly.file>
<!--This property is for hadoops netty. HBase netty
comes in via hbase-thirdparty hbase-shaded-netty-->
<netty.hadoop.version>3.10.5.Final</netty.hadoop.version>
<!--
These property is for transitive netty dependencies from thirdparty dependencies, like
hadoop and zookeeper. HBase netty comes in via hbase-thirdparty hbase-shaded-netty
In the old time, netty-all includes all the classes but maven does not know it so it is
possible that we have netty-all and netty-handler both on the classpath but they have
different version and cause conflicts. Newer version of netty-all solved the this problem
by depending all other netty modules, but we'd better still specify the version by our own
in the dependencyManagement section as it could still lead to different versions of netty
modules and cause trouble if we only rely on transitive dependencies.
-->
<netty3.version>3.10.6.Final</netty3.version>
<netty4.version>4.1.100.Final</netty4.version>
<!-- end HBASE-15925 default hadoop compatibility values -->
<audience-annotations.version>0.13.0</audience-annotations.version>
<!--
Expand All @@ -825,6 +834,7 @@
<commons-io.version>2.11.0</commons-io.version>
<commons-lang3.version>3.9</commons-lang3.version>
<commons-math.version>3.6.1</commons-math.version>
<commons-cli.version>1.5.0</commons-cli.version>
<disruptor.version>3.4.4</disruptor.version>
<httpclient.version>4.5.13</httpclient.version>
<httpcore.version>4.4.13</httpcore.version>
Expand All @@ -846,7 +856,7 @@
<protobuf.plugin.version>0.6.1</protobuf.plugin.version>
<thrift.path>thrift</thrift.path>
<thrift.version>0.14.1</thrift.version>
<zookeeper.version>3.5.7</zookeeper.version>
<zookeeper.version>3.8.3</zookeeper.version>
<jline.version>2.11</jline.version>
<slf4j.version>1.7.30</slf4j.version>
<clover.version>4.0.3</clover.version>
Expand Down Expand Up @@ -1417,6 +1427,11 @@
<artifactId>commons-math3</artifactId>
<version>${commons-math.version}</version>
</dependency>
<dependency>
<groupId>commons-cli</groupId>
<artifactId>commons-cli</artifactId>
<version>${commons-cli.version}</version>
</dependency>
<dependency>
<!-- commons-logging is only used by hbase-http's HttpRequestLog and hbase-server's
HBaseTestingUtil.
Expand All @@ -1431,44 +1446,12 @@
<version>${zookeeper.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</exclusion>
<exclusion>
<groupId>com.github.spotbugs</groupId>
<artifactId>spotbugs-annotations</artifactId>
</exclusion>
<exclusion>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-core</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
</exclusion>
</exclusions>
</dependency>
Expand Down Expand Up @@ -1791,6 +1774,18 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
<version>${netty3.version}</version>
</dependency>
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-bom</artifactId>
<version>${netty4.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
Expand Down Expand Up @@ -3560,14 +3555,6 @@
<groupId>org.codehaus.jackson</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
Expand Down Expand Up @@ -3659,14 +3646,6 @@
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
Expand Down Expand Up @@ -3696,14 +3675,6 @@
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>javax.servlet</groupId>
<artifactId>servlet-api</artifactId>
Expand Down Expand Up @@ -3902,17 +3873,6 @@
<groupId>stax</groupId>
<artifactId>stax-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<!-- MiniMRCluster needs the hadoop netty-all transitive include... a
netty-all 4.0.52 version.... so we have to let it through.
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
-->
<exclusion>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
Expand Down Expand Up @@ -3941,14 +3901,6 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down Expand Up @@ -3990,13 +3942,6 @@
<groupId>org.slf4j</groupId>
<artifactId>slf4j-reload4j</artifactId>
</exclusion>
<!--
Needed in test context when hadoop-3.3 runs.
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
-->
<exclusion>
<groupId>org.codehaus.jackson</groupId>
<artifactId>*</artifactId>
Expand Down Expand Up @@ -4062,14 +4007,6 @@
<groupId>stax</groupId>
<artifactId>stax-api</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty</artifactId>
</exclusion>
<exclusion>
<groupId>io.netty</groupId>
<artifactId>netty-all</artifactId>
</exclusion>
<exclusion>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
Expand Down