Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update ZooKeeper to 3.6.2 and Curator to 5.1.0 #8590

Merged
merged 29 commits into from
Feb 11, 2021
Merged

Update ZooKeeper to 3.6.2 and Curator to 5.1.0 #8590

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
add680a
Update ZooKeeper to 3.6.2 and Curator to 5.1.0
eolivelli Nov 17, 2020
47cbe33
port from #8348
eolivelli Nov 17, 2020
b33b9ba
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 14, 2021
60835b1
Exclude shaded ZooKeeper from spotbugs
eolivelli Jan 14, 2021
e30bf32
fix spotbugs
eolivelli Jan 14, 2021
d01491f
fix licenses and tests
eolivelli Jan 14, 2021
fe2f2db
fix more tests
eolivelli Jan 14, 2021
9e62f7c
More test deps
eolivelli Jan 15, 2021
5440699
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 18, 2021
d887b9e
Upgrade Curator in Flume connector
eolivelli Jan 19, 2021
39143a7
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 20, 2021
a90fe83
Fix refletion usage of ZK internals
eolivelli Jan 20, 2021
c8f571d
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 21, 2021
624441f
Make AdminApiOffloadTest less flaky
eolivelli Jan 21, 2021
58a5dff
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 22, 2021
a7220ec
Fix MockZooKeeper
eolivelli Jan 22, 2021
c3a9c2d
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Jan 25, 2021
d604fc3
Merge branch 'master' into fix/zk-3.6
eolivelli Jan 29, 2021
40e5275
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 1, 2021
dffbfb4
remove line
eolivelli Feb 1, 2021
930f84d
Fixed use of multi() in MockZookeeper
merlimat Feb 2, 2021
c2015a8
More ZK dependency fixes
merlimat Feb 2, 2021
bf3e784
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 2, 2021
822b1dc
Merge remote-tracking branch 'origin/master' into fix/zk-3.6
eolivelli Feb 5, 2021
8b2c460
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 8, 2021
957ea1e
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 9, 2021
10487e0
Fix license check
eolivelli Feb 9, 2021
379dac7
Merge branch 'master' into fix/zk-3.6
eolivelli Feb 10, 2021
a300cac
revert typo
eolivelli Feb 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions distribution/server/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,26 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-web</artifactId>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-graphite</artifactId>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-jvm</artifactId>
</dependency>

<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</dependency>

<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
Expand Down
10 changes: 6 additions & 4 deletions distribution/server/src/assemble/LICENSE.bin.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -494,9 +494,9 @@ The Apache Software License, Version 2.0
- joda-time-2.10.1.jar
- joda-time-joda-time-2.10.1.jar
* Dropwizard
- io.dropwizard.metrics-metrics-core-3.1.0.jar
- io.dropwizard.metrics-metrics-graphite-3.1.0.jar
- io.dropwizard.metrics-metrics-jvm-3.1.0.jar
- io.dropwizard.metrics-metrics-core-3.2.5.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bookkeeper uses Zookeeper 3.6.2 and dropwizard 3.1.0, I think it makes sense to keep dependency versions in sync to avoid surprises later

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlg99 I can't remember why I had to make this bump.
Probably it is the same version as in ZK.

I would prefer to upgrade this dep in BK 4.13.

- io.dropwizard.metrics-metrics-graphite-3.2.5.jar
- io.dropwizard.metrics-metrics-jvm-3.2.5.jar
* Prometheus
- io.prometheus-simpleclient_httpserver-0.5.0.jar
* Java JSON WebTokens
Expand All @@ -511,7 +511,9 @@ The Apache Software License, Version 2.0
- io.vertx-vertx-core-3.5.3.jar
- io.vertx-vertx-web-3.5.3.jar
* Apache ZooKeeper
- org.apache.zookeeper-zookeeper-jute-3.5.7.jar
- org.apache.zookeeper-zookeeper-jute-3.6.2.jar
* Snappy Java
- org.xerial.snappy-snappy-java-1.1.7.jar

BSD 3-clause "New" or "Revised" License
* Google auth library
Expand Down
11 changes: 10 additions & 1 deletion managed-ledger/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,16 @@
<classifier>tests</classifier>
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain why do we need this? I failed to see this dependency was used here. The same question apply to the other dependency.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because from zk 3.6 onwards the "zookeeper" Maven artifact does not import those dependencies anymore to the dependant projects.
This is because we (ZooKeeper project community) want to say that this artifact is only the "zookeeper java client" and the zookeeper java client does not need such jars.
But we still do not provide a "zookeeper server" jar

In Pulsar we are also running the ZooKeeper server (both for production and for tests) and so we need to explicitly add those two jar to the build, otherwise the ZooKeeper server won't start

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to leave a comment with the reason we're adding these, otherwise down the road it will be difficult to remember the rationale.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sijie PTAL the description was added on the new deps

<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>
<!--
junit is a runtime dependency of zookeeper tests, so only
add this dependency here.
Expand Down
43 changes: 41 additions & 2 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,10 @@ flexible messaging model and an intuitive client API.</description>
<commons-compress.version>1.19</commons-compress.version>

<bookkeeper.version>4.12.1</bookkeeper.version>
<zookeeper.version>3.5.7</zookeeper.version>
<zookeeper.version>3.6.2</zookeeper.version>
<snappy.version>1.1.7</snappy.version> <!-- ZooKeeper server -->
<dropwizardmetrics.version>3.2.5</dropwizardmetrics.version> <!-- ZooKeeper server -->
<curator.version>5.1.0</curator.version>
<netty.version>4.1.51.Final</netty.version>
<netty-tc-native.version>2.0.33.Final</netty-tc-native.version>
<jetty.version>9.4.35.v20201120</jetty.version>
Expand Down Expand Up @@ -296,7 +299,31 @@ flexible messaging model and an intuitive client API.</description>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper-jute</artifactId>
<version>${zookeeper.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>${dropwizardmetrics.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-graphite</artifactId>
<version>${dropwizardmetrics.version}</version>
</dependency>
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-jvm</artifactId>
<version>${dropwizardmetrics.version}</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>${snappy.version}</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
Expand Down Expand Up @@ -342,6 +369,18 @@ flexible messaging model and an intuitive client API.</description>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>${curator.version}</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper</groupId>
Expand Down
14 changes: 13 additions & 1 deletion pulsar-broker/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,13 +133,25 @@
<version>${project.version}</version>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-zookeeper-utils</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
</dependency>

<!-- functions related dependencies (begin) -->

Expand Down
5 changes: 4 additions & 1 deletion pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiOffloadTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@

/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
Expand Down Expand Up @@ -33,7 +34,6 @@
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;

import org.apache.bookkeeper.mledger.LedgerOffloader;
import org.apache.bookkeeper.mledger.ManagedLedgerInfo;
import org.apache.pulsar.broker.auth.MockedPulsarServiceBaseTest;
Expand Down Expand Up @@ -298,6 +298,9 @@ public void testOffload(boolean isPartitioned) throws Exception {
when(topicOffloader.getOffloadDriverName()).thenReturn("mock");
doReturn(topicOffloader).when(pulsar).createManagedLedgerOffloader(any());

Awaitility.await().atMost(3, TimeUnit.SECONDS)
.until(() -> pulsar.getTopicPoliciesService().cacheIsInitialized(TopicName.get(topicName)));

//4 set topic level offload policies
admin.topics().setOffloadPolicies(topicName, offloadPolicies);
Awaitility.await().atMost(10, TimeUnit.SECONDS).untilAsserted(()
Expand Down
21 changes: 20 additions & 1 deletion pulsar-io/flume/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,30 @@
<scope>test</scope>
<version>1.9.0</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-framework</artifactId>
<version>${curator.version}</version>
</dependency>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-test</artifactId>
<version>4.2.0</version>
<version>${curator.version}</version>
<scope>test</scope>
</dependency>
<!-- zookeeper server -->
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>
<!-- zookeeper server -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>com.github.stefanbirkner</groupId>
<artifactId>system-rules</artifactId>
Expand Down
1 change: 0 additions & 1 deletion .../src/main/java/org/apache/pulsar/io/flume/node/PollingZooKeeperConfigurationProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.cache.ChildData;
import org.apache.curator.framework.recipes.cache.NodeCache;
import org.apache.curator.framework.recipes.cache.NodeCacheListener;
import org.apache.flume.FlumeException;
import org.apache.flume.conf.FlumeConfiguration;
import org.apache.flume.lifecycle.LifecycleAware;
Expand Down
14 changes: 14 additions & 0 deletions pulsar-metadata/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,20 @@
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper</groupId>
Expand Down
14 changes: 14 additions & 0 deletions pulsar-package-management/bookkeeper-storage/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,20 @@
<scope>test</scope>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>

<!-- zookeeper server -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>managed-ledger</artifactId>
Expand Down
12 changes: 6 additions & 6 deletions pulsar-sql/presto-distribution/LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -397,9 +397,9 @@ The Apache Software License, Version 2.0
* Objectsize
- objectsize-0.0.12.jar
* Dropwizard Metrics
- metrics-core-3.1.0.jar
- metrics-graphite-3.1.0.jar
- metrics-jvm-3.1.0.jar
- metrics-core-3.2.5.jar
- metrics-graphite-3.2.5.jar
- metrics-jvm-3.2.5.jar
* Prometheus
- simpleclient-0.5.0.jar
- simpleclient_common-0.5.0.jar
Expand Down Expand Up @@ -435,7 +435,7 @@ The Apache Software License, Version 2.0
* GSON
- gson-2.8.6.jar
* Snappy
- snappy-java-1.1.7.3.jar
- snappy-java-1.1.7.jar
* Jackson
- jackson-module-parameter-names-2.10.0.jar
- jackson-module-parameter-names-2.11.1.jar
Expand All @@ -458,8 +458,8 @@ The Apache Software License, Version 2.0
- memory-0.8.3.jar
- sketches-core-0.8.3.jar
* Apache Zookeeper
- zookeeper-3.5.7.jar
- zookeeper-jute-3.5.7.jar
- zookeeper-3.6.2.jar
- zookeeper-jute-3.6.2.jar
* Apache Yetus Audience Annotations
- audience-annotations-0.5.0.jar
* Swagger
Expand Down
1 change: 0 additions & 1 deletion pulsar-testclient/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,6 @@
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>${zookeeper.version}</version>
</dependency>

<dependency>
Expand Down
21 changes: 21 additions & 0 deletions pulsar-zookeeper-utils/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,18 @@
<groupId>io.grpc</groupId>
<artifactId>grpc-all</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.curator</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
</dependency>

<!-- `grpc-all` is excluded from `stream-storage-server` at root pom file -->
<dependency>
<groupId>io.grpc</groupId>
Expand All @@ -78,6 +87,18 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>io.dropwizard.metrics</groupId>
<artifactId>metrics-core</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>testmocks</artifactId>
Expand Down
9 changes: 6 additions & 3 deletions ...ar-zookeeper-utils/src/main/java/org/apache/pulsar/zookeeper/LocalBookkeeperEnsemble.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;

import org.apache.bookkeeper.bookie.BookieException.InvalidCookieException;
Expand Down Expand Up @@ -204,7 +205,7 @@ private void runZookeeper(int maxCC) throws IOException {
serverFactory.configure(new InetSocketAddress(zkPort), maxCC);
serverFactory.startup(zks);

zkDataCleanupManager = new DatadirCleanupManager(zkDataDir, zkDataDir, 0, 1 /* hour */);
zkDataCleanupManager = new DatadirCleanupManager(zkDataDir, zkDataDir, 3, 1 /* hour */);
zkDataCleanupManager.start();
} catch (Exception e) {
LOG.error("Exception while instantiating ZooKeeper", e);
Expand All @@ -227,8 +228,10 @@ private void runZookeeper(int maxCC) throws IOException {
public void disconnectZookeeper(ZooKeeper zooKeeper) {
ServerCnxn serverCnxn = getZookeeperServerConnection(zooKeeper);
try {
Method method = serverCnxn.getClass().getMethod("close");
method.invoke(serverCnxn);
LOG.info("disconnect ZK server side connection {}", serverCnxn);
Class disconnectReasonClass = Class.forName("org.apache.zookeeper.server.ServerCnxn$DisconnectReason");
Method method = serverCnxn.getClass().getMethod("close", disconnectReasonClass);
method.invoke(serverCnxn, Stream.of(disconnectReasonClass.getEnumConstants()).filter(s->s.toString().equals("CONNECTION_CLOSE_FORCED")).findFirst().get());
} catch (Exception ex) {
throw new RuntimeException(ex);
}
Expand Down
Loading