Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Issue 9728] Upgrade GRPC and bookkeeper #9729

Merged
merged 9 commits into from
Mar 5, 2021
+133 −62
Merged

[Issue 9728] Upgrade GRPC and bookkeeper #9729

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
7cbc36b
Upgraded bookkeeper to 4.13.0; grpc to 1.33
dlg99 Feb 10, 2021
66f7393
updated LICENSE
dlg99 Feb 25, 2021
0fc6eac
fixed the rest of the license check
dlg99 Feb 26, 2021
5760089
no idea why license check results on CI are different from local :/
dlg99 Feb 26, 2021
1ba4bdd
one more license fix
dlg99 Feb 26, 2021
69e8d6f
Integration tests failed to start container/service there because of …
dlg99 Feb 26, 2021
a12ab19
another try to get the permark.jar in
dlg99 Feb 26, 2021
5fc4c94
another take on perfmark.jar and integration tests
dlg99 Mar 1, 2021
5178309
add perfmark.jar to LICENSE
dlg99 Mar 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions distribution/server/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -214,6 +214,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>compile</scope>
</dependency>

<!-- bookkeeper http server -->
<dependency>
<groupId>org.apache.bookkeeper.http</groupId>
108 changes: 61 additions & 47 deletions distribution/server/src/assemble/LICENSE.bin.txt
View file
Original file line number Diff line number Diff line change
@@ -320,8 +320,11 @@ The Apache Software License, Version 2.0
- com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.11.1.jar
- com.fasterxml.jackson.module-jackson-module-jsonSchema-2.11.1.jar
- org.codehaus.jackson-jackson-core-asl-1.9.11.jar
- org.codehaus.jackson-jackson-mapper-asl-1.9.11.jar
* Caffeine -- com.github.ben-manes.caffeine-caffeine-2.6.2.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.12.0.jar
* Conscrypt -- org.conscrypt-conscrypt-openjdk-uber-2.5.1.jar
* Proto Google Common Protos -- com.google.api.grpc-proto-google-common-protos-1.17.0.jar
* Joda -- org.joda-joda-convert-2.2.1.jar
* Bitbucket -- org.bitbucket.b_c-jose4j-0.7.2.jar
* Gson
@@ -395,32 +398,32 @@ The Apache Software License, Version 2.0
- org.apache.logging.log4j-log4j-1.2-api-2.14.0.jar
* Java Native Access JNA -- net.java.dev.jna-jna-4.2.0.jar
* BookKeeper
- org.apache.bookkeeper-bookkeeper-common-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-proto-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-server-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.12.1.jar
- org.apache.bookkeeper-circe-checksum-4.12.1.jar
- org.apache.bookkeeper-cpu-affinity-4.12.1.jar
- org.apache.bookkeeper-statelib-4.12.1.jar
- org.apache.bookkeeper-stream-storage-api-4.12.1.jar
- org.apache.bookkeeper-stream-storage-common-4.12.1.jar
- org.apache.bookkeeper-stream-storage-java-client-4.12.1.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.12.1.jar
- org.apache.bookkeeper-stream-storage-proto-4.12.1.jar
- org.apache.bookkeeper-stream-storage-server-4.12.1.jar
- org.apache.bookkeeper-stream-storage-service-api-4.12.1.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.12.1.jar
- org.apache.bookkeeper.http-http-server-4.12.1.jar
- org.apache.bookkeeper.http-vertx-http-server-4.12.1.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.12.1.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.12.1.jar
- org.apache.bookkeeper.tests-stream-storage-tests-common-4.12.1.jar
- org.apache.distributedlog-distributedlog-common-4.12.1.jar
- org.apache.distributedlog-distributedlog-core-4.12.1-tests.jar
- org.apache.distributedlog-distributedlog-core-4.12.1.jar
- org.apache.distributedlog-distributedlog-protocol-4.12.1.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.12.1.jar
- org.apache.bookkeeper-bookkeeper-common-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-common-allocator-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-proto-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-server-4.13.0.jar
- org.apache.bookkeeper-bookkeeper-tools-framework-4.13.0.jar
- org.apache.bookkeeper-circe-checksum-4.13.0.jar
- org.apache.bookkeeper-cpu-affinity-4.13.0.jar
- org.apache.bookkeeper-statelib-4.13.0.jar
- org.apache.bookkeeper-stream-storage-api-4.13.0.jar
- org.apache.bookkeeper-stream-storage-common-4.13.0.jar
- org.apache.bookkeeper-stream-storage-java-client-4.13.0.jar
- org.apache.bookkeeper-stream-storage-java-client-base-4.13.0.jar
- org.apache.bookkeeper-stream-storage-proto-4.13.0.jar
- org.apache.bookkeeper-stream-storage-server-4.13.0.jar
- org.apache.bookkeeper-stream-storage-service-api-4.13.0.jar
- org.apache.bookkeeper-stream-storage-service-impl-4.13.0.jar
- org.apache.bookkeeper.http-http-server-4.13.0.jar
- org.apache.bookkeeper.http-vertx-http-server-4.13.0.jar
- org.apache.bookkeeper.stats-bookkeeper-stats-api-4.13.0.jar
- org.apache.bookkeeper.stats-prometheus-metrics-provider-4.13.0.jar
- org.apache.bookkeeper.tests-stream-storage-tests-common-4.13.0.jar
- org.apache.distributedlog-distributedlog-common-4.13.0.jar
- org.apache.distributedlog-distributedlog-core-4.13.0-tests.jar
- org.apache.distributedlog-distributedlog-core-4.13.0.jar
- org.apache.distributedlog-distributedlog-protocol-4.13.0.jar
- org.apache.bookkeeper.stats-codahale-metrics-provider-4.13.0.jar
* Apache HTTP Client
- org.apache.httpcomponents-httpclient-4.5.5.jar
- org.apache.httpcomponents-httpcore-4.4.9.jar
@@ -452,24 +455,32 @@ The Apache Software License, Version 2.0
* Google Error Prone Annotations - com.google.errorprone-error_prone_annotations-2.3.4.jar
* Apache Thrifth - org.apache.thrift-libthrift-0.12.0.jar
* OkHttp
- com.squareup.okhttp-okhttp-2.5.0.jar
- com.squareup.okhttp-okhttp-2.7.4.jar
* Okio - com.squareup.okio-okio-1.13.0.jar
* Javassist -- org.javassist-javassist-3.25.0-GA.jar
* gRPC
- io.grpc-grpc-all-1.18.0.jar
- io.grpc-grpc-auth-1.18.0.jar
- io.grpc-grpc-context-1.18.0.jar
- io.grpc-grpc-core-1.18.0.jar
- io.grpc-grpc-netty-1.18.0.jar
- io.grpc-grpc-okhttp-1.18.0.jar
- io.grpc-grpc-protobuf-1.18.0.jar
- io.grpc-grpc-protobuf-lite-1.18.0.jar
- io.grpc-grpc-stub-1.18.0.jar
- io.grpc-grpc-protobuf-nano-1.18.0.jar
- io.grpc-grpc-testing-1.18.0.jar
* gRPC
- io.grpc-grpc-all-1.33.0.jar
- io.grpc-grpc-auth-1.33.0.jar
- io.grpc-grpc-context-1.33.0.jar
- io.grpc-grpc-core-1.33.0.jar
- io.grpc-grpc-netty-1.33.0.jar
- io.grpc-grpc-okhttp-1.33.0.jar
- io.grpc-grpc-protobuf-1.33.0.jar
- io.grpc-grpc-protobuf-lite-1.33.0.jar
- io.grpc-grpc-stub-1.33.0.jar
- io.grpc-grpc-testing-1.33.0.jar
- io.grpc-grpc-alts-1.33.0.jar
- io.grpc-grpc-api-1.33.0.jar
- io.grpc-grpc-grpclb-1.33.0.jar
- io.grpc-grpc-netty-shaded-1.33.0.jar
- io.grpc-grpc-services-1.33.0.jar
- io.grpc-grpc-xds-1.33.0.jar
* Perfmark
- io.perfmark-perfmark-api-0.19.0.jar
* OpenCensus
- io.opencensus-opencensus-api-0.18.0.jar
- io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar
- io.opencensus-opencensus-contrib-http-util-0.24.0.jar
- io.opencensus-opencensus-proto-0.2.0.jar
* Jodah
- net.jodah-typetools-0.5.0.jar
* Apache Avro
@@ -514,10 +525,16 @@ The Apache Software License, Version 2.0
- org.apache.zookeeper-zookeeper-jute-3.6.2.jar
* Snappy Java
- org.xerial.snappy-snappy-java-1.1.7.jar
* Google HTTP Client
- com.google.http-client-google-http-client-jackson2-1.34.0.jar
- com.google.http-client-google-http-client-1.34.0.jar
- com.google.auto.value-auto-value-annotations-1.7.jar
- com.google.re2j-re2j-1.2.jar

BSD 3-clause "New" or "Revised" License
* Google auth library
- com.google.auth-google-auth-library-credentials-0.9.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-credentials-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
- com.google.auth-google-auth-library-oauth2-http-0.20.0.jar -- licenses/LICENSE-google-auth-library.txt
* LevelDB -- (included in org.rocksdb.*.jar) -- licenses/LICENSE-LevelDB.txt
* JSR305 -- com.google.code.findbugs-jsr305-3.0.2.jar -- licenses/LICENSE-JSR305.txt
* JavaHamcrest -- org.hamcrest-hamcrest-core-1.3.jar -- licenses/LICENSE-Hamcrest.txt
@@ -531,17 +548,13 @@ MIT License
- org.slf4j-jul-to-slf4j-1.7.25.jar
- org.slf4j-slf4j-api-1.7.25.jar
- org.slf4j-jcl-over-slf4j-1.7.25.jar
* Animal Sniffer Annotations
- org.codehaus.mojo-animal-sniffer-annotations-1.17.jar
* The Checker Framework
- org.checkerframework-checker-compat-qual-2.5.2.jar
- org.checkerframework-checker-qual-3.5.0.jar

Protocol Buffers License
* Protocol Buffers
- com.google.protobuf-protobuf-java-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf-protobuf-java-util-3.11.4.jar -- licenses/LICENSE-protobuf.txt
- com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar -- licenses/LICENSE-protobuf.txt

CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Java Annotations API
@@ -571,7 +584,8 @@ CDDL-1.1 -- licenses/LICENSE-CDDL-1.1.txt
* Mimepull -- org.jvnet.mimepull-mimepull-1.9.13.jar

Eclipse Distribution License 1.0 -- licenses/LICENSE-EDL-1.0.txt
* Jakarta Activation -- jakarta.activation-jakarta.activation-api-1.2.1.jar
* Jakarta Activation
- jakarta.activation-jakarta.activation-api-1.2.1.jar
* Jakarta XML Binding -- jakarta.xml.bind-jakarta.xml.bind-api-2.3.2.jar

Eclipse Public License 1.0 -- licenses/LICENSE-AspectJ.txt
1 change: 0 additions & 1 deletion distribution/server/src/assemble/NOTICE.bin.txt
View file
Original file line number Diff line number Diff line change
@@ -201,7 +201,6 @@ http://gcc.gnu.org/onlinedocs/libstdc++/manual/license.html
* Performance and memory usage improvement

------------------------------------------------------------------------------------
- org.checkerframework-checker-compat-qual-2.5.2.jar

A few parts of the Checker Framework have more permissive licenses.

5 changes: 5 additions & 0 deletions docker/pulsar-all/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -38,6 +38,11 @@
<type>pom</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>pulsar-offloader-distribution</artifactId>
18 changes: 16 additions & 2 deletions pom.xml
View file
Original file line number Diff line number Diff line change
@@ -96,7 +96,7 @@ flexible messaging model and an intuitive client API.</description>
<!-- apache commons -->
<commons-compress.version>1.19</commons-compress.version>

<bookkeeper.version>4.12.1</bookkeeper.version>
<bookkeeper.version>4.13.0</bookkeeper.version>
<zookeeper.version>3.6.2</zookeeper.version>
<snappy.version>1.1.7</snappy.version> <!-- ZooKeeper server -->
<dropwizardmetrics.version>3.2.5</dropwizardmetrics.version> <!-- ZooKeeper server -->
@@ -124,7 +124,8 @@ flexible messaging model and an intuitive client API.</description>
<typetools.version>0.5.0</typetools.version>
<protobuf3.version>3.11.4</protobuf3.version>
<protoc3.version>${protobuf3.version}</protoc3.version>
<grpc.version>1.18.0</grpc.version>
<grpc.version>1.33.0</grpc.version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we jump to 1.36.0 ?

I known that in BK we stick to 1.33 but IIRC it is due to the etcd client. In Pulsar we do not have an etcd client.
As it will still take more time to test 2.8 before a release, we could upgrade to latest GRPC version now and have all of the time to see if it is working well with Pulsar

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@eolivelli Etcd client broke because grpc 1.36 depends on netty 1.52.
From what I understood, the culprit was was this change netty/netty#10407 in 4.1.52 that got reverted netty/netty#10980
The change got reverted because TLS_FALSE_START slightly changes the "flow" during handshake which may cause suprises for the end-user
We can try mixing netty 1.51 (or 1.59, where the change is supposedly reverted, it was not out when I tested bookkeeper) with grpc 1.36 but I am not sure how much I can rely on existing tests (and many of them don't pass currently anyways, with or without my changes). I'd very much prefer not to experiment with this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine to me

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be useful to switch to use gprc-bom also in Pulsar, in the similar way as it was done for Bookkeeper: https://github.com/apache/bookkeeper/blob/7327e3b1c636ed42bdbe231fc30bcb73e527fee8/pom.xml#L462-L468 .
gprc-bom became available in 1.25 and that's why it hasn't been used before.

<perfmark.version>0.19.0</perfmark.version>
<protoc-gen-grpc-java.version>${grpc.version}</protoc-gen-grpc-java.version>
<gson.version>2.8.6</gson.version>
<sketches.version>0.8.3</sketches.version>
@@ -838,6 +839,19 @@ flexible messaging model and an intuitive client API.</description>
<version>${grpc.version}</version>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<version>${perfmark.version}</version>
<scope>runtime</scope>
<exclusions>
<exclusion>
<artifactId>error_prone_annotations</artifactId>
<groupId>com.google.errorprone</groupId>
</exclusion>
</exclusions>
</dependency>

<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
1 change: 1 addition & 0 deletions pulsar-broker-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -82,6 +82,7 @@
<include>net.java.dev.jna:*</include>
<include>com.carrotsearch:*</include>
<include>io.prometheus:*</include>
<include>io.perfmark:*</include>
<include>com.github.ben-manes.caffeine:*</include>
<include>org.glassfish.jersey.*:*</include>
<include>org.rocksdb:*</include>
1 change: 1 addition & 0 deletions pulsar-client-admin-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -88,6 +88,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>com.squareup.*:*</include>
<include>com.google.*:*</include>
1 change: 1 addition & 0 deletions pulsar-client-all/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -133,6 +133,7 @@
<include>org.glassfish.hk2*:*</include>
<include>com.fasterxml.jackson.*:*</include>
<include>io.grpc:*</include>
<include>io.perfmark:*</include>
<include>com.yahoo.datasketches:*</include>
<include>io.netty:*</include>
<include>com.squareup.*:*</include>
1 change: 1 addition & 0 deletions pulsar-client-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -122,6 +122,7 @@
<include>com.fasterxml.jackson.core:jackson-core</include>
<include>com.fasterxml.jackson.dataformat</include>
<include>io.netty:*</include>
<include>io.perfmark:*</include>
<include>org.eclipse.jetty:*</include>
<include>com.yahoo.datasketches:*</include>
<include>commons-*:*</include>
6 changes: 6 additions & 0 deletions pulsar-functions/instance/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -115,6 +115,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper</groupId>
<artifactId>bookkeeper-common</artifactId>
4 changes: 4 additions & 0 deletions pulsar-functions/localrun-shaded/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -132,6 +132,10 @@
<pattern>io.grpc</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.io.grpc</shadedPattern>
</relocation>
<relocation>
<pattern>io.perfmark</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.io.perfmark</shadedPattern>
</relocation>
<relocation>
<pattern>org.joda</pattern>
<shadedPattern>org.apache.pulsar.functions.runtime.shaded.org.joda</shadedPattern>
7 changes: 7 additions & 0 deletions pulsar-functions/localrun/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -58,6 +58,13 @@
<groupId>io.grpc</groupId>
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

</dependencies>

<build>
6 changes: 6 additions & 0 deletions pulsar-functions/proto/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -61,6 +61,12 @@
</exclusions>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

</dependencies>
<build>
<plugins>
24 changes: 12 additions & 12 deletions pulsar-sql/presto-distribution/LICENSE
View file
Original file line number Diff line number Diff line change
@@ -412,18 +412,18 @@ The Apache Software License, Version 2.0
- async-http-client-2.12.1.jar
- async-http-client-netty-utils-2.12.1.jar
* Apache Bookkeeper
- bookkeeper-common-4.12.1.jar
- bookkeeper-common-allocator-4.12.1.jar
- bookkeeper-proto-4.12.1.jar
- bookkeeper-server-4.12.1.jar
- bookkeeper-stats-api-4.12.1.jar
- bookkeeper-tools-framework-4.12.1.jar
- circe-checksum-4.12.1.jar
- codahale-metrics-provider-4.12.1jar
- cpu-affinity-4.12.1.jar
- http-server-4.12.1.jar
- prometheus-metrics-provider-4.12.1.jar
- codahale-metrics-provider-4.12.1.jar
- bookkeeper-common-4.13.0.jar
- bookkeeper-common-allocator-4.13.0.jar
- bookkeeper-proto-4.13.0.jar
- bookkeeper-server-4.13.0.jar
- bookkeeper-stats-api-4.13.0.jar
- bookkeeper-tools-framework-4.13.0.jar
- circe-checksum-4.13.0.jar
- codahale-metrics-provider-4.13.0.jar
- cpu-affinity-4.13.0.jar
- http-server-4.13.0.jar
- prometheus-metrics-provider-4.13.0.jar
- codahale-metrics-provider-4.13.0.jar
* Apache Commons
- commons-cli-1.2.jar
- commons-codec-1.10.jar
6 changes: 6 additions & 0 deletions pulsar-zookeeper-utils/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -70,6 +70,12 @@
<artifactId>grpc-all</artifactId>
</dependency>

<dependency>
<groupId>io.perfmark</groupId>
<artifactId>perfmark-api</artifactId>
<scope>runtime</scope>
</dependency>

<dependency>
<groupId>org.apache.bookkeeper.stats</groupId>
<artifactId>prometheus-metrics-provider</artifactId>