Dependency issue with google-cloud-pubsub and Apache Beam packages

[chalcrow]

I get a Maven install conflict between the Google Cloud pub/sub artifact:

<dependency>
  <groupId>com.google.cloud</groupId>
  <artifactId>google-cloud-pubsub</artifactId>
  <version>1.53.0</version>
</dependency>

and the Apache Beam google-cloud-dataflow-java-sdk-all artifact:

    <dependency>
      <groupId>com.google.cloud.dataflow</groupId>
      <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
      <version>2.5.0</version>
    </dependency>

google-cloud-pubsub has various sub-dependencies that rely on io.grpc:grpc-core:jar:1.16.1, and subdependencies of google-cloud-dataflow-java-sdk-all depend on various versions of io.grpc:grpc-core:jar that do not match this.

The full Maven conflict error message I get is

Could not resolve version conflict among [com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0

I've documented my specific issue and attempts to fix it more fully here:

https://stackoverflow.com/questions/53424272/maven-conflict-in-java-app-with-google-cloud-core-grpc-dependency

Can someone please recommend a workaround? I need one urgently.

[sduskis]

We need to document how to use google-cloud-java libraries in Beam.

I suggest that you upgrade to beam 2.8.0 and use google-cloud-pubsub 1.49.0.

The core of the problem is that Beam and the google-cloud-library clients use different grpc versions. First, I would suggest upgrading to beam 2.8.0 which uses grpc 1.13.1. I did some digging, and found that google-cloud-pubsub 1.49.0 used grpc 1.13.1 as well.

[chalcrow]

Thanks Solomon Part of the issue I'm getting is that Apache Beam has a number of dependencies on different grpc versions (at least for v2.5.0 of the Beam dependency). I can match up the 1st grpc dependency but then there's a conflict with the next one. I can try changing the version numbers in my POM as youv'e suggested. v2.8.0 of the Beam artifact isn't listed on the Maven web page. I've raised the issue of inconsistent grpc dependencies in the Beam Jira. I don't know if it's a recognised issue that's been fixed in v.2.8.0

…

On Tue, 27 Nov. 2018, 03:41 Solomon Duskis ***@***.*** wrote: We need to document how to use google-cloud-java libraries in Beam. I suggest that you upgrade to beam 2.8.0 and use google-cloud-pubsub 1.49.0. The core of the problem is that Beam and the google-cloud-library clients use different grpc versions. First, I would suggest upgrading to beam 2.8.0 which uses grpc 1.13.1. I did some digging, and found that google-cloud-pubsub 1.49.0 used grpc 1.13.1 as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4095 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHsIUm6dMfiYPITU0IGRkyt7fncXC7jNks5uzBmrgaJpZM4YwBI4> .

[chalcrow]

Hi Solomon, this didn't work:

  <properties>
    <beam.version>2.8.0</beam.version>
    <maven-compiler-plugin.version>3.6.2</maven-compiler-plugin.version>
    <maven-exec-plugin.version>1.6.0</maven-exec-plugin.version>
  </properties>


  <build>
   <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>${maven-compiler-plugin.version}</version>
        <configuration>
          <source>1.8</source>
          <target>1.8</target>
        </configuration>
      </plugin>
    </plugins>

    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.codehaus.mojo</groupId>
          <artifactId>exec-maven-plugin</artifactId>
          <version>${maven-exec-plugin.version}</version>
          <configuration>
            <cleanupDaemonThreads>false</cleanupDaemonThreads>
          </configuration>
        </plugin>
        <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
        <plugin>
        	<groupId>org.eclipse.m2e</groupId>
        	<artifactId>lifecycle-mapping</artifactId>
        	<version>1.0.0</version>
        	<configuration>
        		<lifecycleMappingMetadata>
        			<pluginExecutions>
        				<pluginExecution>
        					<pluginExecutionFilter>
        						<groupId>
        							org.apache.maven.plugins
        						</groupId>
        						<artifactId>
        							maven-compiler-plugin
        						</artifactId>
        						<versionRange>
        							[@maven-compiler-plugin.version@,)
        						</versionRange>
        						<goals>
        							<goal>compile</goal>
        							<goal>testCompile</goal>
        						</goals>
        					</pluginExecutionFilter>
        					<action>
        						<ignore></ignore>
        					</action>
        				</pluginExecution>
        			</pluginExecutions>
        		</lifecycleMappingMetadata>
        	</configuration>
        </plugin>
      </plugins>
    </pluginManagement>
  </build>

  <dependencies>
    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-sdks-java-core</artifactId>
      <version>${beam.version}</version>
    </dependency>

  
    <dependency>
      <groupId>com.google.cloud.dataflow</groupId>
      <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
      <version>2.8.0</version>
    </dependency>
    
    <dependency>
      <groupId>org.apache.beam</groupId>
      <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
      <version>2.8.0</version>
    </dependency>
    
    <dependency>
  	  <groupId>com.google.cloud</groupId>
  	  <artifactId>google-cloud-pubsub</artifactId>
      <version>1.53.0</version>
    </dependency>


...

Could not resolve version conflict among [org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-core:jar:1.13.1, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-netty:jar:1.13.1 -> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> com.google.cloud:google-cloud-spanner:jar:0.54.0-beta -> io.opencensus:opencensus-contrib-grpc-util:jar:0.12.3 -> io.grpc:grpc-core:jar:1.9.0, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.4.0 -> io.grpc:grpc-core:jar:1.10.1, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-all:jar:1.13.1 -> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-all:jar:1.13.1 -> io.grpc:grpc-okhttp:jar:1.13.1 -> io.grpc:grpc-core:jar:[1.13.1,1.13.1], org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-all:jar:1.13.1 -> io.grpc:grpc-protobuf-nano:jar:1.13.1 -> io.grpc:grpc-core:jar:1.13.1, org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.8.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.8.0 -> io.grpc:grpc-all:jar:1.13.1 -> io.grpc:grpc-testing:jar:1.13.1 -> io.grpc:grpc-core:jar:[1.13.1,1.13.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.53.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-protobuf-lite:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.53.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1]] -> [Help 1]

[sduskis]

google-cloud-pubsub 1.49.0 uses the right version of grpc, so changing the google-cloud-pubsub version ought to help.

[KrishnapriyaMahajanJ]

I am facing the same issue. Any Update on this???

Could not resolve version conflict among [com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-netty:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 -> io.grpc:grpc-core:jar:1.5.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.grpc:grpc-core:jar:1.7.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 -> io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 -> io.grpc:grpc-core:jar:1.6.1, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-okhttp:jar:1.2.0 -> io.grpc:grpc-core:jar:[1.2.0,1.2.0], com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-lite:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0 -> io.grpc:grpc-all:jar:1.2.0 -> io.grpc:grpc-protobuf-nano:jar:1.2.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.5.0 -> org.apache.beam:beam-runners-core-construction-java:jar:2.5.0 -> org.apache.beam:beam-model-job-management:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud.dataflow:google-cloud-dataflow-java-sdk-all:jar:2.5.0 -> org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.5.0 -> org.apache.beam:beam-runners-core-construction-java:jar:2.5.0 -> io.grpc:grpc-core:jar:1.2.0, com.google.cloud:google-cloud-pubsub:jar:1.54.0 -> com.google.cloud:google-cloud-core-grpc:jar:1.54.0 -> io.grpc:grpc-protobuf:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.54.0 -> io.grpc:grpc-netty-shaded:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1], com.google.cloud:google-cloud-pubsub:jar:1.54.0 -> io.grpc:grpc-stub:jar:1.16.1 -> io.grpc:grpc-core:jar:1.16.1, com.google.cloud:google-cloud-pubsub:jar:1.54.0 -> io.grpc:grpc-auth:jar:1.16.1 -> io.grpc:grpc-core:jar:[1.16.1,1.16.1]]

[sduskis]

@KrishnapriyaJha, can you please upgrade to Beam 2.8.0?

[chalcrow]

@KrishnapriyaJha - there's a potential workaround here - https://stackoverflow.com/questions/53424272/maven-conflict-in-java-app-with-google-cloud-core-grpc-dependency/53558702#53558702.

Also I was able to integrate a Google PubSub subscription into the Beam framework so that I didn't need to get the Google library working with the Beam one. My answer here may be helpful:

https://stackoverflow.com/questions/53421932/update-singleton-hashmap-using-google-pub-sub/53549860#53549860

[KrishnapriyaMahajanJ]

@chalcrow Thanks It worked.

[KishoreNamala]

@chalcrow - I encountered this error and used the workaround mentioned in the above link. It resolved the compile error but broke my pubsub client while trying to publish to a topic.

Here is the error from my pubsub client (trying to create a publisher with custom batch settings).

Exception in thread "main" java.lang.NoSuchFieldError: transportTracerFactory
	at io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder.buildTransportFactory(NettyChannelBuilder.java:394)
	at io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:266)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createSingleChannel(InstantiatingGrpcChannelProvider.java:223)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.createChannel(InstantiatingGrpcChannelProvider.java:169)
	at com.google.api.gax.grpc.InstantiatingGrpcChannelProvider.getTransportChannel(InstantiatingGrpcChannelProvider.java:156)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:157)
	at com.google.cloud.pubsub.v1.stub.GrpcPublisherStub.create(GrpcPublisherStub.java:161)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:149)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:78)
	at com.google.cloud.pubsub.v1.Publisher$Builder.build(Publisher.java:614)
	at com.techm.data.service.PublisherSvc.getPublisherWithCustomBatchSettings(PublisherSvc.java:155)
	at com.techm.data.service.PublisherSvc.publishMessageBatchToATopic(PublisherSvc.java:116)
	at com.techm.data.client.CreateTopicAndPublishMessages.publishObjectMessages(CreateTopicAndPublishMessages.java:99)
	at com.techm.data.client.CreateTopicAndPublishMessages.main(CreateTopicAndPublishMessages.java:158)

And here is the method creating the publisher

	public Publisher getPublisherWithCustomBatchSettings(ProjectTopicName topicName) throws Exception {
		// [START pubsub_publisher_batch_settings]
		// Batch settings control how the publisher batches messages
		long requestBytesThreshold = 5000L; // default : 1 byte
		long messageCountBatchSize = 10L; // default : 1 message

		Duration publishDelayThreshold = Duration.ofMillis(100); // default : 1 ms

		// Publish request get triggered based on request size, messages count & time
		// since last publish
		BatchingSettings batchingSettings = BatchingSettings.newBuilder()
				.setElementCountThreshold(messageCountBatchSize).setRequestByteThreshold(requestBytesThreshold)
				.setDelayThreshold(publishDelayThreshold).build();

		Publisher publisher = Publisher.newBuilder(topicName).setBatchingSettings(batchingSettings).build();
		// [END pubsub_publisher_batch_settings]
		return publisher;
	}