Skip to content

Conversation

@lhotari
Copy link
Member

@lhotari lhotari commented Jun 16, 2021

Fixes #11 #12

Motivation

GitHub Actions build is not up-to-date for pulsar-adapters.

Modifications

  • add GitHub Actions workflows for running unit tests and integration tests separately.
  • Integration tests require org.apache.pulsar.tests:integration:jar:tests:2.8.0 library that is not published in maven central
    • checkout apache/pulsar @ v2.8.0 and install library to maven local
  • Integration tests require apachepulsar/pulsar-test-latest-version:latest which is not up-to-date in Docker Hub
    • build the image locally based on apachepulsar/pulsar-all:2.8.0

Open issues

  • There are some failing tests
    • KafkaProducerSimpleConsumerTest fails in CI
    • A few integration tests fail with error messages Consumer connect fail! consumer state:Connecting and Cannot seek on a partition where we are not subscribed

@lhotari lhotari self-assigned this Jun 16, 2021
@lhotari lhotari added the enhancement New feature or request label Jun 16, 2021
@lhotari
Copy link
Member Author

lhotari commented Jun 16, 2021

Some details of integration test failures:

Error:  Tests run: 26, Failures: 4, Errors: 0, Skipped: 0, Time elapsed: 256.701 s <<< FAILURE! - in TestSuite
Error:  testConsumerSeek(org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest)  Time elapsed: 4.226 s  <<< FAILURE!
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$ConnectException: Consumer connect fail! consumer state:Connecting
	at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:461)
	at org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest.testConsumerSeek(KafkaApiTest.java:507)
Caused by: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$ConnectException: Consumer connect fail! consumer state:Connecting
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
	at org.apache.kafka.clients.consumer.KafkaConsumer.commitSync(KafkaConsumer.java:459)
	... 34 more
Caused by: org.apache.pulsar.client.api.PulsarClientException$ConnectException: Consumer connect fail! consumer state:Connecting
	at org.apache.pulsar.client.impl.PersistentAcknowledgmentsGroupingTracker.doImmediateAck(PersistentAcknowledgmentsGroupingTracker.java:418)
	at org.apache.pulsar.client.impl.PersistentAcknowledgmentsGroupingTracker.doCumulativeAck(PersistentAcknowledgmentsGroupingTracker.java:328)
	at org.apache.pulsar.client.impl.PersistentAcknowledgmentsGroupingTracker.addAcknowledgment(PersistentAcknowledgmentsGroupingTracker.java:233)
	at org.apache.pulsar.client.impl.ConsumerImpl.doAcknowledge(ConsumerImpl.java:519)
	at org.apache.pulsar.client.impl.ConsumerBase.doAcknowledgeWithTxn(ConsumerBase.java:543)
	at org.apache.pulsar.client.impl.ConsumerBase.acknowledgeCumulativeAsync(ConsumerBase.java:503)
	at org.apache.pulsar.client.impl.ConsumerBase.acknowledgeCumulativeAsync(ConsumerBase.java:488)
	at org.apache.kafka.clients.consumer.KafkaConsumer.lambda$doCommitOffsets$9(KafkaConsumer.java:513)
	at java.base/java.util.HashMap.forEach(HashMap.java:1336)
	at org.apache.kafka.clients.consumer.KafkaConsumer.doCommitOffsets(KafkaConsumer.java:510)
	... 35 more

Error:  testCustomRouter(org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest)  Time elapsed: 6.247 s  <<< FAILURE!
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:629)
	at org.apache.kafka.clients.consumer.KafkaConsumer.resetOffsets(KafkaConsumer.java:651)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:384)
	at org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest.testCustomRouter(KafkaApiTest.java:443)
Caused by: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:622)
	... 36 more

Error:  testExplicitPartitions(org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest)  Time elapsed: 5.49 s  <<< FAILURE!
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:629)
	at org.apache.kafka.clients.consumer.KafkaConsumer.resetOffsets(KafkaConsumer.java:651)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:384)
	at org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest.testExplicitPartitions(KafkaApiTest.java:370)
Caused by: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:622)
	... 36 more

Error:  testPartitions(org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest)  Time elapsed: 3.611 s  <<< FAILURE!
java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:629)
	at org.apache.pulsar.tests.integration.compat.kafka.KafkaApiTest.testPartitions(KafkaApiTest.java:314)
Caused by: java.lang.IllegalArgumentException: Cannot seek on a partition where we are not subscribed
	at org.apache.kafka.clients.consumer.KafkaConsumer.seekToEnd(KafkaConsumer.java:622)
	... 38 more

[INFO] 
[INFO] Results:
[INFO] 
Error:  Failures: 
Error:    KafkaApiTest.testConsumerSeek:507 » Runtime java.util.concurrent.ExecutionExce...
Error:    KafkaApiTest.testCustomRouter:443 » Completion java.lang.IllegalArgumentExcept...
Error:    KafkaApiTest.testExplicitPartitions:370 » Completion java.lang.IllegalArgument...
Error:    KafkaApiTest.testPartitions:314->lambda$testPartitions$4:318 » Completion java...

Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

let's merge this patch and then fix the tests.

with this patch we will have CI feedback

@codelipenghui ?

@eolivelli
Copy link
Contributor

@codelipenghui we should merge this patch ASAP, in order to prevent problems like #21

@eolivelli eolivelli merged commit 74c0dd8 into apache:master Jun 17, 2021
@lhotari lhotari mentioned this pull request Jun 18, 2021
@eolivelli eolivelli added this to the 2.8.0 milestone Jun 26, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI does not run on pull requests

3 participants