Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

While using eachBatch(), the consumer isn't processing messages from each batch and just calling multiple batches #1700

Open
anupamd2-zluri opened this issue Jun 27, 2024 · 1 comment

Comments

@anupamd2-zluri
Copy link

Kafka Setup

2 Broker URLs
1 Topic with 4 Partitions - workflow-executions
1 Consumer Group (raw-data-consumer) with 4 Consumers (Kubernetes PODs)

Currently we are using eachMessage() to process the messages from the Brokers but we are not able to achieve the required throughput.

We are doing a POC of using eachBatch() to process the messages in batches.

While using eachBatch(), we found out that the messages from each batch isn't being processed but the handler function keeps on calling the eachBatch(). Is this the expected behaviour?

image image image
@shardul-cometchat
Copy link

I am facing a similar issue for eachMessage and the only solution I could find till now is setting partitionsConsumedConcurrently to 1 based on this: #1492

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants