Is there a way where I can get batch of messages instead of a single message.
What I'm trying to achieve is to get batch of messages and convert them to Python dataframe using from_json, and then do some operations in the dataframe and store the parquet files to s3.
The problem here is I can't just wait make the dataframe from only few messages. It has to be atleast 2L messages.
For this I can't iterate over 2L times, and poll the consume to get one by one message and store it and then process further.
Instead if there is a way where we can tell the consumer poll to return next 100 messages, it will be good. So it should return 100 messages if 100 messages are available or the n number of messages that are available.
I think this is general use case where most of the time we don't process message by message but a batch of messages.
Is there a way where I can get batch of messages instead of a single message.
What I'm trying to achieve is to get batch of messages and convert them to Python dataframe using from_json, and then do some operations in the dataframe and store the parquet files to s3.
The problem here is I can't just wait make the dataframe from only few messages. It has to be atleast 2L messages.
For this I can't iterate over 2L times, and poll the consume to get one by one message and store it and then process further.
Instead if there is a way where we can tell the consumer poll to return next 100 messages, it will be good. So it should return 100 messages if 100 messages are available or the n number of messages that are available.
I think this is general use case where most of the time we don't process message by message but a batch of messages.