Which thread does decompression happen in?

[bitemyapp]

Hey y'all, I'm testing throughput with my Rust application which is using rdkafka.

I'm asking here as well as librdkafka in case the answer is different for a StreamConsumer or MessageStream: what thread does the decompression work? Is it the application thread that calls consumer.recv().await/msg_stream.next().await like the Java client or something else? I'm hitting a wall at 500 MB/sec post-decomp w/ ~18 cores available and that feels wrong to me.

I attempted, strenuously, to find an answer to this question yesterday and was only able to find the answer for the Java client in a mailing list thread, Neha Narkhede said it was the application thread. I've tried reading the code but it's hard for me to follow the C code in librdkafka.

[bitemyapp]

Got an answer from Confluent. Decompression happens in the broker thread, in the fetcher queue. You have a thread per broker per consumer. I don't think you should be limited by zstd decompression ordinarily but my use-case is targeting relatively high throughput.