Documentation on PyKafka vs kafka-python

[microamp]

Hello. I'd like to play around with Kafka, but I don't know which client to use to start with. I know there is at least one other Python client called kafka-python. I wonder if there is any documentation on comparison between the two. I'll start with PyKafka in the meantime. :)

[emmettbutler]

@microamp Thanks, this is a great idea. There's currently no documentation on this, but to my knowledge the main differences are the specifics of the Python API and PyKafka's implementation of the BalancedConsumer. PyKafka strives to keep the API as pythonic as possible, which means using useful features of the language where appropriate for client code simplicity. This includes things like context managers for object cleanup and futures for asynchronous error handling. PyKafka's balanced consumer implements the Kafka project's notion of the "high level consumer", which uses ZooKeeper to balance consumption of partitions between multiple nodes in a consumer group. From what I understand, kafka-python is waiting until Kafka 0.9, when this functionality will be supported natively by the Kafka server itself, to implement self-balancing consumers.
Also, the last time we did a speed test (which was admittedly a while ago at this point), PyKafka's consumer outperformed kafka-python. I unfortunately no longer have the results from that test, so you may not want to bet too hard on PyKafka being significantly faster or slower - just figured I'd mention it.

[emmettbutler]

Some more research - there are differences in the versions of python supported by each library. PyKafka supports 2.7, 3.4, 3.5, and pypy, while kafka-python adds 2.6 and removes 3.5 support. kafka-python also requires a ZooKeeper connection for offset management, which PyKafka does not. kafka-python supports versions of Kafka from 0.8.0 to 0.8.2, where PyKafka only supports 0.8.2.

[microamp]

@emmett9001

Thanks a lot for the reply. I find the information very helpful.

It's good to know that PyKafka supports Python 3.4+. It was still work in progress the last time I checked a few months back. Good work guys.

[ottomata]

A difference between kafka-python and pykafka is the producer interface. kafka-python does not require that you know the topic when instantiating the producer. This is convenient if you need to produce to topics dynamically based on input (which I do!) :)

[amontalenti]

@ottomata That seems like an interesting request for us to look at. Want to open a separate issue about that?

[ottomata]

Sure!

[cscheffler]

@emmett9001 @ottomata Just got pointed at this thread and thought I'd make a late contribution.

We compared pykafka and kafka-python about 2 months ago while trying to decide which one to use. In the end, the deciding factor for us was that balanced consumers were much easier to manage in pykafka.

Also, we discovered later, a pykafka producer doesn't die on Kafka broker restart, while our kafka-python producers did.

Below are performance figures from a 3-node Kafka cluster running in EC2, using a single producer or consumer. The three numbers for each test are the quartiles measured for the test.

• pykafka producer: 41400 – 46500 – 50200 messages per second
• pykafka consumer: 12100 – 14400 – 23700 messages per second
• kafka-python producer: 26500 – 27700 – 29500 messages per second
• kafka-python consumer: 35000 – 37300 – 39100 messages per second

So, for clarification, the median performance of a pykafka producer was 46500 messages per second, with a quartile range of 41400 (25th percentile) to 50200 (75th percentile). Hope that makes sense.

[emmettbutler]

This is awesome, thanks for the performance numbers @cscheffler. Do you have anything to share on the methodology you used to find them?