Kafka Custom Serializer Examples with JSON, Smile and Kryo

This example shows how to implement a few different Kafka serialisers (new in Kafka API 0.9.0.0) that can be used to (de)serialize a Java object from/to a Kafka queue. I've written a blog post which you can find here that goes more deeply into explaining how this works.

Building

You can create a runnable jar with Maven: just run mvn clean install and you should find a runnable jar in the target directory.

Running

The main Example class can be run as either a producer (produce option) or consumer (consume option). You also need to specify the format (string,json,smile,kryo) and the Kafka topic used. Optionally you can also specify a Kafka URL.

Command line options:

The command line has 3 mandatory options (produce/consume, the serializer and the Kafka topic) and 1 option option (Kafka host:port, default is Kafka default localhost:9092).

Serializer	Description
string	StringReaderSerializer: uses basic spring concatenation / splitting
json	JacksonReaderSerializer: uses the jackson library to serialize to JSON
smile	JacksonReaderSerializer: uses the jackson library to serialize to Smile (binary JSON)
kryo	KryoReaderSerializer: uses the kryo library

Examples

java -jar target/kafka-serializer-1.0-SNAPSHOT-jar-with-dependencies.jar produce string test-string Run a producer with StringReaderSerializer writing to the test-string topic

java -jar target/kafka-serializer-1.0-SNAPSHOT-jar-with-dependencies.jar consume json test-json localhost:9100 Run a consumer with JacksonReaderSerializer reading from the test-json topic connecting to Kafka on port 9100