Skip to content

SiftScience/python-kafka-replayer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Python Kafka Replayer

https://circleci.com/gh/SiftScience/python-kafka-replayer/tree/master.svg?style=svg

kafka_replayer is a library that helps consume time ranges of messages from Kafka topics. While the standard Kafka consumer API allows seeking to a specific offset and replaying from there, using offsets as the replay abstraction is cumbersome and potentially error-prone. This library does the translation from timestamps to offsets transparently.

This library is written in Python, and leverages kafka-python's consumer to poll Kafka for messages.

Installing

$ pip install kafka_replayer

Using

import json
import kafka_replayer

des_fn = lambda x: json.loads(x) if x else None
replayer = kafka_replayer.KafkaReplayer('my-topic',
                                        bootstrap_servers=['localhost:9092'],
                                        key_deserializer=des_fn,
                                        value_deserializer=des_fn)

# Replay all records between the start and end millis timestamps
for record in replayer.replay(1469467314341, 1469467907549):
    print record

Documentation

http://pythonhosted.org/kafka_replayer/

License

See LICENSE.

About

Replays a time range of a Kafka topic

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages