Event callbacks

[edenhill]

Events:

• Topic state & leader change
• Broker connection up/down (with total counts of brokers up/down)
• ..?

Callback set with rd_kafka_event_cb_set(rk, ..).
Served by rd_kafka_poll()

[rthalley]

+1 on this. I want to be able to log state changes in topic+partition leadership, so I can say whether the topic+partition is "up" or "down".

[DEvil0000]

+1
I need to stop/pause consumers depending on producer state
consumer->producer

[edenhill]

@DEvil0000 Pause/Resume is now available on master branch.

[DEvil0000]

Thanks, is the state now also observable?

[edenhill]

No sir
Den 23 jan 2016 11:02 skrev "A. Binzxxxxxx" notifications@github.com:

Thanks, is the state now also observable?

—
Reply to this email directly or view it on GitHub
#137 (comment)
.

[rolandyoung]

+1 Some error states (such as misconfiguration of the broker URL) would cause our application log to be spammed rather enthusiastically with errors such as

FAIL: 127.0.0.1:10002/0: Connect to ipv4#127.0.0.1:10002 failed: Connection refused

To avoid this we use a heuristic approach: if we see an error, we set a state variable to 'in error' and increment an error count; if this reaches 10, we stop logging until we come out of the error state, which is the tricky bit. At the moment the best we have come up with is to watch the output queue (our application is a producer): if it has got shorter, or if it's empty and we have seen no new errors for a few seconds, we log that the error state is over and reset the error count. This mostly works, but is very occasionally prone to spurious flipping between states, especially if we are not producing any messages at the time.

We would find it very useful indeed to have a way of knowing whether the application is connected to at least one broker.

[edenhill]

Repeated errors for the same broker (and address!) should be suppressed, but if the broker resolves to multiple addresses or multiple address families (which localhost does on some OS:es, providing both ipv4 and ipv6 addresses and Kafka typically only listens to one family) there might be an endless ping-pong of logs since librdkafka round-robins all addresses for a broker.

We would find it very useful indeed to have a way of knowing whether the application is connected to at least one broker.

I'm guessing you want this to be push notifications (e.g., a callback or event served through ..poll()) rather than an API you need to call to check it, right?

If so, an event like thing, what would be triggering factors and what info should the event contain?
Trigger on: broker up & down
Info: The broker, number of brokers in up/down state?