add error_cb to confluent example

[naartjie]

Hi. I thought it might be handy to add the error_cb to the confluent example, in case someone else gets tripped up by errors getting swallowed up, like in #722.

[ghost]

It looks like @naartjie hasn't signed our Contributor License Agreement, yet.

The purpose of a CLA is to ensure that the guardian of a project's outputs has the necessary ownership or grants of rights over all contributions to allow them to distribute under the chosen licence.
Wikipedia

You can read and sign our full Contributor License Agreement here.

Once you've signed reply with [clabot:check] to prove it.

Appreciation of efforts,

clabot

[naartjie]

[clabot:check]

[ghost]

@confluentinc It looks like @naartjie just signed our Contributor License Agreement. 👍

Always at your service,

clabot

[mhowlett]

good idea! even better, it would be good to add the following note to the callback method so that people understand better what the error events are for (this creates a lot of confusion).

# called to expose error events e.g. connection failures or all brokers down that don't correspond
# to any specific API function call. Note that the client will try to automatically recover from errors that
# are not marked as fatal. Non-fatal errors events should be interpreted as informational rather 
# than catastrophic.

[mhowlett]

(but stick to 80 char wide limit - i don't think the above does)

[edenhill]

There is a bit of librdkafka nuance here:
if an error_cb is not configured any broker/connection errors will be logged by default,
but if an error_cb is configured the connection errors will be enqueued on the event queue that is served by poll() and that will eventually trigger the callback.
The problem, as shown in the original issue, is that while there is no error_cb provided by the python script, the confluent-kafka-python bindings still set up an internal default error_cb that simply ignores the errors if no py error_cb was set.
So while I think that adding error_cbs is a good addition to the examples, we should perhaps not register the C error_cb if no Py error_cb is set, allowing the default logger to emit the connection failures.
But then there is the case with fatal errors that needs to be handled by the C error_cb regardless if a Py error_cb was set... see https://github.com/confluentinc/confluent-kafka-python/blob/master/confluent_kafka/src/confluent_kafka.c#L1224

Another problem is that if you register an error_cb and call something blocking that is not poll(), such as metadata(), you will not get any error logs (because error_cb is set) and you wont get any error_cbs because you are not calling poll().

[mhowlett]

this is such a complicated arrangement to have exposed to the end user - no one is going to use this correctly - we need to simplify it.

[rnpridgeon]

It may look a read a little easier if we set the error cb in the conf dict itself.

i.e. https://github.com/confluentinc/confluent-kafka-python/blob/master/tests/integration/integration_test.py#L369

[edenhill]

I think we should close this PR and fix so that the Producer does not set the C error_cb unless a Py error_cb is specified. This way librdkafka will log broker failures by default, and it wont require the application to provide an error_cb, nor poll() to have it triggered.

[rnpridgeon]

#745