SSL handshake error gets swallowed

[naartjie]

Description

This issue might be with the way librdkafka does the error propagation, and the issue might need to be logged there. I am logging it here, because I encountered it while using this library.

It seems when there is an error in the SSL handshake connecting to the broker, the client code doesn't get an error, and looks like it's silently swallowed. You can see it in the logs by setting debug in config:

 'debug' : 'security,broker'

The logs show something like this:

sasl_ssl://xyz.gcp.confluent.cloud:9092/bootstrap: failed: err: Local: SSL error: (errno: Undefined error: 0)

(Here is another issue which appears to have the same problem: #697)

How to reproduce

Set the SSL CA manually, to one which does not match the CA of the broker. I used this one to test, since it doesn't match Confluent Cloud's one, it gets us a repro:

  'ssl.ca.location': '/dir/invalid-ca.pem',

The error won't be reported in client code - the client just sits there waiting, but it will be visible in librdkafka debug logs.

Checklist

• confluent-kafka-python: 1.2.0
• librdkafka version: 1.2.0
• Apache Kafka broker version: Confluent Cloud
• Client configuration:

{
    'bootstrap.servers': 'xyz.gcp.confluent.cloud:9092',
    'sasl.mechanisms': 'PLAIN',
    'security.protocol': 'SASL_SSL',
    'sasl.username': '${USERNAME}',
    'sasl.password': '${PASSWORD}'
}

• Operating system: OSX 10.14.6

[colinhicks]

Hi @naartjie,

Supplying error_cb to the producer or consumer instance should provide a way to handle connection errors like you describe.

For example, if I provide a similar producer configuration as above, plus a sample error_cb:

def error_cb(err):
    print(err.str())

 p = Producer({
    'bootstrap.servers': 'pkc-______.us-east1.gcp.confluent.cloud:9092',
    'sasl.mechanisms': 'PLAIN',    
    'security.protocol': 'SASL_SSL',
    'sasl.username': '<>',
    'sasl.password': '<>',
    'ssl.ca.location': 'invalid.pem',
}, error_cb=error_cb)

Then, when I flush the producer instance, the callback prints respective errors:

sasl_ssl://pkc-______.us-east1.gcp.confluent.cloud:9092/bootstrap: SSL handshake failed: s3_clnt.c:1269: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:  (after 126ms in state CONNECT)

[naartjie]

Thanks @colinhicks, that does the trick.

I'll make a PR to update the examples, as that's what tripped me up.

[edenhill]

The Python wrapper should not set error_cb unless it is defined by the application, since setting the error_cb in librdkafka will silent the broker error logs (in favour of the error callback)

[rnpridgeon]

#745