Memory leak from Producer

[oranb]

Description

Experiencing memory leak on Kafka producer simple application.
Application starts from around 20MB and can get to even 8G of RAM after few days.

Producer queue is empty, and poll is being called on regular basis.

Using objgraph and heapy it seems that the root cause for the leak is not from the code who wraps confluent_kafka

delivery.report.only.error is not set to true as described in #84 as possible cause for memory leak.

How to reproduce

Memory is getting bigger gradually over time depending on the size of the data which being sent.

Checklist

Please provide the following information:

• confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 0.11.5 for confluent-kafka-python, 0.11.6 for librdkafka
• Apache Kafka broker version: 1.0.1
• Client configuration: { "acks": 1, "linger.ms": 100, "request.timeout.ms": 300000, "retries": 50 }
• Operating system: Ubuntu 14.04 trusty
• Provide client logs (with 'debug': '..' as necessary)
• Provide broker log excerpts
• Critical issue

[edenhill]

You are most likely hitting #525

[oranb]

Thanks for the quick response.

Does #525 cause leaks even if there are no errors? The leaks happening regardless of errors.

How can it be workaround before taking fix version?
I tried not using the delivery report callback, but memory leak persists.

[oranb]

Update - After farther checking it seems root cause was found.

The leak is due to message headers. Without adding headers to messages on produce(), there is no leak.

Seems related to #458 only it was fixed for 0.11.6, no? I'm using 0.11.6 and it still occurs.

[edenhill]

The headers memory leak shouldve been fixed in 0.11.6, unless there is another issue.

Can you provide a small reproducible program?

[edenhill]

@oranb Can you provide a small reproducible program?

[rnpridgeon]

Closing for now due to inactivity. Please reopen with a reproducible application if the issue continues with v1.1.0