Memory leak connecting a gazillion times to a pubsub

[skinkie]

I am currently using a small suite of software that uses zeromq to distribute realtime data.

https://github.com/StichtingOpenGeo/universal/blob/master/universal-pubsub.c

After some data downtime, we noticed that the pubsub's sucked up memory. Our clients typically reconnect every 60s if no data was received to overcome other network issues. I created a small test tool to figure out if there might be an issue with ZeroMQ.

https://github.com/StichtingOpenGeo/universal/blob/master/universal-sub-test.c

This shows up in ZeroMQ thus it makes me wonder: when should some destroys fly in?

==31474== 1,180,296 bytes in 1,521 blocks are possibly lost in loss record 45 of 47
==31474==    at 0x4C2A790: operator new(unsigned long, std::nothrow_t const&) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==31474==    by 0x4E60B98: zmq::session_base_t::create(zmq::io_thread_t*, bool, zmq::socket_base_t*, zmq::options_t const&, zmq::address_t const*) (in /usr/lib64/libzmq.so.3.1.0)
==31474==    by 0x4E7042E: zmq::tcp_listener_t::in_event() (in /usr/lib64/libzmq.so.3.1.0)
==31474==    by 0x4E4EEED: zmq::epoll_t::loop() (in /usr/lib64/libzmq.so.3.1.0)
==31474==    by 0x4E70B89: thread_routine (in /usr/lib64/libzmq.so.3.1.0)
==31474==    by 0x588D313: start_thread (in /lib64/libpthread-2.20.so)
==31474==    by 0x517843C: clone (in /lib64/libc-2.20.so)

[hintjens]

Quite separately, if you're working in C I'd strongly recommend using CZMQ,
it will make your life much easier.

On Tue, Nov 11, 2014 at 3:30 AM, Stefan de Konink notifications@github.com
wrote:

I am currently using a small suite of software that uses zeromq to
distribute realtime data.

https://github.com/StichtingOpenGeo/universal/blob/master/universal-pubsub.c

After some data downtime, we noticed that the pubsub's sucked up memory.
Our clients typically reconnect every 60s if no data was received to
overcome other network issues. I created a small test tool to figure out if
there might me an issue with ZeroMQ.

https://github.com/StichtingOpenGeo/universal/blob/master/universal-sub-test.c

This shows up in ZeroMQ thus it makes me wonder: when should some destroys
fly in?

==31474== 1,180,296 bytes in 1,521 blocks are possibly lost in loss record 45 of 47
==31474== at 0x4C2A790: operator new(unsigned long, std::nothrow_t const&) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==31474== by 0x4E60B98: zmq::session_base_t::create(zmq::io_thread_t_, bool, zmq::socket_base_t_, zmq::options_t const&, zmq::address_t const*) (in /usr/lib64/libzmq.so.3.1.0)
==31474== by 0x4E7042E: zmq::tcp_listener_t::in_event() (in /usr/lib64/libzmq.so.3.1.0)
==31474== by 0x4E4EEED: zmq::epoll_t::loop() (in /usr/lib64/libzmq.so.3.1.0)
==31474== by 0x4E70B89: thread_routine (in /usr/lib64/libzmq.so.3.1.0)
==31474== by 0x588D313: start_thread (in /lib64/libpthread-2.20.so)
==31474== by 0x517843C: clone (in /lib64/libc-2.20.so)

—
Reply to this email directly or view it on GitHub
#1256.

[skinkie]

Quite separately, if you're working in C I'd strongly recommend using CZMQ,
it will make your life much easier.

Sadly not the life of my users, getting the right dependencies on CZMQ
distributions is hell.

[hintjens]

It is? Surely it's just one additional library..
On Nov 11, 2014 2:07 PM, "Stefan de Konink" notifications@github.com
wrote:

Quite separately, if you're working in C I'd strongly recommend using
CZMQ,
it will make your life much easier.

Sadly not the life of my users, getting the right dependencies on CZMQ
distributions is hell.

—
Reply to this email directly or view it on GitHub
#1256 (comment).

[skinkie]

Yes it is. Try to find it on some "stable" binary distribution such as Debian or Red Hat. Anyway not quite the bikeshed I want to get into here. There is a memoryleak inside libzmq, unrelated to my programming skills.

[hintjens]

Ah, it usually works better with github, indeed.

Sorry for bikeshedding. The code without CZMQ is just harder to
understand... actually the test case is opaque and I'm not sure what it's
supposed to be showing. You're looping on closing/opening sockets, and this
will always create lots of TCP timewait sockets which will take a while to
leave the system. Where is the memory leak exactly?

On Tue, Nov 11, 2014 at 5:48 PM, Stefan de Konink notifications@github.com
wrote:

Yes it is. Try to find it on some "stable" binary distribution such as
Debian or Red Hat. Anyway not quite the bikeshed I want to get into here.
There is a memoryleak inside libzmq, unrelated to my programming skills.

—
Reply to this email directly or view it on GitHub
#1256 (comment).

[skinkie]

Where is the memory leak exactly?

Please Pieter, the bug is not difficult to understand. The pubsub is
leaking memory for each connect.

[hintjens]

As far as I can understand it you're creating and destroying sub sockets. Here is a minimal CZMQ program that does this:

#include <czmq.h>
int main (void)
{
    zsock_t *pub = zsock_new_pub ("tcp:127.0.0.1:5678");
    int busy = 100;
    while (busy--) {
        zsock_t *sub = zsock_new_sub ("tcp:127.0.0.1:5678", "");
        zclock_sleep (10);
        zsock_destroy (&sub);
    }
    zsock_destroy (&pub);
    return 0;
}

When I run this under valgrind, it reports:

==30032==
==30032== HEAP SUMMARY:
==30032==     in use at exit: 0 bytes in 0 blocks
==30032==   total heap usage: 443 allocs, 443 frees, 140,997 bytes allocated
==30032==
==30032== All heap blocks were freed -- no leaks are possible
==30032==
==30032== For counts of detected and suppressed errors, rerun with: -v
==30032== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)

So I'm left asking what it is you're doing different. Your code is not self-evident, or else I'm stupid. Anyhow my example works, it is certainly leaking data for each connect, afaict.

[skinkie]

The above example shows that even with a source distribution such as gentoo which installs czmq-2.2.0 the example doesn't compile. So while I know for a fact that czmq makes the life of the programmer easy - I have used it - in practise it is too unstable to share code.

test.c: In function ‘main’:
test.c:5:5: error: unknown type name ‘zsock_t’
     zsock_t *pub = zsock_new_pub ("tcp:127.0.0.1:5678");
     ^
test.c:5:20: warning: initialization makes pointer from integer without a cast
     zsock_t *pub = zsock_new_pub ("tcp:127.0.0.1:5678");
                    ^
test.c:8:9: error: unknown type name ‘zsock_t’
         zsock_t *sub = zsock_new_sub ("tcp:127.0.0.1:5678", "");
         ^
test.c:8:24: warning: initialization makes pointer from integer without a cast
         zsock_t *sub = zsock_new_sub ("tcp:127.0.0.1:5678", "");

Indeed, the github czmq version works. And in your example I am unable to find a memory leak. I am currently trying to spot the difference.

[skinkie]

Adding this will get you the first two memory leaks:

#include <czmq.h>

int main (void)
{
    zsock_t *pub = zsock_new_pub ("tcp://127.0.0.1:5678");
    zsock_set_sndhwm (pub, 8192);
    zsock_set_rcvhwm (pub, 8192);

    int busy = 100;
    while (busy--) {
        zsock_t *sub = zsock_new_sub ("tcp://127.0.0.1:5678", "");
        zclock_sleep (10);
        zsock_destroy (&sub);
    }
    zsock_destroy (&pub);
    return 0;
}

I won't doubt that czmq is doing a good job in memory management. But do observe the following examples:

#include <czmq.h>

int main (void) {
    zsock_t *pubsub   = zsock_new_pub  ("tcp://127.0.0.1:1234");

    int busy = 100;
    while (busy--) {
        sleep(1);
    }

    zsock_destroy (&pubsub);
    return 0;
}

#include <czmq.h>

int main (void)
{
    int busy = 100;
    while (busy--) {
        zsock_t *sub = zsock_new_sub ("tcp://127.0.0.1:1234", "");
        zclock_sleep (10);
        zsock_destroy (&sub);
    }
    return 0;
}

Using the czmq testers "there is no memory leak" after the application close. But look at what the application memory use is after the second tester finished. The memory is increasing and is only freed because of zsock_destroy (&pubsub).

[hintjens]

Sorry, there was a bug in my example, should have been tcp:// instead of
tcp. It asserts in zsock_set_sndhwm() otherwise (pub is NULL), no memory
leaks though.

So you're saying that the pub socket allocates memory for each sub
connection, and this memory is not freed until the pub socket is destroyed.
It's possible the pub socket doesn't see the TCP connection closing, yes.

On Wed, Nov 12, 2014 at 12:14 AM, Stefan de Konink <notifications@github.com

wrote:

Adding this will get you the first two memory leaks:

#include <czmq.h>
int main (void)
{
zsock_t *pub = zsock_new_pub ("tcp:127.0.0.1:5678");
zsock_set_sndhwm (pub, 8192);
zsock_set_rcvhwm (pub, 8192);

int busy = 100;
while (busy--) {
    zsock_t *sub = zsock_new_sub ("tcp:127.0.0.1:5678", "");
    zclock_sleep (10);
    zsock_destroy (&sub);
}
zsock_destroy (&pub);
return 0;

}

I won't doubt that czmq is doing a good job in memory management. But do
observe the following examples:

#include <czmq.h>
int main (void) {
zsock_t *pubsub = zsock_new_pub ("tcp://127.0.0.1:1234");

int busy = 100;
while (busy--) {
    sleep(1);
}

zsock_destroy (&pubsub);
return 0;

}

#include <czmq.h>
int main (void)
{
int busy = 100;
while (busy--) {
zsock_t *sub = zsock_new_sub ("tcp://127.0.0.1:1234", "");
zclock_sleep (10);
zsock_destroy (&sub);
}
return 0;
}

Using the czmq testers "there is no memory leak" after the application
close. But look at what the application memory use is after the second
tester finished. The memory is increasing and is only freed because of
zsock_destroy (&pubsub).

—
Reply to this email directly or view it on GitHub
#1256 (comment).

[skinkie]

I noticed the problem with tcp:// while splitting up. Even in your example you can observe that the memory usage is increasing. So even in your own example, the question is still there: why isn't the memory being released?

skinkie  21426  5.0  0.9 239120 77304 pts/1    Sl+  00:35   0:03 ./test

[hintjens]

I've no idea how this works internally in libzmq. All I'm concerned with
here is reproducing the test to a point where others know what is going on.
You may want to continue this discussion on zeromq-dev.

On Wed, Nov 12, 2014 at 12:37 AM, Stefan de Konink <notifications@github.com

wrote:

I noticed the problem with tcp:// while splitting up. Even in your example
you can see observe that the memory is increasing. So even in your own
example, the question is still there: why isn't the memory being released?

skinkie 21426 5.0 0.9 239120 77304 pts/1 Sl+ 00:35 0:03 ./test

—
Reply to this email directly or view it on GitHub
#1256 (comment).

[skinkie]

I guess it is quite related to the topic "PUSH does not seem to release PULL connection message memory".

[hintjens]

Sounds like the same thing, yes.

On Wed, Nov 12, 2014 at 12:56 AM, Stefan de Konink <notifications@github.com

wrote:

I guess it is quite related to the topic "PUSH does not seem to release
PULL connection message memory".

—
Reply to this email directly or view it on GitHub
#1256 (comment).

[greroger]

Digging into the push/pull leak I found that the pipe's are never destroyed, they are just marked as inactive and moved to the back of the list. They are still tracked and destroyed with the context, but every time a socket is reconnected there is a new set of pipes.

session_base_t::process_term() is called and it calls pipe_t::terminate()
The pipe_t term command is sent and received, the delimiter is sent but is never received so it has one sub pipe waiting on the delimiter and the other waiting on the term ack. It then stays in this state until the context is destroyed.