Problem: zmq_ctx_term asserts with connect-before-bind and sockets with identity over inproc transport

[sjlnk]

Following program crashes on my machine (x86_64 linux, commit 50e277b).

#include <zmq.h>
#include <error.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

void check(int rc)
{
    rc = rc;
    if (rc < 0)
    {
        fprintf(stderr, "%s:%d ERROR: %u\n",
                __FILE__, __LINE__, errno);
        exit(1);
    }
}

int test_sock(const char* addr, int sock_type, int connect, int n)
{
    int i;
    int rc;
    // typically fails very quickly, less than 20 iterations
    for (i = 0; i < n; i++)
    {
        printf("\r%d", i);
        fflush(stdout);
        void *ctx = zmq_ctx_new();
        assert(ctx);

        void *sock = zmq_socket(ctx, sock_type);
        assert(sock);

        if (connect) {
            check(zmq_connect(sock, addr));
        } else {
            check(zmq_bind(sock, addr));
        }

        check(zmq_close(sock));
        check(zmq_ctx_term(ctx));
    }
    printf("\n");
}

int main(void)
{
    test_sock("inproc://testing", ZMQ_REP, 1, 1000);
    test_sock("inproc://testing", ZMQ_ROUTER, 1, 1000);
    return 0;
}

Error message:
Assertion failed: written (/home/seb/src/libzmq/src/ctx.cpp:583)

What is the function of that assertion? Would it be okay to let it silently fail here, especially when there is nobody receiving the message. Also, smells like a race condition because not every iteration causes a crash...

[sjlnk]

I have modified tests/test_ctx_destroy.cpp to detect this bug. I'd make a PR but head @4039858754a713b4faa6b0d347cb6e81216f99e0 is failing to build. That's why I'm using an older commit. What is the standard procedure in situation like this?

[bluca]

From the docs: https://github.com/zeromq/libzmq/blob/master/doc/zmq_connect.txt#L41

The first exception is when using the inproc:// transport: you must call _zmq_bind()_ before calling _zmq_connect()_.

Also when quickly iterating I would recommend not creating and deleting the context. Use the same one.

[sjlnk]

According to http://zeromq.org/docs:changes-4-0-0 this behavior has been modified some years ago. Unless it has been modified back to the original behavior then I should be able to connect inproc:// transport before binding. Indeed other socket types work, only REP and ROUTER are problematic here.

My guess is that those sockets have options.recv_identity set to true and others have not.

I realize that creating and destroying contexts like this is not a good way to use this library but it shouldn't cause a crash.

[bluca]

Oh sorry, I guess the docs need updating

[sjlnk]

Is the assertion that triggers this crash really necessary?

[bluca]

Yes, as it's sending the ID in the pipe.

If you change it to use the same context, does it still happen in the same way? What about if you bind first?

[sjlnk]

This example works. My guess is that sockets that have recv_identity=true don't support connect-before-bind functionality over inproc:// properly. When there is something suitable to connect to then there is no problem.

#include <zmq.h>
#include <error.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

void check(int rc)
{
    rc = rc;
    if (rc < 0)
    {
        fprintf(stderr, "%s:%d ERROR: %u\n",
                __FILE__, __LINE__, errno);
        exit(1);
    }
}

int main(void)
{
    int i;
    int rc;
    const char* addr = "inproc://testing";

    for (i = 0; i < 100000; i++)
    {
        printf("\r%d", i);
        fflush(stdout);

        void *ctx = zmq_ctx_new();
        assert(ctx);

        void *rep = zmq_socket(ctx, ZMQ_REP);
        void *req = zmq_socket(ctx, ZMQ_REQ);
        assert(rep);
        assert(req);

        check(zmq_bind(req, addr)); // should crash when commented out
        check(zmq_connect(rep, addr));

        check(zmq_close(rep));
        check(zmq_close(req));
        check(zmq_ctx_term(ctx));
    }
    printf("\n");

    return 0;
}

[bluca]

It does seem the case. Off the top of my head I wouldn't know if that's expected or simply no one has coded it up yet. If you have a solution PRs are very welcome of course :-)

[sjlnk]

I have not studied the internals of libzmq that much. It would take some time for me to get a hang of it. If I get the motivation, sure I'll have a look.

[bluca]

I think I found the issue, PR coming soon.

Basically the assert happens because zmq_ctx_term tries to resolve all pending inproc connections before closing, to avoid hanging. In that moment, the connect will have been closed but the pipes are still pending. But the first thing the inproc connect does, with a router socket, is to send the identity. But given the socket is closed, the pipe is in a state that doesn't accept messages, and thus the write fails.

My fix is to check if the socket is closed before sending the identity. I added a test that reproduced the problem, and it doesn't happen anymore.

Also one note about your test case: if you use the same inproc endpoint for many connects, make sure the individual binds are in the same order, otherwise you won't know what's connected to what.