4.2.0 won't compile on AIX 7.1

[mbeddo]

zeromq 4.1.6 compiles on my AIX box and passes all tests. However, attempting to build zeromq 4.2.0 I find compile problems. Details follow:

$ xlC -qversion
IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
Version: 12.01.0000.0000
$ oslevel
7.1.0.0

$ configure CC=xlc CXX=xlC
$ make
Making all in doc
make[1]: Entering directory /home/meb/zeromq-4.2.0/doc' make[1]: Nothing to be done for all'.
make[1]: Leaving directory /home/meb/zeromq-4.2.0/doc' make[1]: Entering directory /home/meb/zeromq-4.2.0'
CXX src/src_libzmq_la-proxy.lo
"src/proxy.cpp", line 157.34: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t".
"src/proxy.cpp", line 188.22: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t".
"src/proxy.cpp", line 189.51: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t".
"src/proxy.cpp", line 197.22: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t".
"src/proxy.cpp", line 198.25: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t".
make[1]: *** [src/src_libzmq_la-proxy.lo] Error 1
make[1]: Leaving directory `/home/meb/zeromq-4.2.0'
make: *** [all-recursive] Error 1

[bluca]

Since I don't have access to an AIX system, would you be able to test this fix?

https://github.com/bluca/libzmq/tree/aix_poll

The include order was changed at some point, I suspect that's the issue.

[mbeddo]

I get as far as make[1]: Entering directory `/home/meb/libzmq-aix_poll' CXX src/src_libzmq_la-proxy.lo "src/proxy.cpp", line 159.34: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t". "src/proxy.cpp", line 190.22: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t". "src/proxy.cpp", line 191.51: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t". "src/proxy.cpp", line 199.22: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t". "src/proxy.cpp", line 200.25: 1540-0217 (S) "rtnevents" is not a member of "struct zmq_pollitem_t". make[1]: *** [src/src_libzmq_la-proxy.lo] Error 1 make[1]: Leaving directory `/home/meb/libzmq-aix_poll' make: *** [all-recursive] Error 1 So this is the same problem (but different line numbers). However, interestingly, if I export OBJECT_MODE=64 (to build 64 bit code) then the code compiles and the library is built, but ALL of the tests fail with the same error message, reproduced here for one test: Bad file number (src/signaler.cpp:162) /bin/sh: 1312030 IOT/Abort trap(coredump) FAIL: tests/test_system I think it might be useful to show the relevant contents of the AIX <sys/poll.h> header file? Here is the portion that might be a source of headaches: … #ifdef __64BIT__ /* pollfd.fd is always a 32 bit entity (XOPEN) */ struct pollfd { int fd; /* file descriptor or file ptr */ short events; /* requested events */ short revents; /* returned events */ }; #else /* __64BIT__ */ struct pollfd { long fd; /* file descriptor or file ptr */ ushort reqevents; /* requested events */ ushort rtnevents; /* returned events */ }; #ifndef __HIDE_SVR4_POLLFD_NAMES #define events reqevents /* SVR3,4 pollfd member name */ #define revents rtnevents /* SVR3,4 pollfd member name */ #endif /* __HIDE_SVR4_POLLFD_NAMES */ #endif /* __64BIT__ */ … From: Luca Boccassi [mailto:notifications@github.com] Sent: Thursday, December 01, 2016 4:41 PM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) Since I don't have access to an AIX system, would you be able to test this fix? https://github.com/bluca/libzmq/tree/aix_poll The include order was changed at some point, I suspect that's the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrNsy7OQfcgCC6wVNjDlmGG9qr42pks5rD1sPgaJpZM4LBvXc>.

[bluca]

Thanks, I've seen the snippet on the mailing list, that's definitely what's causing the problem.

Could you please try changing the very first include of poll.h in src/proxy.cpp and src/signaler.cpp from poll.h to sys/poll.h ?

[mbeddo]

Hi Luca There was no improvement when I made the changes. On my AIX box, /usr/include/poll.h only has one line: “#include <sys/poll.h>” I hacked the makefile to include the “-P” flag so I could see what the structures looked like after the preprocessor. The structure “pollfd” looks like this (32-bit compiling): struct pollfd { long fd; ushort reqevents; ushort rtnevents; }; Then, of course, there are the macros that define “events” to be “reqevents” and “revents” to be “rtnevents”. So why does the compiler complain that "rtnevents" is not a member of "struct zmq_pollitem_t"? Because it’s not! The definition of zmq_pollitem_t is typedef struct zmq_pollitem_t { void *socket; #if defined _WIN32 SOCKET fd; #else int fd; #endif short events; short revents; } zmq_pollitem_t; The types don’t match (long vs int, ushort vs short). I find that “struct pollfd” isn’t actually used in proxy.cpp, so maybe the problem is elsewhere. There must be another place that uses a “struct pollfd” to make AIX system calls and this is somehow mapped onto the zmq_pollitem_t structure which is used everywhere? I really hate AIX. Nothing’s ever easy. Strangely, when I compile the codes for 64 bit, the pollfd structure matches the int/short/short members of zmq_pollitem_t and everything builds fine with no warnings or errors, but 72 of 77 tests fail! From: Luca Boccassi [mailto:notifications@github.com] Sent: Friday, December 02, 2016 1:51 AM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) Thanks, I've seen the snippet on the mailing list, that's definitely what's causing the problem. Could you please try changing the very first include of poll.h in src/proxy.cpp and src/signaler.cpp from poll.h to sys/poll.h ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrPLuqHsxnciSVqsFJ9h9E09-Mt_5ks5rD9vXgaJpZM4LBvXc>.

[bluca]

Ah yes of course, the zmq_pollitem_t members won't match. Fixing it without breaking API and ABI is going to be interesting.

Perhaps adding a conditional #define at the bottom, something like:

typedef struct zmq_pollitem_t
{
    void *socket;
#if defined _WIN32
    SOCKET fd;
#else
    int fd;
#endif
    short events;
    short revents;
#if defined (ZMQ_HAVE_AIX)
#define reqevents events
#define rtnevents revents
#endif
} zmq_pollitem_t;

To make things even funnier they are different types, as the 64 bit ones are unsigned short instead of short...
Could you please try this?

Also it was added recently, specifically for AIX use, a different poller implementation: pollset
You might want to try that if it gets you going.

[mbeddo]

Code compiles and links, but only 5/77 tests passed. Snippet below: make check-TESTS make[2]: Entering directory `/home/meb/zeromq-4.2.0' make[3]: Entering directory `/home/meb/zeromq-4.2.0' PASS: tests/test_ancillaries ./config/test-driver[107]: 5047260 IOT/Abort trap(coredump) FAIL: tests/test_system ./config/test-driver[107]: 6882788 IOT/Abort trap(coredump) FAIL: tests/test_pair_inproc ./config/test-driver[107]: 6095392 IOT/Abort trap(coredump) FAIL: tests/test_pair_tcp ./config/test-driver[107]: 3015096 IOT/Abort trap(coredump) Not sure I follow your comment about AIX pollset, so not sure what to try next. From: Luca Boccassi [mailto:notifications@github.com] Sent: Friday, December 02, 2016 11:29 AM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) Ah yes of course, the zmq_pollitem_t members won't match. Fixing it without breaking API and ABI is going to be interesting. Perhaps adding a conditional #define at the bottom, something like: typedef struct zmq_pollitem_t { void *socket; #if defined _WIN32 SOCKET fd; #else int fd; #endif short events; short revents; #if defined (ZMQ_HAVE_AIX) #define reqevents events #define rtnevents revents #endif } zmq_pollitem_t; To make things even funnier they are different types, as the 64 bit ones are unsigned short instead of short... Could you please try this? Also it was added recently, specifically for AIX use, a different poller implementation: pollset You might want to try that if it gets you going. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrKVollTzwbNqTpNB9JqWHOoTTbLyks5rEGN4gaJpZM4LBvXc>.

[bluca]

Is that with the patches or with the pollset?

[mbeddo]

Last week I downloaded “aix_poll” from https://github.com/bluca/libzmq/tree/aix_poll. That didn’t work. I then tried your idea: Perhaps adding a conditional #define at the bottom, something like: typedef struct zmq_pollitem_t { void *socket; #if defined _WIN32 SOCKET fd; #else int fd; #endif short events; short revents; #if defined (ZMQ_HAVE_AIX) #define reqevents events #define rtnevents revents #endif } zmq_pollitem_t; And the code compiles and links, but 72/77 tests fail. That’s what I’m reporting on. I don’t understand what is meant by “with the pollset.” From: Luca Boccassi [mailto:notifications@github.com] Sent: Monday, December 05, 2016 10:55 AM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) Is that with the patches or with the pollset? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrNlIdYy5CoQesw5es86ze8GG1x90ks5rFFAIgaJpZM4LBvXc>.

[bluca]

A short while ago an AIX-specific poll implementation was added, so I was wondering if that would help you. You can select it with:

./configure --with-poller=pollset

[mbeddo]

With libzmq-aix_poll, with the new defines in the zmq_pollitem_t structure, and with pollset: $ configure CC=xlc CXX=xlC --with-poller=pollset $ make clean $ make $ make check … make[2]: Leaving directory `/home/meb/libzmq-aix_poll' make check-TESTS make[2]: Entering directory `/home/meb/libzmq-aix_poll' PASS: tests/test_ancillaries Bad file number (src/signaler.cpp:162) /bin/sh: 6621048 IOT/Abort trap(coredump) FAIL: tests/test_system Bad file number (src/signaler.cpp:162) /bin/sh: 2229000 IOT/Abort trap(coredump) FAIL: tests/test_pair_inproc Bad file number (src/signaler.cpp:162) /bin/sh: 6226650 IOT/Abort trap(coredump) FAIL: tests/test_pair_tcp … Result is the same – 72 of 77 tests failed. From: Luca Boccassi [mailto:notifications@github.com] Sent: Monday, December 05, 2016 11:15 AM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) A short while ago an AIX-specific poll implementation was added, so I was wondering if that would help you. You can select it with: ./configure --with-poller=pollset — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrOPmpA0-QdhPdQIiR92ohg5v30jTks5rFFSugaJpZM4LBvXc>.

[bluca]

Could you please run one of the tests manually to see what's the precise error?

LD_LIBRARY_PATH=src/.libs tests/.libs/test_pair_tcp

(This works on Linux, I hope it works on AIX too!)

Also, could you please do one more try with this branch and see if it still compiles? If it does I'll open a PR so that we can make progress. Once it at least builds then we can continue with the test case failures.

https://github.com/bluca/libzmq/tree/aix_poll

[mbeddo]

Using the source code I’ve been working with you on, I run the test and this is what I got. $ LD_LIBRARY_PATH=src/.libs tests/.libs/test_pair_tcp Bad file number (src/signaler.cpp:162) IOT/Abort trap(coredump) $ I next tried the aix_poll branch you pointed me to. At first, configure complained about missing libsodium, so I reran with $ configure CC=xlc CXX=xlC –without-libsodium The library made with a single warning: CXX src/src_libzmq_la-socket_base.lo "src/socket_base.cpp", line 162.26: 1540-2411 (W) A dynamic cast is present, but the correct RTTI option is not specified. I tried a “make check” but got this: CXX tests/test_stream_exceeds_buffer.o "tests/test_stream_exceeds_buffer.cpp", line 16.30: 1540-0274 (S) The name lookup for "AF_INET" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 16.39: 1540-0274 (S) The name lookup for "SOCK_STREAM" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 19.38: 1540-0274 (S) The name lookup for "SOL_SOCKET" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 19.50: 1540-0274 (S) The name lookup for "SO_REUSEADDR" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 28.10: 1540-0274 (S) The name lookup for "bind" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 30.10: 1540-0274 (S) The name lookup for "listen" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 40.23: 1540-0274 (S) The name lookup for "accept" did not find a declaration. "tests/test_stream_exceeds_buffer.cpp", line 46.10: 1540-0274 (S) The name lookup for "send" did not find a declaration. make[2]: *** [tests/test_stream_exceeds_buffer.o] Error 1 make[2]: Leaving directory `/home/meb/libzmq' make[1]: *** [check-am] Error 2 make[1]: Leaving directory `/home/meb/libzmq' make: *** [check-recursive] Error 1 But since the “make check” built test_pair_tcp I decided to try that: $ LD_LIBRARY_PATH=src/.libs tests/.libs/test_pair_tcp $ So it seemed to work (?). From: Luca Boccassi [mailto:notifications@github.com] Sent: Monday, December 05, 2016 4:04 PM To: zeromq/libzmq Cc: Mike Beddo; Author Subject: Re: [zeromq/libzmq] 4.2.0 won't compile on AIX 7.1 (#2234) Could you please run one of the tests manually to see what's the precise error? LD_LIBRARY_PATH=src/.libs tests/.libs/test_pair_tcp (This works on Linux, I hope it works on AIX too!) Also, could you please do one more try with this branch and see if it still compiles? If it does I'll open a PR so that we can make progress. Once it at least builds then we can continue with the test case failures. https://github.com/bluca/libzmq/tree/aix_poll — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2234 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AUEOrG7lTecFTr5rHOfxzv3BwxiP7Scjks5rFJhMgaJpZM4LBvXc>.

[bluca]

Ok, let's take it one bit at a time. Since the last version of the branch gives you a build and a test pass I'll start with that.

Once it's merged, we can go on.

Thanks for the assistance!