OMPI interfering with rand/random series

[rhc54]

Per mail to devel, from @matcabral

Doing quick test with rand()/srand() I found that MPI_Init() seems to be calling a function in their family that is affecting the values in the user application. Please see below my simple test and the results. Yes, moving the second call to srand() after MPI_init() solves the problem. However, I’m confused since this was supposedly addressed in version 1.7.5. From release notes:

1.7.5 20 Mar 2014:

• OMPI now uses its own internal random number generator and will not perturb srand() and friends.

I tested on OMPI 1.10.2 and 1.10.3. The result is deterministic.

Any ideas?

Thanks,
Regards,

#include <stdio.h>
#include <stdlib.h>
#include <mpi.h>
int main(int argc, char *argv[])
{
    int rand1;
    int rand2;
  int name_len;
    srand(100000);
    rand1 = rand();
    srand(100000);
    MPI_Init(&argc, &argv);
    rand2 = rand();
    if (rand1 != rand2) {
        printf("%d != %d\n", rand1, rand2);
        fflush(stdout);
    }
    else {
        printf("%d == %d\n", rand1, rand2);
        fflush(stdout);
    }
    MPI_Finalize();
    return 0;
}

host1:/tmp> mpirun -np 1 -host host1 -mca pml ob1 -mca btl tcp,self ./rand1
964940668 != 865007240

[rhc54]

Doing a quick scan thru the code, I see several places that violate the "leave rand/random alone" rules:

ompi/mca/io/romio314/romio/adio/common/shfp_fname.c
25:   is rank 0's process id and xxxx is a random number. If the
42:        srand(time(NULL));
43:        i = rand();


mpi/mca/io/romio314/romio/test-internal/heap_test.c
41:void fill_random_test(test_params_t *params);
123:    fill_random_test(&test);
179:    "   -r <size>  Create a random test and verify of size <size>\n"
217:void fill_random_test(test_params_t *params) {
224:    srand(seed);
235:    params->offsets[i] = (rand() % max_key);


ompi/mca/topo/treematch/treematch/tm_malloc.c
74:  srandom(0);
77:    extra_data[i] = (char) random() % 256;

opal/mca/event/libevent2022/libevent/poll.c
183:    i = random() % nfds;

opal/mca/event/libevent2022/libevent/select.c
180:    i = random() % nfds;

opal/mca/event/libevent2022/libevent/evutil.c
2103:_evutil_weakrand(void)
2106:   return rand();
2108:   return random();

Not sure what we can do about these as they are all from downstream libraries. Thoughts?

[ggouaillardet]

At first glance, this requirement does look wrong to me.
any third party library can potentially use the random number generator, so if someone wants a deterministic behavior, he/she should change his/her app and use srand_r() instead of srand().

Open MPI can be updated, but that would not work when built with and external libevent that uses random() and friends.

anyway, here is a proof-of-concept of what can be done for the internal libevent, i guess similar stuff can be done for romio and treematch

diff --git a/opal/mca/event/libevent2022/configure.m4 b/opal/mca/event/libevent2022/configure.m4
index 5ace9dc..42fe0d9 100644
--- a/opal/mca/event/libevent2022/configure.m4
+++ b/opal/mca/event/libevent2022/configure.m4
@@ -159,8 +159,9 @@ AC_DEFUN([MCA_opal_event_libevent2022_CONFIG],[

     AC_MSG_RESULT([$event_args])

+    CPPFLAGS="$CPPFLAGS -Drandom=opal_random"
     OPAL_CONFIG_SUBDIR([$libevent_basedir/libevent],
-        [$event_args $opal_subdir_args],
+        [$event_args $opal_subdir_args 'CPPFLAGS=$CPPFLAGS'],
         [libevent_happy="yes"], [libevent_happy="no"])
     if test "$libevent_happy" = "no"; then
         AC_MSG_WARN([Event library failed to configure])
diff --git a/opal/util/alfg.c b/opal/util/alfg.c
index fc13f90..6a2a3cf 100644
--- a/opal/util/alfg.c
+++ b/opal/util/alfg.c
@@ -8,6 +8,8 @@
  * $HEADER$
  */

+#include <string.h>
+
 #include "opal_config.h"

 #include "alfg.h"
@@ -52,6 +54,8 @@ static uint32_t galois(unsigned int *seed){
     return lsb;
 }

+static opal_rng_buff_t alfg_buffer;
+
 /**
  * @brief   Routine to seed the ALFG register
  *
@@ -80,6 +84,7 @@ int opal_srand(opal_rng_buff_t *buff, uint32_t seed) {
             buff->alfg[j] = buff->alfg[j] ^ ((galois(&seed_cpy))<<i);
         }
     }
+    memcpy(&alfg_buffer, buff, sizeof(alfg_buffer));

     return 1;

@@ -114,4 +119,9 @@ uint32_t opal_rand(opal_rng_buff_t *buff){

 }

-
+long int opal_random() {
+    uint32_t r1, r2;
+    r1 = opal_rand(&alfg_buffer);
+    r2 = opal_rand(&alfg_buffer);
+    return ((long int)r1 >> 32) | (long int)r2;
+}
diff --git a/opal/util/alfg.h b/opal/util/alfg.h
index 08718d8..6dc1292 100644
--- a/opal/util/alfg.h
+++ b/opal/util/alfg.h
@@ -32,4 +32,6 @@ OPAL_DECLSPEC int opal_srand(opal_rng_buff_t *buff, uint32_t seed);

 OPAL_DECLSPEC uint32_t opal_rand(opal_rng_buff_t *buff);

+OPAL_DECLSPEC long int opal_random();
+
t #endif /* OPAL_ALFG_H */

[jsquyres]

We have a customer who is running into this exact issue.

@ggouaillardet Your POC is nearly correct -- @thananon is going to submit a PR shortly. One question: why do you have opal_random() return long int? rand(3) is supposed to return int.

[ggouaillardet]

I might have seen long int somewhere, but I cannot find it any more, so let's use int !

[jsquyres]

@bosilca Parmetis+treematch: awesome; thanks.

[jsquyres]

Just to cross-reference: Ralph filed this issue upstream with libevent: libevent/libevent#384

[matcabral]

what about the use of srand() in romio ?

[rhc54]

Looks like it is just in the one place (shp_fname.c) as the other is an internal test. Might try the same fix as for libevent?