Description
I'm seeing an error trying to run a simple OMPI job on a 2 node cluster where one node is a PPC64 BE byte order and the other is a
X86_64 LE byte order node. OMPI 1.8.4 is configured with --enable-heterogeneous:
./configure --with-openib=/usr CC=gcc CXX=g++ F77=gfortran FC=gfortran
--enable-mpirun-prefix-by-default --prefix=/usr/mpi/gcc/openmpi-1.8.4/
--with-openib-libdir=/usr/lib64/ --libdir=/usr/mpi/gcc/openmpi-1.8.4/lib64/
--with-contrib-vt-flags=--disable-iotrace --enable-mpi-thread-multiple
--with-threads=posix --enable-heterogeneous && make -j8 && make -j8 install
And the job started this way:
/usr/mpi/gcc/openmpi-1.8.4/bin/mpirun -np 2 -host
ppc64,atlas3 --allow-run-as-root --mca btl_openib_addr_include 102.1.1.0/24
--mca btl openib,sm,self /usr/mpi/gcc/openmpi-1.8.4/tests/IMB-3.2/IMB-MPI1
pingpong
But we see the following error. Note atlas3 is using the vendor ID that is in the wrong byte order (0x25140000 instead of 0x1425):
The Open MPI receive queue configuration for the OpenFabrics devices
on two nodes are incompatible, meaning that MPI processes on two
specific nodes were unable to communicate with each other. This
generally happens when you are using OpenFabrics devices from
different vendors on the same network. You should be able to use the
mca_btl_openib_receive_queues MCA parameter to set a uniform receive
queue configuration for all the devices in the MPI job, and therefore
be able to run successfully.
Local host: ppc64-rhel71
Local adapter: cxgb4_0 (vendor 0x1425, part ID 21505)
Local queues: P,65536,64
Remote host: atlas3
Remote adapter: (vendor 0x25140000, part ID 22282240)
Remote queues:
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64