MPI execution : How to ?

[nyue]

Hi,

Is there some documentation of how I should supply command line option to run ospray_studio in MPI mode ?

I wish to run some benchmark on MPI rendering.

Cheers

[BruceCherniak]

OSPRay Studio passes command line arguments directly on to OSPRay for this. Take a look at https://www.ospray.org/documentation.html#mpi-offload-rendering.

The syntax will vary a little depending on your setup and the network fabric you connect nodes with.

[nyue]

Right. I will give that a go. Thank you.

[nyue]

Most OS comes with either MPICH or OpenMPI implementation of MPI

Which would be closer to the syntax in the documentation ?

Which MPI implementation was the OSPray Studio binaries built with ?

Cheers

[BruceCherniak]

I'm not as familiar with the OSPRay pre-built binaries. Please look here https://www.ospray.org/downloads.html. However, you may need to build your own, depending on your config.

As I recall, OSPRay is compatible with most MPI implementations (OpenMPI, MPICH, Intel® MPI, MVAPICH) but needs to be compiled against the target version. I'm not the expert here, but will try to get you a better answer.

[johguenther]

Most MPI implementations are binary compatible, thus OSPRay binaries work with Intel MPI, MPICH or OpenMPI, no re-compilation needed.

Re commandline: those should also very similar, the only potential difference I'm aware of is mpiexec vs. mpirun (and some special options are unique to the MPI implementation, but common arguments like -n should work identically). Did you run into issues?

[Twinklebear]

I think OpenMPI is the one that sticks out as not binary compatible, we build the OSPRay MPI releases against MPICH which means they'll be binary compatible with Intel MPI, MPICH, MVAPICH, and many others (any that are MPICH ABI compatible).

To run Studio with the MPI offload device I think this should work:

mpirun -n 3 ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload

Or you can use the separate app/worker launch mode:

mpirun -n 1 ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload : -n 2 <path to ospray binaries>/ospray_mpi_worker

[nyue]

Ah! Noted regarding OpenMPI

With the second option

mpirun -n 1 ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload : -n 2 <path to ospray binaries>/ospray_mpi_worker

I am going to try this out on an AWS parallel cluster I spin up

Will I be able to specify hosts name too ?

e.g.

mpirun -n 1 --hosts headnode ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload : -n 2 --hosts compute0, compute1 <path to ospray binaries>/ospray_mpi_worker

[Twinklebear]

Yeah with both single launch and split launch you can pass a list of host names. The host names are a global option, so you'd pass just one list of host names and the processes pick the names in order. So for example it'd be like:

mpirun -hosts headnode,compute0,compute1 \
    -n 1 ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload : \
    -n 2 <path to ospray binaries>/ospray_mpi_worker

Then ospStudio will run on the headnode, and ospray_mpi_worker will run on the compute0 and compute1 nodes. You may also want to explicitly pass the global option -ppn 1 to specify that only one process should be launched per node:

mpirun -hosts headnode,compute0,compute1 -ppn 1 \
    -n 1 ./ospStudio <args> --osp:load-modules=mpi --osp:device=mpiOffload : \
    -n 2 <path to ospray binaries>/ospray_mpi_worker

We do also support OpenMPI, but you'd have to build from source (easiest route there is via the superbuild)

[nyue]

With OSPray, it worked

/opt/intel/mpi/2021.4.0/bin/mpirun -hosts ip-172-31-25-178,queue0-dy-queue0-t2medium-1,queue0-dy-queue0-t2medium-2,queue0-dy-queue0-t2medium-3,queue0-dy-queue0-t2medium-4 -ppn 1 -n 1 /shared/ospray-2.9.0.x86_64.linux/bin/ospExamples --osp:load-modules=mpi --osp:device=mpiOffload : -n 4 /shared/ospray-2.9.0.x86_64.linux/bin/ospray_mpi_worker

With OSPray Studio, I get "OSPray Studio not responding" pop up dialog from the Ubuntu 20.04 desktop

/opt/intel/mpi/2021.4.0/bin/mpirun -hosts ip-172-31-25-178,queue0-dy-queue0-t2medium-1,queue0-dy-queue0-t2medium-2,queue0-dy-queue0-t2medium-3,queue0-dy-queue0-t2medium-4 -ppn 1 -n 1 /shared/ospray_studio-0.10.0-Linux/bin/ospStudio --osp:load-modules=mpi --osp:device=mpiOffload : -n 4 /shared/ospray-2.9.0.x86_64.linux/bin/ospray_mpi_worker

[BruceCherniak]

Thanks. I'll take a look.

[BruceCherniak]

We found the issue here. Resolution will be a commit on the OSPRay side. Stay tuned.