PML UCX error

[jonathandenman]

Hello,

Currently I am trying to run a simulation using OpenFoam-5.x in parallel on skylake, haswell, and EPYC128 nodes. I am currently using openmpi/4.1.4 module on the cluster at my school that uses SLURM architecture. The issue is not with my openfoam program, but with the openmpi module that I am using. I want to optimize the speed of my CFD runs so I would like to use UCX. This is the first go to setting that this version of Openmpi/4.1.4 is using. Because with the baseline "mpirun -np 8 pimpleFoam -parallel (executable)" I get the following error message.

No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

  Host:      cn536
  Framework: pml
--------------------------------------------------------------------------
[cn536:1868530] PMIX ERROR: UNREACHABLE in file ../../../../../../../opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2198

when adding the additional parameter to my mpirun command "--mca pml ^ucx" full command is "mpirun -np 8 --mca pml ^ucx pimpleFoam -parallel " this error disappears, but the simulation doesn't run as fast as it could as it is using something besides UCX in the MPI process. I am not an expert with MPI or sure if this is the right place to ask. I am just wondering how I can get openmpi/4.1.4 running with the optimal settings for large highly parallelized programs.

[jonathandenman]

Currently the fastest full command that I have been able to get to work has been.....

mpirun -np 8 --mca btl vader,openib,self --bind-to core --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

[jsquyres]

FYI @open-mpi/ucx

[rakhmets]

Hi,
UCX libraries cannot be found according to the error message.
You can verify this using ldd command. E.g.

$ ldd <path_to_ompi>/lib/openmpi/mca_pml_ucx.so

If the output of the command says, that UCX libraries are missing, e.g.:

        libucp.so.0 => not found
        libuct.so.0 => not found
        libucs.so.0 => not found
        libucm.so.0 => not found

, then you should add them to LD_LIBRARY_PATH. Since you are loading OpenMPI using modules, maybe you have an appropriate module containing UCX.

[jonathandenman]

Hi, UCX libraries cannot be found according to the error message. You can verify this using ldd command. E.g.

$ ldd <path_to_ompi>/lib/openmpi/mca_pml_ucx.so

If the output of the command says, that UCX libraries are missing, e.g.:

        libucp.so.0 => not found
        libuct.so.0 => not found
        libucs.so.0 => not found
        libucm.so.0 => not found

, then you should add them to LD_LIBRARY_PATH. Since you are loading OpenMPI using modules, maybe you have an appropriate module containing UCX.

Thanks for the response. So I have modules set up for openmpi 4.1.4 and UCX 1.13.1.

When I type in the ldd command and specify mca_pml_ucx.so file. It shows that it can find those libraries inside of the UCX module directory.

libucp.so.0 => /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucp.so.0 (0x00007ffff6ebc000)

    libuct.so.0 => /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libuct.so.0 (0x00007ffff6c84000)

    libucm.so.0 => /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucm.so.0 (0x00007ffff6a6a000)

    libucs.so.0 => /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucs.so.0 (0x00007ffff680d000)

Is there any other reason that this error could be happening? I tested the dependencies on login node as well as the compute node that I am trying to use for my parallel computations.

[rakhmets]

Currently the fastest full command that I have been able to get to work has been.....

mpirun -np 8 --mca btl vader,openib,self --bind-to core --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

You mentioned that you disabled UCX using --mca pml ^ucx. However, you didn't use this option in the command line above.
Seems that this option --mca opal_warn_on_missing_libcuda 0 helped you to resolve the issue. Setting the option to zero ignores libcuda.so loading failure.
What issues do you have with this command line? Does the performance remain below the expectations? Do you need CUDA support in your experiments?

BTW, you can add -x UCX_LOG_LEVEL=info to make sure that the desired version of UCX was loaded during the launch of MPI program. E.g.

mpirun -np 8 -x UCX_LOG_LEVEL=info --mca btl vader,openib,self --bind-to core --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

[jonathandenman]

Currently the fastest full command that I have been able to get to work has been.....
mpirun -np 8 --mca btl vader,openib,self --bind-to core --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

You mentioned that you disabled UCX using --mca pml ^ucx. However, you didn't use this option in the command line above. Seems that this option --mca opal_warn_on_missing_libcuda 0 helped you to resolve the issue. Setting the option to zero ignores libcuda.so loading failure. What issues do you have with this command line? Does the performance remain below the expectations? Do you need CUDA support in your experiments?

BTW, you can add -x UCX_LOG_LEVEL=info to make sure that the desired version of UCX was loaded during the launch of MPI program. E.g.

mpirun -np 8 -x UCX_LOG_LEVEL=info --mca btl vader,openib,self --bind-to core --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

My working command as of right now is....

mpirun -np 8 --mca pml ^ucx --mca btl vader,openib,self --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel

I was mistaken in my second post. Whenever I submit without the --mca pml ^ucx flag I am met with the errors I stated in my first post.

My working command as of right now is....

---

No components were able to be opened in the pml framework.

This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.

Host: cn557
Framework: pml

[cn557:1724353] PML ucx cannot be selected

I am not sure if this is the same error, but this is another message that pops up as well.

---

[cn556:3971937] PMIX ERROR: UNREACHABLE in file ../../../../../../../opal/mca/pmix/pmix3x/pmix/src/server/pmix_server.c at line 2198

The other flags in the message are essentially to counteract OpenFabrics error messages that I get on each processor.

---

However, the test simulation can still run when these messages appear about openFabrics.

No OpenFabrics connection schemes reported that they were able to be
used on a specific port. As such, the openib BTL (OpenFabrics
support) will be disabled for this port.

Local host: cn556
Local device: mlx5_0
Local port: 1
CPCs attempted: rdmacm, udcm

When I enter...

mpirun -np 8 -x UCX_LOG_LEVEL=info --mca btl vader,openib,self --bind-to core:overload--allowed --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

command from your last response the job fails to start with the usual UCX error messages, but I get the following message in my log file.

[1683910050.570416] [cn557:1726804:0] ucp_context.c:1849 UCX INFO Version 1.13.1 (loaded from /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucp.so.0)
[1683910050.612583] [cn557:1726804:0] ucp_worker.c:2267 UCX DIAG multi-threaded worker is requested, but library is built without multi-thread support
[1683910050.671150] [cn556:3982999:0] ucp_context.c:1849 UCX INFO Version 1.13.1 (loaded from /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucp.so.0)
[1683910050.677678] [cn557:1726804:0] parser.c:1918 UCX INFO UCX_* env variables: UCX_LOG_LEVEL=info UCX_POSIX_USE_PROC_LINK=n
[1683910050.683388] [cn556:3983002:0] ucp_context.c:1849 UCX INFO Version 1.13.1 (loaded from /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucp.so.0)
[1683910050.713732] [cn556:3982999:0] ucp_worker.c:2267 UCX DIAG multi-threaded worker is requested, but library is built without multi-thread support
[1683910050.725100] [cn556:3983002:0] ucp_worker.c:2267 UCX DIAG multi-threaded worker is requested, but library is built without multi-thread support
[1683910050.795981] [cn556:3983002:0] parser.c:1918 UCX INFO UCX_* env variables: UCX_LOG_LEVEL=info UCX_POSIX_USE_PROC_LINK=n
[1683910050.778667] [cn556:3982999:0] parser.c:1918 UCX INFO UCX_* env variables: UCX_LOG_LEVEL=info UCX_POSIX_USE_PROC_LINK=n

When I enter the command without --bind-to core:overload-allowed flag....

mpirun -np 8 -x UCX_LOG_LEVEL=info --mca btl vader,openib,self --mca opal_warn_on_missing_libcuda 0 --mca orte_base_help_aggregate 0 --mca btl_openib_warn_no_device_params_found 0 --mca btl_openib_allow_ib true pimpleFoam -parallel >my_prog.out

I get the same errors with the following messages in my log file.

[1683910753.295297] [cn557:1728440:0] ucp_context.c:1849 UCX INFO Version 1.13.1 (loaded from /gpfs/sharedfs1/admin/hpc2.0/apps/ucx/1.13.1/lib/libucp.so.0)
[1683910753.322923] [cn557:1728440:0] ucp_worker.c:2267 UCX DIAG multi-threaded worker is requested, but library is built without multi-thread support
[1683910753.387500] [cn557:1728440:0] parser.c:1918 UCX INFO UCX_* env variables: UCX_LOG_LEVEL=info UCX_POSIX_USE_PROC_LINK=n

Is this starting to give a better idea as to what is wrong between openmpi 4.1.4 and ucx 1.13.1 ?

[rakhmets]

Thanks for the details.
This might be the issue:

ucp_worker.c:2267 UCX DIAG multi-threaded worker is requested, but library is built without multi-thread support

The message appears if MPI_Init_thread is executed with MPI_THREAD_MULTIPLE parameter, and the UCX library is built with --enable-mt (default option).
The call MPI_Init_thread is inside openFoam. I guess, the behavior is controlled by -parallel option of pimpleFoam application.

[jonathandenman]

Yes, I am not sure about the inner workings of the -parallel flag for Open Foam, but I know that I have never gotten this error with open Foam before. The parallel flag essentially gives you the ability to spread the workload across as many processors as you need because depending on the computations you are running you can easily run out of memory and computation can take hours or slow down exponentially.

[jonathandenman]

You decompose your computational domain into multiple processors and so each one is bound to a certain spatial area in the domain you are investigating, but after computations are done they each need to convey information to each other and this needs to be done efficiently and that is why MPI is so important.

Could this be a sign that the UCX library is built wrong or something. I am not very savvy with the computer programming side of things when it comes to MPI .

[rakhmets]

If I understand correctly OpenFOAM code, MPI_THREAD_MULTIPLE is required for collated file input and output (https://www.openfoam.com/news/main-news/openfoam-v1712/parallel). And it is not required if every process writes its own directory.
Maybe smth like this will work for you.

mpirun -np 8 pimpleFoam -parallel -fileHandler uncollated