Deadlock in mpi_bcast using openmpi 4.1.5 from fortran

[pums974]

Background information

What version of Open MPI are you using? (e.g., v3.0.5, v4.0.2, git branch name and hash, etc.)

openmpi-4.1.5-1

Describe how Open MPI was installed (e.g., from a source/distribution tarball, from a git clone, from an operating system distribution package, etc.)

From archlinux extra repository or building it locally from PKGBUILD

Please describe the system on which you are running

• Operating system/version: archlinux up to date (09 march 2023)
• Computer hardware: dell laptop, intel processor
• Network type:

---

Details of the problem

The following fortran code reach a deadlock when using multiple process

program test_mpi
    use mpi_f08
    implicit none
    character(len=10), dimension(16) :: strings
    integer :: dummy_int
    integer :: ierr

    call mpi_init(ierr)

    strings = ""

    dummy_int = 16

    ! without these two bcast, code is working properly
    call mpi_bcast(dummy_int, 1, mpi_integer, 0, MPI_COMM_WORLD, ierr)
    call mpi_bcast(dummy_int, 1, mpi_integer, 0, MPI_COMM_WORLD, ierr)

    print*, "everyone print this"
    call mpi_bcast(strings, 16*10, mpi_character, 0, MPI_COMM_WORLD, ierr)
    print*, "no one print that"

    call mpi_finalize(ierr)
end program test_mpi

Running it with one process gives the expected result

shell$ mpifort -o test_mpi test_mpi.f90 && mpirun -np 1 test_mpi
 everyone print this
 no one print that

while running it with at least 2 process result in a deadlock

shell$ mpirun -np 2 test_mpi
 everyone print this
 everyone print this

The same code compiled with openmpi-4.1.4 gives the following correct result

shell$ mpirun -np 2 test_mpi
 everyone print this
 everyone print this
 no one print that
 no one print that

[ggouaillardet]

I am unable to reproduce this issue on an up-to-date archlinux VM (with 4 cpus).

How many cpus does your VM has?

you can run lscpu or lstopo to figure out.

that could be related to #10416.

What if you

mpirun --mca coll ^han,adapt -np 2 test_mpi

can you try to rebuild Open MPI after manually downloading and applying the patch at https://github.com/open-mpi/ompi/issues/11478.patch ?
It addresses the other issue and might as well fix this one.

[pums974]

It is not a vm, it's a pure linux computer with 16 logical cores

shell$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               GenuineIntel
  Model name:            11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
    CPU family:          6
    Model:               141
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            1
    CPU(s) scaling MHz:  32%
    CPU max MHz:         4800,0000
    CPU min MHz:         800,0000
    BogoMIPS:            4993,00
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl 
                         xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsav
                         e avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l2 invpcid_single cdp_l2 ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 sme
                         p bmi2 erms invpcid rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves split_lock_detect dtherm ida arat pln pts 
                         hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid movdiri movdir64b fsrm avx512_vp2inters
                         ect md_clear flush_l1d arch_capabilities
Virtualization features: 
  Virtualization:        VT-x
Caches (sum of all):     
  L1d:                   384 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    10 MiB (8 instances)
  L3:                    24 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Not affected

I have the same results using

mpirun --mca coll ^han,adapt -np 2 test_mpi

But while I was answering you, I realized that my problem only occurs when my laptop is connected my ethernet network (connected directly or via my screen which act as a dock).

If connected via WIFI, the bug disappear.

My computer is a Dell Precision 3561 (few month old). I have no other indication of an hardware issue.

[pums974]

I'm sorry but I cannot download the patch you want me to try

[ggouaillardet]

My bad, the patch is at https://github.com/open-mpi/ompi/pull/11472.patch

That being said, and based on your description, this is a network issue and the patch (that can help with the topology) is likely helpless here.

[ggouaillardet]

On second thought, the patch will likely hide the network issue, and is hence worth testing.

[pums974]

I am away from my office, I'll try it monday

[pieterbruegel]

I have a similar bug. However, in my case, running with -np 10 uses all 64 CPUs, and after a few seconds, the program stops. Is it a kind of deadlock? I completely don't know, how to ask about this issue. When I run @pums974 code on 1 CPU I have
everyone print this no one print that and for 2 CPUs everyone print this no one print that everyone print this no one print that.

[ggouaillardet]

@pieterbruegel that could be the same bug.
Meanwhile, you can try to
mpirun --mca btl_tcp_if_include <interface> --mca oob_tcp_if_include <interface> ...
where <interface> is an interface or ip/mask that is known to be working (e.g. no firewall)

[pums974]

@ggouaillardet your are right,
mpirun --mca btl_tcp_if_include lo --mca oob_tcp_if_include lo -np 2 test_mpi
make it run (using wifi interface also works)

[pums974]

You were also right about the patch, it solves my issue.

Thanks a lot, and thank you for your reactivity !

[pieterbruegel]

@ggouaillardet In my case, the source of the issue had lied on the mpi version of ScaLAPACK or fftw library (I'm not sure which one). Thanks for your help!