mpas_dmpar.f90 build error on Summit with PGI compiler and OMP_NUM_THREADS=2

[dqwu]

A similar issue is #3236

Steps to reproduce with a low-res G case:

git clone https://github.com/E3SM-Project/E3SM.git
cd E3SM
git submodule update --init
cd cime/scripts
./create_newcase --compiler=pgi --case CMPASO-NYF_T62_oQU120 --compset CMPASO-NYF --res T62_oQU120
cd CMPASO-NYF_T62_oQU120
./xmlchange NTHRDS_OCN=2
./case.setup
./case.build

Error message from output log:

/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.13.4-glp7fk6jzzpzrv3ielkg5mjcar2umm7n/bin/cmake -E cmake_copy_f90_mod mpas-source/src/mpas_tracer_advection_std.mod mpas-source/src/CMakeFiles/common.dir/mpas_tracer_advection_std.mod.stamp PGI
/autofs/nccs-svm1_sw/summit/.swci/0-core/opt/spack/20180914/linux-rhel7-ppc64le/gcc-4.8.5/cmake-3.13.4-glp7fk6jzzpzrv3ielkg5mjcar2umm7n/bin/cmake -E touch mpas-source/src/CMakeFiles/common.dir/__/__/operators/mpas_tracer_advection_std.f90.o.provides.build
PGF90-F-0000-Internal compiler error. print_token(): missing token       0  (bld/cmake-bld/framework/mpas_dmpar.f90: 8236)

Seems like OMP related, see code around line 8236 of framework/mpas_dmpar.f90:

      !$omp do private(commListPtr, listPosition, bufferOffset, nAdded, fieldCursor, exchListPtr, iExch, iBuffer) 
      do listItem = 1, commListSize
        commListPtr => exchangeGroup % sendList
        do listPosition = 2, listItem
          commListPtr => commListPtr % next
        end do
        ...
      end do
      !$omp end do

[philipwjones]

I ran into this with the current default pgi compiler. It disappears in later compiler versions (eg 19.9) and appears to be a compiler bug. @minxu74 or @sarats - might want to update the compiler for E3SM?

[jgfouca]

Unassigning myself until it is confirmed that it's a build system problem.

[minxu74]

@philipwjones @dqwu It can be compiled successfully when bumping PGI version to 19.9, but it failed with core dumpings during ocn initialization and there is no useful information in the log files.

[philipwjones]

@minxu74 I confirmed the run failure with 19.9 in standalone mode as well. Other attempts to build with other pgi compiler versions exhibited either the original error on compile or the run-time failure. From the log, it appears that the run-time failure is during a read. Looking into this a bit more. @dqwu - if this is getting in your way, one workaround is to remove omp directives from the dmpar routine. I believe that has fixed this issue in the past at the cost of threading within the MPAS framework communication routines. Need to look into that too...

[philipwjones]

@dqwu - belay that last. Removing all openmp directives from dmpar will break some things...

[rljacob]

If OMP_NUM_THREADS=1 does everything work?

[philipwjones]

Yes for single-thread, but OpenMP enabled. And it actually works (B4B) in a standalone ocean run in multi-threaded (2 threads) on Summit if the pointer arguments in the private clause are removed. This is not likely to be an acceptable solution but can be a local mod if people need a workaround near-term. I think it works for the ocean because variables are already default first private, but don't think we can guarantee that for other MPAS components. It's still likely a bug in PGI compiler but I haven't been able to come up with any other workaround.

[minxu74]

Hi @philipwjones Did you report it to OLCF? If not, I will send them an email to report the bug. Thanks.

[philipwjones]

@minxu74 - no I haven't reported it yet because I was still trying to figure out the underlying cause and create a smaller test code to illustrate it. Probably should go ahead and report it anyway, so feel free to do so.