Investigate decomp_suite failures with dynpicard option

[phil-blain]

Running the decomp_suite with the VP dynamics results in some segfaults (due to NaN initialisation), some errors ("bad departure points") and some non BFB restarts, see CICE-Consortium#518.

I'll use this issue to document my findings in investigating those.

[phil-blain]

Cases that are missing data

$ ./results.csh |\grep MISS
MISS daley_intel_smoke_gx3_8x2x8x10x20_debug_droundrobin_dynpicard_maskhalo_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day missing-data
MISS daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day missing-data
MISS daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day missing-data

these in fact are cases were the model segfaulted/crashed, so it is the data for the case itself that is missing, not the one that we are comparing against.

EDIT 2022/05: reported in CICE-Consortium#608 and subsequently fixed.

daley_intel_smoke_gx3_8x2x8x10x20_debug_droundrobin_dynpicard_maskhalo_run2day

core file reveals diag[xy] contain NaNs (due to initialization probably) at cicecore/cicedynB/dynamics/ice_dyn_vp.F90:3517. From a quick look it seems the OpenMP directive is missing i,j,ij in the private variables.

daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day

core file reveals same as above (need to switch thread in the core with thread $num, see https://sourceware.org/gdb/onlinedocs/gdb/Threads.html. The right thread to use is the one stopped in ../sysdeps/unix/sysv/linux/x86_64/sigaction.c:62)

daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day

core file reveals same as above.

Cases that fail to run

$ ./results.csh |\grep FAIL | \grep ' run'
FAIL daley_intel_restart_gx1_64x1x16x16x10_dwghtfile_dynpicard run
FAIL daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard run
FAIL daley_intel_smoke_gx1_64x1x16x16x10_debug_dwghtfile_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gbox180_16x1x6x6x60_debug_debugblocks_dspacecurve_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gx3_8x2x8x10x20_debug_droundrobin_dynpicard_maskhalo_run2day run -1 -1 -1
FAIL daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day run -1 -1 -1

daley_intel_restart_gx1_64x1x16x16x10_dwghtfile_dynpicard, daley_intel_smoke_gx1_64x1x16x16x10_debug_dwghtfile_dynpicard_run2day

missing an input file, see CICE-Consortium#602 (comment)

daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard

"ERROR: bad departure points"

daley_intel_smoke_gbox180_16x1x6x6x60_debug_debugblocks_dspacecurve_dynpicard_run2day

NaNs in gridbox_corners... probably not related to VP solver...

#4  0x0000000000de0fbc in ice_grid::gridbox_corners () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/infrastructure/ice_grid.F90:2219
#5  0x0000000000d76302 in ice_grid::init_grid2 () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/infrastructure/ice_grid.F90:570
#6  0x0000000000401b83 in cice_initmod::cice_init () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE_InitMod.F90:121
#7  0x00000000004019c7 in cice_initmod::cice_initialize () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE_InitMod.F90:52
#8  0x0000000000401671 in icemodel () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE.F90:43
#9  0x00000000004015f2 in main ()
#10 0x00000000021d930f in __libc_start_main (main=..., argc=..., argv=..., init=..., fini=..., rtld_fini=..., stack_end=...) at ../csu/libc-start.c:308
#11 0x00000000004014da in _start () at ../sysdeps/x86_64/start.S:120
(gdb) f 4
#4  0x0000000000de0fbc in ice_grid::gridbox_corners () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/infrastructure/ice_grid.F90:2219
2219                work_g2(:,:) = lont_bounds(icorner,:,:,iblk) + c360

array lont_bounds(icorner,:,:,iblk) is all NaN.

EDIT the above was already reported in CICE-Consortium#599 (comment) ("Problems in ice_grid.F90"), on the gbox128 grid.

The other 3 cases are mentioned above in the MISS section.

[phil-blain]

Cases that fail 'test'

$ ./results.csh |\grep FAIL | \grep ' test'
FAIL daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard test 
FAIL daley_intel_restart_gx1_64x1x16x16x10_dwghtfile_dynpicard test 
FAIL daley_intel_restart_gx3_20x2x5x4x30_dsectrobin_dynpicard_short test 
FAIL daley_intel_restart_gx3_1x4x25x29x16_droundrobin_dynpicard test 
FAIL daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard test 
FAIL daley_intel_restart_gx3_16x2x3x3x100_droundrobin_dynpicard test 
FAIL daley_intel_smoke_gx1_64x1x16x16x10_debug_dwghtfile_dynpicard_run2day test 
FAIL daley_intel_smoke_gbox180_16x1x6x6x60_debug_debugblocks_dspacecurve_dynpicard_run2day test 
FAIL daley_intel_smoke_gx3_8x2x8x10x20_debug_droundrobin_dynpicard_maskhalo_run2day test 
FAIL daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day test 
FAIL daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day test 

daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard

fails to restart exactly

daley_intel_restart_gx3_20x2x5x4x30_dsectrobin_dynpicard_short

fails to restart exactly

daley_intel_restart_gx3_1x4x25x29x16_droundrobin_dynpicard

fails to restart exactly

daley_intel_restart_gx3_16x2x3x3x100_droundrobin_dynpicard

fails to restart exactly

The other cases also fail to run (see above).

[phil-blain]

Others failing tests are because they are not BFB.

[phil-blain]

I fixed the buggy OpenMP directive:

diff --git i/cicecore/cicedynB/dynamics/ice_dyn_vp.F90 w/cicecore/cicedynB/dynamics/ice_dyn_vp.F90
index 457a73a..367d29e 100644
--- i/cicecore/cicedynB/dynamics/ice_dyn_vp.F90
+++ w/cicecore/cicedynB/dynamics/ice_dyn_vp.F90
@@ -3507,7 +3507,7 @@ subroutine precondition(zetaD       ,        &
          wx = vx
          wy = vy
       elseif (precond_type == 'diag') then ! Jacobi preconditioner (diagonal)
-         !$OMP PARALLEL DO PRIVATE(iblk)
+         !$OMP PARALLEL DO PRIVATE(iblk, ij, i, j)
          do iblk = 1, nblocks
             do ij = 1, icellu(iblk)
                i = indxui(ij, iblk)

So let's do another round (~/cice-dirs/suites/vp-decomp-openmp-fix)

Cases that are missing data

daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day

run segfaulted.

#8  0x00000000009ca4e4 in ice_dyn_vp::calc_l2norm_squared (nx_block=..., ny_block=..., icellu=..., indxui=..., indxuj=..., tpu=..., tpv=..., l2norm=...)
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:2479
#9  0x0000000000a29749 in ice_dyn_vp::L_ice_dyn_vp_mp_pgmres__3290__par_loop4_2_56 ()
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:3292
#10 0x0000000001f68883 in __kmp_invoke_microtask ()
#11 0x0000000001f12b2a in __kmp_invoke_task_func ()
#12 0x0000000001f143d6 in __kmp_fork_call ()
#13 0x0000000001edfb25 in __kmpc_fork_call ()
#14 0x0000000000a09894 in ice_dyn_vp::pgmres (zetad=..., cb=..., vrel=..., umassdti=..., bx=..., by=..., diagx=..., diagy=..., tolerance=..., maxinner=..., maxouter=..., 
    solx=..., soly=..., nbiter=...) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:3290
#15 0x0000000000a35680 in ice_dyn_vp::precondition (zetad=..., cb=..., vrel=..., umassdti=..., vx=..., vy=..., diagx=..., diagy=..., precond_type=..., wx=..., wy=..., 
    .tmp.PRECOND_TYPE.len_V$69f7=...) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:3528
#16 0x00000000009da5f8 in ice_dyn_vp::fgmres (zetad=..., cb=..., vrel=..., umassdti=..., halo_info_mask=..., bx=..., by=..., diagx=..., diagy=..., tolerance=..., maxinner=..., 
    maxouter=..., solx=..., soly=..., nbiter=...) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:2858
#17 0x000000000094ab6e in ice_dyn_vp::anderson_solver (icellt=..., icellu=..., indxti=..., indxtj=..., indxui=..., indxuj=..., aiu=..., ntot=..., waterx=..., watery=..., 
    bxfix=..., byfix=..., umassdti=..., sol=..., fpresx=..., fpresy=..., zetad=..., cb=..., halo_info_mask=...)
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:932
#18 0x0000000000914ba7 in ice_dyn_vp::implicit_solver (dt=...) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:483
#19 0x00000000014c41d7 in ice_step_mod::step_dyn_horiz (dt=...) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/general/ice_step_mod.F90:886
#20 0x000000000040cd41 in cice_runmod::ice_step () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE_RunMod.F90:284
#21 0x000000000040b849 in cice_runmod::cice_run () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE_RunMod.F90:83
#22 0x0000000000400cd0 in icemodel () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/drivers/standalone/cice/CICE.F90:49
#23 0x0000000000400c32 in main ()
#24 0x00000000020f7edf in __libc_start_main (main=..., argc=..., argv=..., init=..., fini=..., rtld_fini=..., stack_end=...) at ../csu/libc-start.c:308
#25 0x0000000000400b1a in _start () at ../sysdeps/x86_64/start.S:120
(gdb) f 8
#8  0x00000000009ca4e4 in ice_dyn_vp::calc_l2norm_squared (nx_block=..., ny_block=..., icellu=..., indxui=..., indxuj=..., tpu=..., tpv=..., l2norm=...)
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice2/cicecore/cicedynB/dynamics/ice_dyn_vp.F90:2479

à cause d'un out-of-range dans le calcul de la norme...:

(gdb) p tpu(i,j)
$3 = -1.4002696118736502e+97
(gdb) p tpv(i,j)
$4 = -2.4873702247927307e+218
(gdb) p tpv(i,j)**2
Cannot perform exponentiation: Numerical result out of range
(gdb) p tpu(i,j)**2
$5 = 1.9607549859367831e+194

daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day

same as above

Cases that failed to run

 ./results.csh |\grep FAIL | \grep ' run'
FAIL daley_intel_restart_gx1_64x1x16x16x10_dwghtfile_dynpicard run
FAIL daley_intel_restart_gx3_1x4x25x29x16_droundrobin_dynpicard run 
FAIL daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard run
FAIL daley_intel_smoke_gx1_64x1x16x16x10_debug_dwghtfile_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gbox180_16x1x6x6x60_debug_debugblocks_dspacecurve_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day run -1 -1 -1
FAIL daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day run -1 -1 -1

daley_intel_restart_gx1_64x1x16x16x10_dwghtfile_dynpicard, daley_intel_smoke_gx1_64x1x16x16x10_debug_dwghtfile_dynpicard_run2day

same as above (missing file)

daley_intel_restart_gx3_1x4x25x29x16_droundrobin_dynpicard, daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard

(horizontal_remap)ERROR: bad departure points

The next two are same as in first suite above.
The last two are those that segfaulted (see just above)

[phil-blain]

Note: re-running the same suite a second time leads to different results:

• some tests that were not BFB the first time are in the second, and vice-versa
• some tests that did not restart exactly the first time do in the second, and vice-versa

This really suggests that there is some non-reproducibility in the code...

[apcraig]

Just a quick update. I'm playing with the OpenMP in the entire code and tested evp, eap, and vp. I can also confirm that running different thread counts with vp produces different answers. If "re-running the same suite a second time leads to different results", that suggests the code is not bit-for-bit reproducible when rerun? I tried to test that and for my quick tests, the same run does seem to be reproducible. It's a little too bad because that's an easier problem to debug. I also tested a 32x1x16x16x16 and 64x1x16x16x16 case and they are not bit-for-bit. Same decomp, no OpenMP, just different block distribution. If I get a chance, I will try to look into this more. At this point, I will probably defer further OpenMP optimization with vp. I think there are several tasks to do

• address vp reproducibility on restart and with different decompositions and/or tasks
• review vp performance
• validate vp threading
  I did fix a few OMP loops in vp and will PR those, but that work is not entirely complete.

[phil-blain]

Hi Tony, thanks for these details and tests. This isssue is definitely still on my list, I hope I'll have to time to go back to the VP solver this Winter/early Spring.

I'll take a look at the PR when you submit it.

[phil-blain]

I'm finally going back to this. I've re-ran the decomp_suite with -s dynpicard on our 2 machines, testing the latest master main. I've ran the suite twice on each machine, and get the same results on the same machine, and across machines, modulo -init=snans,arrays which is inactive on daley but active on banting, and modulo the exact failure mode. So at least that's that.

Summary:

$ ./results.csh |tail -5
203 measured results of 203 total results
157 of 203 tests PASSED
0 of 203 tests PENDING
0 of 203 tests MISSING data
46 of 203 tests FAILED

cases that fail "run"

$ ./results.csh |\grep FAIL|\grep ' run'
FAIL daley_intel_restart_gx3_6x2x50x58x1_droundrobin_dynpicard run
FAIL daley_intel_smoke_gx3_6x2x50x58x1_debug_droundrobin_dynpicard_run2day run -1 -1 -1

on banting I also get banting_intel_smoke_gbox180_16x1x6x6x60_debug_debugblocks_dspacecurve_dynpicard_run2day but this is due to CICE-Consortium#599 (comment) ("Problems in ice_grid.F90")

daley_intel_restart_gx3_6x2x50x58x1_droundrobin_dynpicard

fails at the first run of the test with SIGILL or SIGSEGV (varies). Usually no core is produced (got a core once but more or less useful since this case is not compiled in debug mode, when I recompiled in debug mode I did not get a core...),

forrtl: severe (168): Program Exception - illegal instruction
Image              PC                Routine            Line        Source
cice               0000000001130484  Unknown               Unknown  Unknown
cice               00000000009C4700  Unknown               Unknown  Unknown
Unknown            00002AAAAFFF5F8B  Unknown               Unknown  Unknown
[NID 00724] 2022-05-12 16:13:50 Apid 16619909: initiated application termination

daley_intel_smoke_gx3_6x2x50x58x1_debug_droundrobin_dynpicard_run2day

fails with SIGSEGV, got a core on one machine but not the other.

 Finished writing ./history/iceh_ic.2005-01-01-03600.nc
*** stack smashing detected ***: <unknown> terminated
forrtl: error (65): floating invalid
Image              PC                Routine            Line        Source
cice               0000000002650764  Unknown               Unknown  Unknown
cice               0000000001EE45C0  Unknown               Unknown  Unknown
cice               0000000001ADBA36  ice_transport_rem        3473  ice_transport_remap.F90
cice               00000000019BF6B6  ice_transport_rem         755  ice_transport_remap.F90
cice               00000000023BF723  Unknown               Unknown  Unknown
cice               00000000023699CA  Unknown               Unknown  Unknown
cice               000000000236B276  Unknown               Unknown  Unknown
cice               00000000023369C5  Unknown               Unknown  Unknown
cice               00000000019A4C79  ice_transport_rem         642  ice_transport_remap.F90
cice               000000000192A728  ice_transport_dri         553  ice_transport_driver.F90
cice               00000000018B5E0C  ice_step_mod_mp_s         959  ice_step_mod.F90
cice               000000000040ED7F  cice_runmod_mp_ic         285  CICE_RunMod.F90
cice               000000000040D7C1  cice_runmod_mp_ci          85  CICE_RunMod.F90
cice               0000000000401690  MAIN__                     49  CICE.F90
cice               00000000004015F2  Unknown               Unknown  Unknown
cice               000000000273228F  Unknown               Unknown  Unknown
cice               00000000004014DA  Unknown               Unknown  Unknown
_pmiu_daemon(SIGCHLD): [NID 01334] [c6-0c2s13n2] [Thu May 12 16:31:38 2022] PE RANK 0 exit signal Aborted

"stack smashing detected" that I've never seen. Here is the backtrace:

(gdb) bt
#0  0x0000000001a05d33 in ice_transport_remap::locate_triangles (nx_block=52, ny_block=60, ilo=2, ihi=51, jlo=2, jhi=59, nghost=1, edge=..., icells=..., indxi=..., indxj=..., dpx=..., dpy=..., dxu=...,
    dyu=..., xp=..., yp=..., iflux=..., jflux=..., triarea=..., l_fixed_area=.FALSE., edgearea=..., .tmp.EDGE.len_V$2530=80)
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/cicedynB/dynamics/ice_transport_remap.F90:1891
#1  0x00000000019bb5d7 in horizontal_remap::L_ice_transport_remap_mp_horizontal_remap__642__par_loop2_2_6 ()
    at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/cicedynB/dynamics/ice_transport_remap.F90:715
#2  0x00000000023bf723 in __kmp_invoke_microtask ()
#3  0x00000000023699ca in __kmp_invoke_task_func ()
#4  0x000000000236b276 in __kmp_fork_call ()
#5  0x00000000023369c5 in __kmpc_fork_call ()
#6  0x00000000019a4c79 in ice_transport_remap::horizontal_remap (dt=3600, ntrace=26, uvel=..., vvel=..., mm=..., tm=..., l_fixed_area=.FALSE., tracer_type=..., depend=..., has_dependents=...,
    integral_order=3, l_dp_midpt=.TRUE., grid_ice=..., uvele=..., vveln=..., .tmp.GRID_ICE.len_V$6dc=256) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/cicedynB/dynamics/ice_transport_remap.F90:642
#7  0x000000000192a728 in ice_transport_driver::transport_remap (dt=3600) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/cicedynB/dynamics/ice_transport_driver.F90:553
#8  0x00000000018b5e0c in ice_step_mod::step_dyn_horiz (dt=3600) at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/cicedynB/general/ice_step_mod.F90:959
#9  0x000000000040ed7f in cice_runmod::ice_step () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/drivers/standalone/cice/CICE_RunMod.F90:285
#10 0x000000000040d7c1 in cice_runmod::cice_run () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/drivers/standalone/cice/CICE_RunMod.F90:85
#11 0x0000000000401690 in icemodel () at /fs/homeu1/eccc/cmd/cmde/phb001/code/cice3/cicecore/drivers/standalone/cice/CICE.F90:49

cases that fail "test"

$ ./results.csh |\grep FAIL|\grep ' test'
FAIL daley_intel_restart_gx3_6x2x50x58x1_droundrobin_dynpicard test
FAIL daley_intel_smoke_gx3_6x2x50x58x1_debug_droundrobin_dynpicard_run2day test

which are the same as above (if "run" fails, "test" fails)

cases that fail "bfbcomp"

first the decomp test itself:

$ ./results.csh |\grep FAIL | \grep daley_intel_decomp
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_slenderX1 bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_roundrobin bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_sectcart bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_sectrobin bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_spacecurve bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop
FAIL daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_rakeX1 bfbcomp daley_intel_decomp_gx3_4x2x25x29x5_dynpicard_squarepop

and then the rest:

$ ./results.csh |\grep FAIL | \grep different-data
FAIL daley_intel_restart_gx3_1x1x50x58x4_droundrobin_dynpicard_thread bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_4x1x25x116x1_dslenderX1_dynpicard_thread bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_6x2x4x29x18_dspacecurve_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_8x2x8x10x20_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_5x2x33x23x4_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_4x2x19x19x10_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_20x2x5x4x30_dsectrobin_dynpicard_short bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_16x2x5x10x20_drakeX2_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_8x2x8x10x20_droundrobin_dynpicard_maskhalo bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_1x4x25x29x16_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_1x8x30x20x32_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_1x1x120x125x1_droundrobin_dynpicard_thread bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_16x2x1x1x800_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_16x2x2x2x200_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_16x2x3x3x100_droundrobin_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_16x2x8x8x80_dspiralcenter_dynpicard bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_10x1x10x29x4_dsquarepop_dynpicard_thread bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_restart_gx3_8x1x25x29x4_drakeX2_dynpicard_thread bfbcomp daley_intel_restart_gx3_4x2x25x29x4_dslenderX2_dynpicard different-data
FAIL daley_intel_smoke_gx3_1x1x25x58x8_debug_droundrobin_dynpicard_run2day_thread bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_20x1x5x116x1_debug_dslenderX1_dynpicard_run2day_thread bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_6x2x4x29x18_debug_dspacecurve_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_8x2x10x12x16_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_5x2x33x23x4_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_4x2x19x19x10_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_20x2x5x4x30_debug_dsectrobin_dynpicard_run2day_short bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_16x2x5x10x20_debug_drakeX2_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_8x2x8x10x20_debug_droundrobin_dynpicard_maskhalo_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_1x6x25x29x16_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_1x8x30x20x32_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_1x1x120x125x1_debug_droundrobin_dynpicard_run2day_thread bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_16x2x1x1x800_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_16x2x2x2x200_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_16x2x3x3x100_debug_droundrobin_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_16x2x8x8x80_debug_dspiralcenter_dynpicard_run2day bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_10x1x10x29x4_debug_dsquarepop_dynpicard_run2day_thread bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data
FAIL daley_intel_smoke_gx3_8x1x25x29x4_debug_drakeX2_dynpicard_run2day_thread bfbcomp daley_intel_smoke_gx3_4x2x25x29x4_debug_dslenderX2_dynpicard_run2day different-data

the good news

• no "BAD DEPARTURE POINTS" error anymore
• no failed restart test

[phil-blain]

daley_intel_restart_gx3_6x2x50x58x1_droundrobin_dynpicard

• I tried to run it in DDT after recompiling in debug mode. It's not obvious where it fails as it's inside an OpenMP loop, but I'm not sure of how to use DDT with OpenMP... I'm experimenting.

• I tried to recompile with the DDT memory debugging library. That was fun. See end of this post.

  • https://developer.arm.com/documentation/101136/2112/DDT/Memory-debugging/Memory-debugging-options?lang=en
  • https://developer.arm.com/documentation/101136/2112/Known-issues-and-notes/Compiler-notes-and-known-issues/Intel-compilers?lang=en

Running with the memory debugging library, however, hides the segfault and the code runs correctly ...

---

On XC-50, all executables use static linking, so it's not possible for DDT to preload its memory debugging library; you have to relink your executable with DDT's memory debugging library.
Instructions can be found in the "Static Linking" section of the Arm Forge user guide (section 12.4.1).
The exact linking flags to use vary if your program is multithreaded and if it uses C++.

• For a non-multithreaded, non C++ program : -Wl,-allow-multiple-definition,-undefined=malloc /path/to/ddt/lib/64/libdmalloc.a

• For multithreaded (OpenMP), non C++ program : -Wl,-wrap=dlopen, -wrap=dlclose,-allow-multiple-definition,-undefined=malloc /path/to/ddt/lib/64/libdmallocth.a

• For a non-multithreaded C++ program : -Wl,-allow-multiple-definition,-undefined=malloc,-undefined=_ZdaPv /path/to/ddt/lib/64/libdmallocxx.a

• For a multithreaded C++ program : -Wl,-wrap=dlopen,-wrap=dlclose,-allow-multiple-definition,-undefined=malloc,-undefined=_ZdaPv /path/to/ddt/lib/64/libdmallocthcxx.a

Note: Adding --wrap=dlopen,--wrap=dlclose for threaded programs makes the link fail at least with the Intel Fortran compiler.
This is normal, as ARM explains in the "Compiler notes and known issues" for the Intel Compilers section:

If you are compiling static binaries, linking on a Cray XT/XE machine in the Arm DDT memory debugging library is not straightforward for F90 applications.
You must manually rerun the last ld command (as seen with ifort -v) to include -L{ddt-path}/lib/64 -ldmalloc in two locations:

• Include immediately prior to where -lc is located.

• Include the -zmuldefs option at the start of the ld line.

This is not easy to understand as the wording is weird. A few notes:

• We use the Cray wrapper ftn to compile and link, not the Intel compiler driver ifort. Replace ftn with ftn -v at link time to see the full ld invocation.

• They mention including -L{ddt-path}/lib/64 -ldmalloc, but:

  • In the "Static Linking" section mentionned above, they recommended to not use -L along with -l and instead link directly to the static library using its full path.

  • In the "Static Linking" section mentionned above, they specify different libraries to use depending on multithreaded-ness and usage of C++, whereas here they only mention libdmalloc.

• The two bullets are supposed to be two locations where -L{ddt-path}/lib/64 -ldmalloc should be included, but the second bullet is not a location, it is a separate recommendation to also add -zmuldefs as the first argument to ld

  • In the "Static Linking" section mentionned above, they recommended to add, amongst other flags, -allow-multiple-definition, which is the same exact thing as -zmuldefs.

  • The location of this argument in the ld invocation does not seem to matter, so we do not need to add it a second time.

  • So in the end it's not clear at all where is the other location!

In practice, -lc appears twice in the ld invocation, and it suffices to add /opt/forge/20.0.1/lib/64/libdmallocth.a (or one of the other 3 libraries) immediately before the first -lc for the link to succeed.

[phil-blain]

OK, I tested the failing tests above without dynpicard and it failed in the same way. This was due to OMP_STACKSIZE being unset and the newly active OpenMP directive in ice_transport_remap.F90 since d1e972a (Update OMP (CICE-Consortium#680), 2022-02-18).

Uncommenting the variable in the machine file (comment added in 8c23df8 (- Update version and copyright. (CICE-Consortium#691), 2022-02-23) makes both test pass.

[phil-blain]

OK so with the little code modifications mentioned in #40 (comment), which I will push tomorrow, the decomp_suite passes [1] with dynpicard when also adding these settings:

precond = diag # or ident, not yet tested
bfbflag = reprosum # maybe works with dpddd and lsum16, not yet tested.

This is very encouraging as it shows not only that the OpenMP implementation is OK, but also that we did not "miss" anything MPI-related (like halo updates, etc) in the VP implementation.

EDIT forgot the end note:

---

[1] I do have one failure, ppp6_intel_restart_gx3_16x2x1x1x800_droundrobin_dynpicard but this test is a known failure even with EVP on our new machines (cf. https://gitlab.science.gc.ca/hpc/hpcr_upgrade_2/issues/244 [internal]). This is most likely a bug in Intel MPI.

[phil-blain]

OK, unsurprisingly it also passes with ident.

[phil-blain]

However, I have 3 failures with precond=pgmres:

$ ./results.csh | \grep FAIL
FAIL ppp6_intel_smoke_gx3_6x2x50x58x1_debug_droundrobin_dynpicard_reprosum_run2day run -1 -1 -1
FAIL ppp6_intel_smoke_gx3_6x2x50x58x1_debug_droundrobin_dynpicard_reprosum_run2day test
FAIL ppp6_intel_smoke_gx3_20x2x5x4x30_debug_dsectrobin_dynpicard_reprosum_run2day_short run -1 -1 -1
FAIL ppp6_intel_smoke_gx3_20x2x5x4x30_debug_dsectrobin_dynpicard_reprosum_run2day_short test
FAIL ppp6_intel_smoke_gx3_10x1x10x29x4_debug_dsquarepop_dynpicard_reprosum_run2day_thread run -1 -1 -1
FAIL ppp6_intel_smoke_gx3_10x1x10x29x4_debug_dsquarepop_dynpicard_reprosum_run2day_thread test