gdasgldas task fails with Restart Tile Space Mismatch

[BrettHoover-NOAA]

Expected behavior
gdasgldas task should complete successfully and global-workflow should continue to cycle fv3gdas

Current behavior
gdasgldas task is failing on the first cycle in which the task is not skipped (first 00z analysis period when enough data has been produced to trigger task).

Machines affected
This error is being expressed on Orion.

To Reproduce
I am seeing this bug in a test of global-workflow being conducted on Orion in the following directories:
expid: /work/noaa/da/bhoover/para/bth_test
code: /work/noaa/da/bhoover/global-workflow
ROTDIR: /work/noaa/stmp/bhoover/ROTDIRS/bth_test
RUNDIR: /work/noaa/stmp/bhoover/RUNDIRS/bth_test

This run is initialized on 2020082200, and designed to terminate 2 weeks later on 2020090500.

Experiment setup:
/work/noaa/da/bhoover/global-workflow/ush/rocoto/setup_expt.py --pslot bth_test --configdir /work/noaa/da/bhoover/global-workflow/parm/config --idate 2020082200 --edate 2020090500 --comrot /work/noaa/stmp/bhoover/ROTDIRS --expdir /work/noaa/da/bhoover/para --resdet 384 --resens 192 --nens 80 --gfs_cyc 1

Workflow setup:
/work/noaa/da/bhoover/global-workflow/ush/rocoto/setup_workflow.py --expdir /work/noaa/da/bhoover/para/bth_test

Initial conditions:
/work/noaa/da/cthomas/ICS/2020082200/

The error is found in the gdasgldas task on 2020082600.

Log file:
/work/noaa/stmp/bhoover/ROTDIRS/bth_test/logs/2020082600/gdasgldas.log

Context
This run is being used by a new Orion user and member of the satellite DA group, only to familiarize myself with the process of carrying out an experiment. There have been no code-changes made for this run. I followed directions for cloning and building the global-workflow, and setting up a cycled experiment, from the available wiki:

https://github.com/NOAA-EMC/global-workflow/wiki/

I did not create the initial condition files, they were instead produced for me. The global-workflow repository was cloned on January 25 2022 (d3028b9)

The task fails with the following error in the log-file:

0: NOAH Restart File Used: noah.rst
0: 1 1536 768 389408
0: Restart Tile Space Mismatch, Halting..
0: endrun is being called
0: application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

The dimension size of 389408 is suspicious, since earlier in the log a different dimension size is referenced, e.g.:

0: MSG: maketiles -- Size of Grid Dimension: 398658 ( 0 )

When I search for "389408" in the log-file, it only appears in two places, one is in the Restart Tile Space Mismatch error, and the other is while running exec/gldas_rst, when reporting the results of a FAST_BYTESWAP:

216.121 + /work/noaa/da/bhoover/global-workflow/exec/gldas_rst
216.121 + 1>& 1 2>& 2
FAST_BYTESWAP ALGORITHM HAS BEEN USED AND DATA ALIGNMENT IS CORRECT FOR 4 )
1536 768 4 9440776
2 tmp0_10cmdown GLDAS STC1
3 tmp10_40cmdown GLDAS STC2
4 tmp40_100cmdown GLDAS STC3
5 tmp100_200cmdown GLDAS STC4
6 soill0_10cmdown GLDAS SLC1
7 soill10_40cmdown GLDAS SLC2
8 soill40_100cmdown GLDAS SLC3
9 soill100_200cmdown GLDAS SLC4
10 soilw0_10cmdown GLDAS SMC1
11 soilw10_40cmdown GLDAS SMC2
12 soilw40_100cmdown GLDAS SMC3
13 soilw100_200cmdown GLDAS SMC4
15 landsfc
18 vtypesfc
71 tmpsfc GLDAS SKNT
72 weasdsfc GLDAS SWE
79 cnwatsfc GLDAS CMC
88 snodsfc GLDAS SNOD
1 1536 768 389408
216.602 + err=0

I believe that the error is related to the difference in tile-size between these two values.

Detailed Description
I have proposed no change or addition to the code for this run.

Additional Information
Prior gdasgldas tasks in the run from initialization to 2020082600 were successful, but they were all skipped either because the analysis was for a non-00z period or because the requisite number of cycles had not been completed to allow the task to trigger. There are no successful gdasgldas tasks in this run that I can use to compare to the one that has failed. I have conferred with more experienced EMC users of fv3gdas and the cause of the problem is not obvious.

Possible Implementation
I have no implementation plan to offer.

[CatherineThomas-NOAA]

@HelinWei-NOAA

[HelinWei-NOAA]

see the email from Dave

Hi Helin and Jun,

I am finding with a recent upgrade of the global workflow that there is a mismatch between the land-sea mask algorithms of the UFS and the GLDAS. This is resulting in a failure of the gdasgldas job, specifically when the Land Information System (LIS) executable is run. The error reported is 'Restart Tile Space Mismatch, Halting..', which can be found in sorce/gldas_model.fd/lsms/noah.igbp/noahrst.F:104. I have only verified this error for a C192/C96 run on 2020080500, where NCH = 97296 and LIS%D%NCH = 99582.

With the recent upgrade, the UFS is modifying the input land sea mask during the first gdasfcst and outputting this modified mask to the tiled surface restart files. Thus, all future forecasts, analyses, etc use this modified mask until the GLDAS reads its own from $FIX/fix_gldas/FIX_T382/lmask_gfs_T382.bfsa.

So I have a few questions. First, the UFS's modification of the land-sea mask is expected, correct? Secondly, should a new fix file be created for the GLDAS with the modified land-sea mask or is the UFS-modified land-sea mask time dependent and thus not fixed? Lastly, should I expect this to be an issue at all resolutions?

---

Since GLDAS will not be included in the next operational implementation, we need to have someone to decide if we should spend more time on this task.

[yangfanglin]

Is the cycling working without gdasgldas ? Have the user tried at the operational resolutions (C768/C384) ?

[BrettHoover-NOAA]

The cycling appears to fail when gdasgldas fails. I have not run at operational resolution, the current run is an initial test-run I'm doing as a new global-workflow user on Orion and the initial conditions were provided to me at C384/C192. Brett

…

On Mon, Jan 31, 2022 at 9:06 PM Fanglin Yang ***@***.***> wrote: Is the cycling working without gdasgldas ? Have the user tried at the operational resolutions (C768/C384) ? — Reply to this email directly, view it on GitHub <#622 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AXNDXO746HD2FAC4IFILZOLUY5E37ANCNFSM5NHOMEAQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[CatherineThomas-NOAA]

@yangfanglin The experiment had no errors when running in the first few days before GLDAS turns on, so I assume that it would run if GLDAS was turned off altogether.

If as @HelinWei-NOAA says that GLDAS will not be used going forward, should we turn it off now in our experiments? The DA team is still running atmosphere-only cases since we have non-coupling data upgrades to worry about.

[yangfanglin]

@CatherineThomas-NOAA If the cycling at C384/C192 resolutions with gdasgldas turned on is working on WCOSS, there is likely issues related to the setups on Orion. I had some discussion with Daryl about the use of gdasgldas in future systems. We can discuss more about this issue with Daryl offline.

[CatherineThomas-NOAA]

@AndrewEichmann-NOAA Has your WCOSS experiment progressed to the point where the gldas step is run? If so, did it fail or run without issue?

[AndrewEichmann-NOAA]

@CatherineThomas-NOAA No - I ran into rstprod access issues and am waiting for a response from helpdesk

[CatherineThomas-NOAA]

Thanks @AndrewEichmann-NOAA. I can run a short warm start test on WCOSS.

[CatherineThomas-NOAA]

The warm start test on WCOSS ran the gldas step without failure. Now that Hera is back, I can try a quick test there as well.

[CatherineThomas-NOAA]

Now that Hera is back, I can try a quick test there as well.

Global-workflow develop does not build on Hera (#561), so I can't run this test at this time.

[DavidHuber-NOAA]

I ran into this issue with the S4 and Jet ports, which I reported to Helin. I have since turned off GLDAS altogether and everything has run OK out to 10 days.

Below is more of the thread between Helin, @junwang-noaa, and myself:

David,

GLDAS should use the same land-sea mask as UFS. If the land-sea mask can be changed during the forecast, that certainly will bring up some issues for GLDAS.

Helin

Dave,

The model does check/modify the land-sea mask according to the input land/lake fraction and soil type values, this is applied to both non-fractional grid and fractional grid. The changes are required to make sure the model has consistent land sea mask, and soil types. The new land sea mask is output in the model sfc file. I expect you do not change the oro data, the soil type data during your run, so this land sea mask won't change. In other words, I think you need to create lmask_gfs_T382.bfsa once with this land sea mask in the history file.

Jun

[arunchawla-NOAA]

@HelinWei-NOAA @barlage and @yangfanglin should Jun's suggestion be followed at the mask be generated for GLDAS from the history file ? It would be good to know if the same issue is seen on other platforms

[yangfanglin]

@arunchawla-NOAA Cathy reported that "the warm start test on WCOSS ran the gldas step without failure". So the failures on other platforms might be a porting issue. @CatherineThomas-NOAA Cathy, can you confirm ? What is the resolution you tested on WCOSS ? I assume you were using the GFS.v16* tag instead of the UFS, right ?