Refactor UFS-DA ATM to use python based approach of UFS_DA Aerosol

[RussTreadon-NOAA]

Description
Refactor UFS-DA ATM g-w components to use UFS-DA Aerosol python based approach implemented via PR #1106.

Specifically the g-w components activated via DO_JEDIVAR="YES" and DO_JEDIENS="YES" will be examined and aligned with the python based approach used by UFS-DA Aerosol g-w components.

Requirements
Refactoring of UFS-DA ATM g-w components should not alter existing ATM analysis results.

Acceptance Criteria (Definition of Done)
This issue will be closed when it is possible to cycle UFS-DA ATM g-w components using the python based approach of UFS-DA Aerosol g-w components and the PR to fold these changes into g-w develop is approved and merged.

Dependencies
Changes to the UFS-DA ATM component may require changes in the GDASApp ATM component (e.g., yaml files, python scripts). GDASApp dependencies, if they arise, will be added to this issue. In turn, this issue will be listed as a dependency in relevant GDASApp issues.

[RussTreadon-NOAA]

GDASApp specific work for this issue is documented in GDASApp issue #357.

[RussTreadon-NOAA]

Work for this issue will be done in RussTreadon-NOAA:feature/refactor_jediatm. This branch was created as a copy of develop at 07fedaa.

[RussTreadon-NOAA]

@CoryMartin-NOAA, @guillaumevernieres , & @aerorahul

gdasatmanlinit successfully ran for 2021122100. It populated DATA=${path}/gdasatmanl_00 with files needed by fv3jedi_var.x. gdasatmanlrun failed. Before fixing this, I noticed something which I have a question about.

Can we implement the current init, run, finalize approach in operations? I thought operational jobs remove the run directory, $DATA, upon completion. If true, separate jobs can not reuse the same $DATA. The init job will remove $DATA upon completion. The run job will not have a pre-existing and populated $DATA in which to run.

Does NCO allow job run directories to remain on disk after a job successfully runs to completion?

[aerorahul]

This is from the NCO EE2 standards doc:

viii. Working directories
Working directories must contain a unique identifier (job id) unless there is an application need to share the directory across multiple jobs (e.g. a forecast job writing output that is needed by a post job running in parallel). Working directories must be removed upon successful completion of the run. All data that is needed for longer than one cycle must be copied to $COMOUT.

[guillaumevernieres]

@RussTreadon-NOAA , we clean the rundir in our post job and that rundir is used for several jobs (b-matrix and var). Seems OK from what @aerorahul just posted (. or ?)

[RussTreadon-NOAA]

Thanks @aerorahul . init, run, and finalize run successively not in parallel like the forecast & post example cited in the EE2 document. It seems our current approach does not align with EE2 standards.

[aerorahul]

You are right. I was focussing on the retention of the DATA directory between jobs.
I have a standing discussion w/ NCO on this serial execution of the init, run and finalize. The motivation is for appropriate use of compute resources. init and finalize are typically single core operations. run takes several hundred/thousand cores.

If we fail to convince NCO, the fall back would to combine the j-jobs into one (they are largely the same). It will entail some refactoring of the exscripts.

If you (and others) think there is too much uncertainty and risk in continuing this path, we can revert to our default operating procedure and abandon the 3 job approach.

Can we have a discussion offline?

[RussTreadon-NOAA]

It's best to come to an agreement with NCO before we venture too far down an unimplementable road. An offline discussion makes sense.

[RussTreadon-NOAA]

The text @aerorahul quoted is found on page 13 (IV.A.viii) of NCO WCOSS Implementation Standards (v11.0.0).

[RussTreadon-NOAA]

@CoryMartin-NOAA & @aerorahul

ush/python/pygfs/task/atm_analysis.py originated as a copy of aero_analysis.py. The yaml_copy section of the finalize function in atm_analysis.py aborts in ush/python/pygw/src/pygw/file_utils.py.

First, here's the section of atm_analysis.py in question

        # copy full YAML from executable to ROTDIR
        src = os.path.join(self.task_config['DATA'], f"{self.task_config['CDUMP']}.t{self.runtime_config['cyc']:02d}z.atmvar.yaml")
        dest = os.path.join(self.task_config['COMOUTatmos'], f"{self.task_config['CDUMP']}.t{self.runtime_config['cyc']:02d}z.atmvar.yaml")
        yaml_copy = {
            'mkdir': self.task_config['COMOUTatmos'],
            'copy': [src, dest]
        }
        FileHandler(yaml_copy).sync()

When executed, here's what we find in the job log file

Created /
Created s
Created c
Created r
Created a
Created t
Created c
Created h
Created 2
Created /
Created N
Created C
Created E
Created P
Created D
Created E
Created V
Created /
Created s
Created t
Created m
Created p
Created 1
Created /
Created R
Created u
Created s
Created s
Created .
Created T
Created r
Created e
Created a
Created d
Created o
Created n
Created /
Created c
Created o
Created m
Created r
Created o
Created t
Created /
Created p
Created r
Created u
Created f
Created s
Created a
Created t
Created m
Created /
Created g
Created d
Created a
Created s
Created .
Created 2
Created 0
Created 2
Created 1
Created 1
Created 2
Created 2
Created 1
Created /
Created 0
Created 0
Created /
Created a
Created t
Created m
Created o
Created s
 filelist is ['/scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml', '/scratch2/NCEPDEV/stmp1/Russ.Treadon/comrot/prufsatm/gdas.20211221/00/atmos/gdas.t00z.atmvar.yaml']
sublist is /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml
len(sublist) is 89
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/scripts/exglobal_atm_analysis_finalize.py", line 25, in <module>
    AtmAnl.finalize()
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/logger.py", line 261, in wrapper
    retval = func(*args, **kwargs)
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygfs/task/atm_analysis.py", line 177, in finalize
    FileHandler(yaml_copy).sync()
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 39, in sync
    sync_factory[action](files)
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 59, in _copy_files
    f"List must be of the form ['src', 'dest'], not {sublist}")
Exception: List must be of the form ['src', 'dest'], not /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml

I added print for filelist, sublist, and len(sublist) to file_utils.py for debug purposes. For some reason, sublist is not being processed as [src, dest] even though this is how it is constructed in yaml_copy.

Do we have a aerofinal job log file I can examine? I'd like to see why/how this sections works in aero_analysis.py but not atm_analysis.py.

[aerorahul]

@CoryMartin-NOAA & @aerorahul

ush/python/pygfs/task/atm_analysis.py originated as a copy of aero_analysis.py. The yaml_copy section of the finalize function in atm_analysis.py aborts in ush/python/pygw/src/pygw/file_utils.py.

First, here's the section of atm_analysis.py in question

        # copy full YAML from executable to ROTDIR

        src = os.path.join(self.task_config['DATA'], f"{self.task_config['CDUMP']}.t{self.runtime_config['cyc']:02d}z.atmvar.yaml")

        dest = os.path.join(self.task_config['COMOUTatmos'], f"{self.task_config['CDUMP']}.t{self.runtime_config['cyc']:02d}z.atmvar.yaml")

        yaml_copy = {

            'mkdir': self.task_config['COMOUTatmos'],

            'copy': [src, dest]

        }

        FileHandler(yaml_copy).sync()

When executed, here's what we find in the job log file

Created /

Created s

Created c

Created r

Created a

Created t

Created c

Created h

Created 2

Created /

Created N

Created C

Created E

Created P

Created D

Created E

Created V

Created /

Created s

Created t

Created m

Created p

Created 1

Created /

Created R

Created u

Created s

Created s

Created .

Created T

Created r

Created e

Created a

Created d

Created o

Created n

Created /

Created c

Created o

Created m

Created r

Created o

Created t

Created /

Created p

Created r

Created u

Created f

Created s

Created a

Created t

Created m

Created /

Created g

Created d

Created a

Created s

Created .

Created 2

Created 0

Created 2

Created 1

Created 1

Created 2

Created 2

Created 1

Created /

Created 0

Created 0

Created /

Created a

Created t

Created m

Created o

Created s

 filelist is ['/scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml', '/scratch2/NCEPDEV/stmp1/Russ.Treadon/comrot/prufsatm/gdas.20211221/00/atmos/gdas.t00z.atmvar.yaml']

sublist is /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml

len(sublist) is 89

Traceback (most recent call last):

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/scripts/exglobal_atm_analysis_finalize.py", line 25, in <module>

    AtmAnl.finalize()

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/logger.py", line 261, in wrapper

    retval = func(*args, **kwargs)

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygfs/task/atm_analysis.py", line 177, in finalize

    FileHandler(yaml_copy).sync()

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 39, in sync

    sync_factory[action](files)

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 59, in _copy_files

    f"List must be of the form ['src', 'dest'], not {sublist}")

Exception: List must be of the form ['src', 'dest'], not /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml

I added print for filelist, sublist, and len(sublist) to file_utils.py for debug purposes. For some reason, sublist is not being processed as [src, dest] even though this is how it is constructed in yaml_copy.

My simple theory is, mkdir needs a list. Try

yaml_copy = {

             'mkdir': [self.task_config['COMOUTatmos']],

             'copy': [src, dest]
 
         }

[RussTreadon-NOAA]

@aerorahul , thank you for the suggestion. The recommended change was made. The mkdir command works but copy still throws an error. The log file has

Created /scratch2/NCEPDEV/stmp1/Russ.Treadon/comrot/prufsatm/gdas.20211221/00/atmos
 filelist is ['/scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml', '/scratch2/NCEPDEV/stmp1/Russ.Treadon/comrot/prufsatm/gdas.20211221/00/atmos/gdas.t00z.atmvar.yaml']
sublist is /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml
len(sublist) is 89
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/scripts/exglobal_atm_analysis_finalize.py", line 25, in <module>
    AtmAnl.finalize()
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/logger.py", line 261, in wrapper
    retval = func(*args, **kwargs)
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygfs/task/atm_analysis.py", line 177, in finalize
    FileHandler(yaml_copy).sync()
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 39, in sync
    sync_factory[action](files)
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/workatm/ush/python/pygw/src/pygw/file_utils.py", line 59, in _copy_files
    f"List must be of the form ['src', 'dest'], not {sublist}")
Exception: List must be of the form ['src', 'dest'], not /scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/gdas.t00z.atmvar.yaml

We pass [src, dest] to copy as a list but it's being processed as src.

FileHandler works elsewhere but the yaml for those executions is structured a bit differently. For example, parm/parm_gdas/atm_crtm_coeff.yaml has

mkdir:
- !ENV ${DATA}/crtm/
copy:
# Emissivity files
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISice.EmisCoeff.bin
  - !ENV ${DATA}/crtm/NPOESS.VISice.EmisCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISland.EmisCoeff.bin
  - !ENV ${DATA}/crtm/NPOESS.VISland.EmisCoeff.bin

The crtm coefficient copy works.

[RussTreadon-NOAA]

@aerorahul , we can follow up next week. The atm init and run work. The hiccup with finalize should be easy to resolve. I just don't see the easy solution at present.