Make atmos_fcst job work using GFSv17 and new COM structure #120

XianwuXue-NOAA · 2023-04-26T17:44:01Z

The job works on both WCOSS2 and HARA.

On branch feature/v13_atmos_fcst renamed: parm/gefs_fcst.parm -> parm/parm_gefs/gefs_fcst.parm Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: rocoto/bin/wcoss2/common.sh modified: rocoto/bin/wcoss2/forecast_hr.sh modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Also modify parm files On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/.gitignore new file: parm/config/config.ufs modified: parm/parm_gefs/gefs.parm modified: parm/parm_gefs/gefs_fcst.parm new file: parm/parm_gefs/gefs_fcst_resources.parm modified: sorc/link_gefs.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/.gitignore modified: parm/config/config.ufs modified: parm/parm_gefs/gefs.parm modified: parm/parm_gefs/gefs_fcst.parm modified: rocoto/parm/setbase modified: rocoto/py/GEFS_XML_For_Tasks.py modified: rocoto/user_full.conf modified: scripts/exgefs_forecast.sh modified: sorc/link_gefs.sh modified: ush/.gitignore Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: rocoto/user_full.conf Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/parm_gefs/gefs_fcst.parm modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: rocoto/bin/hera/forecast_hr.sh modified: rocoto/bin/wcoss2/forecast_hr.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: rocoto/bin/hera/forecast_hr.sh modified: rocoto/parm/setbase Refs: NOAA-EMC#94

XianwuXue-NOAA · 2023-04-26T17:56:46Z

parm/config/config.ufs

+#export jchunk2d=$((2*RESTILE))
+#export ichunk3d=$((4*RESTILE))
+#export jchunk3d=$((2*RESTILE))
+#export kchunk3d=1


Lines 210~252 are for GEFS fcst job, especially for d16_35 and chem

XianwuXue-NOAA · 2023-04-26T18:30:53Z

parm/parm_gefs/gefs_fcst.parm

+elif [[ "$imp_physics" -eq 11 ]]; then # GFDL
+    export ncld=5
+    if [[ $cplchm = ".true." ]]; then
+        export FIELD_TABLE="$HOMEgfs/parm/parm_fv3diag/chm_field_table_gfdl"


This is for chem, maybe we don't use it in gefs.v13

XianwuXue-NOAA · 2023-04-26T18:32:20Z

parm/parm_gefs/gefs_fcst.parm

+# Disable the use of coupler.res; get model start time from model_configure
+export USE_COUPLER_RES="NO"
+
+    # Write more variables to output


In original config.fcst, there is no condition for gefs

@WalterKolczynski-NOAA @aerorahul

XianwuXue-NOAA · 2023-04-26T19:11:48Z

@WalterKolczynski-NOAA @aerorahul Could you please review my changes for fcst job? I have a few comments for the config files, could you also give me some suggestions? I have ran c00, p01 and p02 (C48) succeeded on Hera, but failed for p02 on WCOSS2 because of instability of the model.

On branch feature/v13_atmos_fcst modified: parm/parm_gefs/gefs_fcst.parm modified: rocoto/user_full.conf Refs: NOAA-EMC#94

WalterKolczynski-NOAA

Instead of duplicating the contents of GFS config files to make a few changes, you should be making changes to the config files in global workflow to support GEFS and/or adding config files that just change the things that are different. We want GEFS to run more more like GFS.

jobs/JGEFS_FORECAST

scripts/exgefs_forecast.sh

WalterKolczynski-NOAA · 2023-04-27T05:33:24Z

scripts/exgefs_forecast.sh

Much of this should be in a config file.

in config.fcst in g-w? If it is, you and @aerorahul should determine can I use cdump=gefs? or only use RUN=gefs? If you want to use run=gefs, then we should modify the config* in g-w to use RUN instead of cdump

Per the slack and in-person conversations, make GEFS versions of the config files and then make sure these settings are defined in there.

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

XianwuXue-NOAA · 2023-04-27T15:42:02Z

@WalterKolczynski-NOAA I have submitted a few commits based on your comment. When you have time, please review it.

On branch feature/v13_atmos_fcst renamed: parm/parm_gefs/gefs_fcst_resources.parm -> parm/config/config.resources modified: parm/parm_gefs/gefs_fcst.parm Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/parm_gefs/gefs_fcst.parm modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_ATMOS_PREP modified: jobs/JGEFS_FORECAST renamed: parm/parm_gefs/gefs.parm -> parm/config/config.base renamed: parm/parm_gefs/gefs_fcst.parm -> parm/config/config.fcst renamed: parm/parm_gefs/gefs_atmos_prep.parm -> parm/config/config.prep Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_ATMOS_PREP Refs: NOAA-EMC#94

The current g-w can not be used as flat structure for GEFS so I change the default structure is vertical structure On branch feature/v13_atmos_fcst modified: rocoto/compile_install_all.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: rocoto/compile_install_all.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

after delete them On branch feature/v13_atmos_fcst Changes to be committed: modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst Changes to be committed: modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.base modified: parm/config/config.fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

XianwuXue-NOAA · 2023-04-28T16:42:03Z

@WalterKolczynski-NOAA @aerorahul I have moved some vars from ex-script of fcst to config.fcst and also did other modification. Please review my changes. Thanks

XianwuXue-NOAA · 2023-05-01T19:19:11Z

@WalterKolczynski-NOAA @aerorahul When you have time, could you review my changes for this PR? Thanks.

On branch feature/v13_atmos_fcst modified: sorc/checkout.sh Refs: NOAA-EMC#94

WalterKolczynski-NOAA

Didn't complete the review of the config files. It seems like a lot of GEFS settings were tacked on to GFS settings without considering whether they are still needed or to update them to match the GFS style (for instance, using $cplchm for branching instead of $DO_AERO). Many settings have been updated for GFS v17/GEFS v13 and we should defer to the new settings as given in GFS unless there is a good reason to deviate (like turning on stochastic physics for some members). Please carefully review all of the settings and determine whether it is conflicting with an existing setting elsewhere. If you have any trouble determining whether a setting should be overridden for GEFS, I'll help and if necessary we'll ask MDAB.

Also, note the changes that were made in the NOAA-EMC/global-workflow#1509. You've updated the commit hash, but there are changes to the config files that will need to be duplicated here.

WalterKolczynski-NOAA · 2023-05-02T04:45:43Z

jobs/JGEFS_FORECAST

-export GESIN=${GESIN:-$(compath.py $envir/com/$NET/${ver})/${RUN}.${PDY}/$cyc/nwges}
-export GESOUT=${GESOUT:-$(compath.py -o $NET/${ver})/${RUN}.${PDY}/$cyc/nwges}
+ver=${gefs_ver:0:5}
+export ROTDIR=${ROTDIR:-$(compath.py $envir/com/$NET/${ver})}


This should be defined before the j-job is called, and definitely should not use compath.py. compath.py should only be used for external packages (GFS, tracker, etc.).

Is it the NCO new requirements? Because all the compath.py are in j-job in the current production.

jobs/JGEFS_FORECAST

parm/config/config.base

WalterKolczynski-NOAA · 2023-05-02T05:00:25Z

parm/config/config.fcst

+fi
+
+# Choose coupling with wave
+if [[ "$DO_WAVE" = "YES" && "$WAVE_CDUMP" != "gdas" ]]; then


Suggested change

if [[ "$DO_WAVE" = "YES" && "$WAVE_CDUMP" != "gdas" ]]; then

if [[ "$DO_WAVE" = "YES" ]]; then

This is the copy of code segments from config.fcst in g-w, I will follow the the config.fcst of g-w changes

parm/config/config.fcst

WalterKolczynski-NOAA · 2023-05-02T05:15:57Z

rocoto/bin/hera/forecast_hr.sh


+export ROTDIR=${COMROOT}/gefs/${ver}


For development, I don't think we need to add the additional unnecessary directory levels that NCO is.

Suggested change

export ROTDIR=${COMROOT}/gefs/${ver}

# Assuming $EXPDIR is already built into $COMROOT

ROTDIR=${COMROOT}

GEFS will not use EXPDIR, This line is to set ROTDIR because I call j-job. This is only be used on Hera.

WalterKolczynski-NOAA · 2023-05-02T05:16:58Z

rocoto/bin/hera/forecast_hr.sh


+export ROTDIR=${COMROOT}/gefs/${ver}
+export ROTDIR_GFS=${HOMEdata}/gfs/${gfs_ver}


Same here. For GFS development we don't include the extra directory levels NCO does. For operations, this needs to use compath.py, since GFS is an external package.

Yes, however, this is only used on Hera. On wcos2, I will use compath.py to get the path, but it can not used on Hera.

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/config/config.base modified: parm/config/config.fcst modified: parm/config/config.resources modified: parm/config/config.ufs Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/config/config.base Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst modified: parm/config/config.ufs modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.resources Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST new file: parm/config/config.efcs modified: parm/config/config.fcst Refs: NOAA-EMC#94

XianwuXue-NOAA · 2023-05-03T00:08:40Z

@WalterKolczynski-NOAA @aerorahul I have decided the config.fcst into two files: config.fcst and config.efcs. Which will shorten the config.fcst. Please re-review my changes for this PR. I think I have resolved all the comments. If I missed something, please let me know.

On branch feature/v13_atmos_fcst modified: parm/config/config.efcs Refs: NOAA-EMC#94

XianwuXue-NOAA · 2023-05-04T22:54:32Z

Since the GEFSv13 workflow will be in g-w, I will merge this PR to close this PR and for the future's reference. I will do the same PR for the other jobs. If you have any suggestions, please let me know. @WalterKolczynski-NOAA @aerorahul

XianwuXue-NOAA added 11 commits April 17, 2023 21:52

Move gefs_fcst.part to parm_gefs

dbd0dda

On branch feature/v13_atmos_fcst renamed: parm/gefs_fcst.parm -> parm/parm_gefs/gefs_fcst.parm Refs: NOAA-EMC#94

Make changes based on previous changes

a0aca62

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: rocoto/bin/wcoss2/common.sh modified: rocoto/bin/wcoss2/forecast_hr.sh modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Use preamble.sh and add lines from previous changes

cfd1e1e

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Merge branch 'feature/v13_atm_only' into feature/v13_atmos_fcst

2ba5729

Optimize and clean ex-script of fcst

d2e5212

On branch feature/v13_atmos_fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Remove imp_physics in the user_full.conf

31877d0

On branch feature/v13_atmos_fcst modified: rocoto/user_full.conf Refs: NOAA-EMC#94

Improve and cleanup j-job, ex-script of fcst

28ff82b

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/parm_gefs/gefs_fcst.parm modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Improve bin file of forecast

3de25db

On branch feature/v13_atmos_fcst modified: rocoto/bin/hera/forecast_hr.sh modified: rocoto/bin/wcoss2/forecast_hr.sh Refs: NOAA-EMC#94

Modify to run fcst on Hera

30036f2

On branch feature/v13_atmos_fcst modified: rocoto/bin/hera/forecast_hr.sh modified: rocoto/parm/setbase Refs: NOAA-EMC#94

XianwuXue-NOAA commented Apr 26, 2023

View reviewed changes

XianwuXue-NOAA requested a review from WalterKolczynski-NOAA April 26, 2023 18:21

XianwuXue-NOAA self-assigned this Apr 26, 2023

XianwuXue-NOAA added the GEFSv13 label Apr 26, 2023

XianwuXue-NOAA added this to the GEFSv13 milestone Apr 26, 2023

XianwuXue-NOAA requested a review from aerorahul April 26, 2023 18:22

XianwuXue-NOAA marked this pull request as ready for review April 26, 2023 18:22

XianwuXue-NOAA commented Apr 26, 2023

View reviewed changes

Modify stochastic parameters

158cb0d

On branch feature/v13_atmos_fcst modified: parm/parm_gefs/gefs_fcst.parm modified: rocoto/user_full.conf Refs: NOAA-EMC#94

WalterKolczynski-NOAA requested changes Apr 27, 2023

View reviewed changes

Modify based on Walter's Comments

b3b49eb

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

XianwuXue-NOAA added 5 commits April 27, 2023 14:20

Rename gefs_fcst_resources.parm to config.resources

1ca635c

On branch feature/v13_atmos_fcst renamed: parm/parm_gefs/gefs_fcst_resources.parm -> parm/config/config.resources modified: parm/parm_gefs/gefs_fcst.parm Refs: NOAA-EMC#94

Clean up some unused statements

c7271ab

On branch feature/v13_atmos_fcst modified: parm/parm_gefs/gefs_fcst.parm modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Fix a bug in j-job of atmos_prep

a76a74b

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_ATMOS_PREP Refs: NOAA-EMC#94

Make the vertical structure as default value

7af6724

The current g-w can not be used as flat structure for GEFS so I change the default structure is vertical structure On branch feature/v13_atmos_fcst modified: rocoto/compile_install_all.sh Refs: NOAA-EMC#94

XianwuXue-NOAA added 7 commits April 27, 2023 19:00

Improve the compile_install_all.sh

aeb3f97

On branch feature/v13_atmos_fcst modified: rocoto/compile_install_all.sh Refs: NOAA-EMC#94

Reformat config.fcst

12f5b14

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

Add VEGTYPE fix file back since p02 failed on hera

5ccc901

after delete them On branch feature/v13_atmos_fcst Changes to be committed: modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Add SEND=NO back

5da028d

On branch feature/v13_atmos_fcst Changes to be committed: modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Move some vars from ex-script to config.fcst

fa53f4f

On branch feature/v13_atmos_fcst modified: parm/config/config.base modified: parm/config/config.fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Fix a minor bug

c772da4

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

Move vars from ex-script of fcst to config.fcst

50f39bf

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

XianwuXue-NOAA mentioned this pull request Apr 28, 2023

Change c00, pXX to memXXX #123

Merged

Update g-w commit

d24d281

On branch feature/v13_atmos_fcst modified: sorc/checkout.sh Refs: NOAA-EMC#94

WalterKolczynski-NOAA requested changes May 2, 2023

View reviewed changes

XianwuXue-NOAA added 6 commits May 2, 2023 09:40

Modify based on Walter's comments and g-w changes

1379fcd

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/config/config.base modified: parm/config/config.fcst modified: parm/config/config.resources modified: parm/config/config.ufs Refs: NOAA-EMC#94

Modify based on reviewer's comments

5dbb48e

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST modified: parm/config/config.base Refs: NOAA-EMC#94

Modify the config files

eaa59b1

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst modified: parm/config/config.ufs modified: scripts/exgefs_forecast.sh Refs: NOAA-EMC#94

Fix a bug to run fcst

0d3371a

On branch feature/v13_atmos_fcst modified: parm/config/config.resources Refs: NOAA-EMC#94

Remove the dead codes

c19fadd

On branch feature/v13_atmos_fcst modified: parm/config/config.fcst Refs: NOAA-EMC#94

Seperate config.fcst to config.fcst & config.efcs

e5c19d0

On branch feature/v13_atmos_fcst modified: jobs/JGEFS_FORECAST new file: parm/config/config.efcs modified: parm/config/config.fcst Refs: NOAA-EMC#94

Fix a bug

3af9001

On branch feature/v13_atmos_fcst modified: parm/config/config.efcs Refs: NOAA-EMC#94

XianwuXue-NOAA merged commit 2970fdf into NOAA-EMC:feature/v13_atm_only May 4, 2023

XianwuXue-NOAA deleted the feature/v13_atmos_fcst branch May 4, 2023 23:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make atmos_fcst job work using GFSv17 and new COM structure #120

Make atmos_fcst job work using GFSv17 and new COM structure #120

XianwuXue-NOAA commented Apr 26, 2023

XianwuXue-NOAA Apr 26, 2023

XianwuXue-NOAA Apr 26, 2023

XianwuXue-NOAA Apr 26, 2023

XianwuXue-NOAA Apr 26, 2023

XianwuXue-NOAA commented Apr 26, 2023

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Apr 27, 2023

XianwuXue-NOAA Apr 27, 2023

WalterKolczynski-NOAA Apr 27, 2023

XianwuXue-NOAA commented Apr 27, 2023

XianwuXue-NOAA commented Apr 28, 2023

XianwuXue-NOAA commented May 1, 2023

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA May 2, 2023

XianwuXue-NOAA May 2, 2023

WalterKolczynski-NOAA May 2, 2023

XianwuXue-NOAA May 2, 2023

WalterKolczynski-NOAA May 2, 2023

XianwuXue-NOAA May 2, 2023

WalterKolczynski-NOAA May 2, 2023

XianwuXue-NOAA May 2, 2023

XianwuXue-NOAA commented May 3, 2023

XianwuXue-NOAA commented May 4, 2023

	if [[ "$DO_WAVE" = "YES" && "$WAVE_CDUMP" != "gdas" ]]; then
	if [[ "$DO_WAVE" = "YES" ]]; then

	export ROTDIR=${COMROOT}/gefs/${ver}
	# Assuming $EXPDIR is already built into $COMROOT
	ROTDIR=${COMROOT}


		export ROTDIR=${COMROOT}/gefs/${ver}
		export ROTDIR_GFS=${HOMEdata}/gfs/${gfs_ver}

Make atmos_fcst job work using GFSv17 and new COM structure #120

Make atmos_fcst job work using GFSv17 and new COM structure #120

Conversation

XianwuXue-NOAA commented Apr 26, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XianwuXue-NOAA commented Apr 26, 2023

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XianwuXue-NOAA commented Apr 27, 2023

XianwuXue-NOAA commented Apr 28, 2023

XianwuXue-NOAA commented May 1, 2023

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XianwuXue-NOAA commented May 3, 2023

XianwuXue-NOAA commented May 4, 2023