Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regional inlinepost #364

Merged
merged 63 commits into from
Mar 11, 2021

Conversation

junwang-noaa
Copy link
Collaborator

@junwang-noaa junwang-noaa commented Jan 12, 2021

Description

1)Code changes have been added for inline post capability for regional applications (HAFS and LAM). The supported output grids are "regional_latlon" for HAFS, "rotated_latlon" and "lambert_comformal" for LAM.
2) remove quilting restriction when generating forecast grid, which is used by esmf field creation for both coupled runs and quilting.
3) rt script is updated. The regional tests using same ICs will use the ICs in fv3_regional_control/INPUT, rather than using same set of INPUT data in each regional_test/INPUT.
4) add suite_FV3_GFS_v15_thompson_mynn.xml suite for LAM 3km parallel run
5) add timing information for each RT tests

Due to the upp update, the global inline post test results will change. Inline post results will be added to fv3_ccpp_regional_quilt and baseline results will be added to new test fv3_ccpp_regional_quilt_hafs. There is no result change for all other tests.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.

Testing

Testing has been done on hera,orion and dell with regional regression etst regional_quilt and regional_quilt_hafs tests and high resolution LAM and HAFS tests. The results have been verified against standalone post results.

RT is done on:

  • hera.gnu
  • hera.intel
  • jet.intel
  • gaea.intel
  • wcoss cray
  • wcoss dell

Note: Orion is really slow, the RT tests still stay in the job queue since this morning. I will make the merge after CI test is done, will have attach the log file when it is done.

Dependencies

Related PR:

  • fv3atm PR #229
  • post PR #255
  • This PR also requires a upp lib in hpc stack after post is merged.

@WenMeng-NOAA
Copy link
Contributor

@junwang-noaa Would you please update postxconfig-NT-fv3lam.txt with the latest UPP develop branch? The UPP side had the update for renaming FV3/LAM dataset names as PRSLEV and NATLEV. Thanks!

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jan 28, 2021

@BinLiu-NOAA @BenjaminBlake-NOAA @JamesAbeles-NOAA @WenMeng-NOAA To run regional inline post on hera, please do the following:

  1. checkout the regional_inlinepost branch:
    git clone --recursive --branch regional_inlinepost https://github.com/junwang-noaa/ufs-weather-model
  2. To build the model on hera:
    cd ufs-weather-model/tests
    module purge
    ./compile.sh hera.intel 'SUITES=FV3_GFS_v15_thompson_mynn 32BIT=Y' 1
  3. in the model_configure, add
    write_dopost: .true.
  4. copy the following files to your run directory:
    cd ufs-weather-model/tests/parm
    cp post_itag Your_run_directory/itag
    cp postxconfig-NT-fv3lam.txt Your_run_directory/postxconfig-NT.txt
    cp postxconfig-NT-fv3lam.txt Your_run_directory/postxconfig-NT_FH00.txt
    cp params_grib2_tbl_new Your_run_directory/params_grib2_tbl_new
    For HAFS, please copy HAFS post control file to postxconfig-NT.txt and postxconfig-NT_FH00.txt in your run directory.If you are using a different post control for first time step output, copy that file to postxconfig-NT_FH00.txt.
  5. submit job card to run the test.

A sample run directory is on hera at:
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_295210/fv3_ccpp_regional_quilt_prod

If you want to run current regional inline post test, after 1), please do
i) remove all other test in rt.conf but the regional quilt test which has the inline post capability:
COMPILE | SUITES=FV3_GFS_v15_thompson_mynn 32BIT=Y | | fv3 |
RUN | fv3_ccpp_regional_quilt | | fv3 |
@WenMeng-NOAA you can add the global post test in this file too:
COMPILE | SUITES=FV3_GFS_2017_gfdlmp,FV3_CPT_v0,FV3_GSD_v0,FV3_GFS_v15_thompson,FV3_RAP,FV3_HRRR,FV3_RRFS_v1beta 32BIT=Y | | fv3 |
RUN | fv3_ccpp_gfdlmprad_32bit_post | | fv3 |

ii) run ./rt.sh
You will have a test run directory that has lam (or GFS) post output.

So far I did limited tests with regional_latlon, roated_latlon and Lamber_conformal grids. Please let me know if you have any questions. Thanks

@WenMeng-NOAA
Copy link
Contributor

@junwang-noaa Could you add a new itag for regional in-line post which process 47 isobaric levels instead 57? You may find the itag used by off-line post at /scratch1/NCEPDEV/stmp2/Wen.Meng/post_fv3_2018101500 on hera.

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jan 28, 2021 via email

@WenMeng-NOAA
Copy link
Contributor

@junwang-noaa That's right. The 57 levels is for GFS only due to the model output 127 layers. The regional application still use 47(64 model layers). I would like to have @BenjaminBlake-NOAA or @ericaligo-NOAA to confirm from their parallel configuration.

@ericaligo-NOAA
Copy link
Collaborator

ericaligo-NOAA commented Jan 28, 2021 via email

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jan 28, 2021 via email

@BenjaminBlake-NOAA
Copy link

@junwang-noaa @WenMeng-NOAA This is what we are using for itag in the regional post, and it looks like we do use 47 isobaric levels.

&NAMPGB
KPO=47,PO=1000.,975.,950.,925.,900.,875.,850.,825.,800.,775.,750.,725.,700.,675.,650.,625.,600.,575.,550.,525.,500.,475.,450.,425.,400.,375.,350.,325.,300.,275.,250.,225.,200.,175.,150.,125.,100.,70.,50.,30.,20.,10.,7.,5.,3.,2.,1.,

@ericaligo-NOAA
Copy link
Collaborator

ericaligo-NOAA commented Jan 29, 2021 via email

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Jan 29, 2021 via email

@ericaligo-NOAA
Copy link
Collaborator

ericaligo-NOAA commented Jan 29, 2021 via email

@junwang-noaa
Copy link
Collaborator Author

@BrianCurtis-NOAA May I ask how orion RT is doing?

@BrianCurtis-NOAA
Copy link
Collaborator

@BrianCurtis-NOAA May I ask how orion RT is doing?

[!1018]bcurtis@Orion-login-2:/work/noaa/nems/emc.nemspara/autort/tests/auto$ squeue --user=bcurtis
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1550783 orion rt_31704 bcurtis PD 0:00 123 (Priority)
1551258 orion rt_31704 bcurtis PD 0:00 5 (Priority)
1551497 orion rt_31704 bcurtis PD 0:00 5 (Priority)
1551505 orion rt_31704 bcurtis PD 0:00 5 (Priority)
1551513 orion rt_31704 bcurtis PD 0:00 8 (Priority)
1551518 orion rt_31704 bcurtis PD 0:00 8 (Priority)
1551536 orion rt_31704 bcurtis PD 0:00 8 (Priority)
1551560 orion rt_31704 bcurtis PD 0:00 8 (Priority)
1551595 orion rt_31704 bcurtis PD 0:00 12 (Priority)
1551634 orion rt_31704 bcurtis PD 0:00 12 (Priority)
1551713 orion rt_31704 bcurtis PD 0:00 12 (Priority)
1551732 orion rt_31704 bcurtis PD 0:00 13 (Priority)
1551755 orion rt_31704 bcurtis PD 0:00 13 (Priority)
1551753 orion rt_31704 bcurtis PD 0:00 6 (Priority)
1551756 orion rt_31704 bcurtis PD 0:00 13 (Priority)
1551758 orion rt_31704 bcurtis PD 0:00 3 (Priority)
1551775 orion rt_31704 bcurtis PD 0:00 3 (Priority)
1551780 orion rt_31704 bcurtis PD 0:00 3 (Priority)
1551815 orion rt_31704 bcurtis PD 0:00 6 (Priority)
1551816 orion rt_31704 bcurtis PD 0:00 6 (Priority)
1551848 orion rt_31704 bcurtis PD 0:00 3 (Priority)
1552233 orion rt_31704 bcurtis PD 0:00 3 (Priority)
1552247 orion rt_31704 bcurtis PD 0:00 5 (Priority)
1552248 orion rt_31704 bcurtis PD 0:00 5 (Priority)

@junwang-noaa junwang-noaa merged commit e157c51 into ufs-community:develop Mar 11, 2021
AnningCheng-NOAA added a commit to AnningCheng-NOAA/ufs-weather-model that referenced this pull request Mar 16, 2021
* upstream/develop:
  Updatetemplate; Fix model slowness when using threading; Update WW3 for porting and threading issues (ufs-community#383)
  update MOM6 to GFDL 20210308 main branch commit (ufs-community#458)
  Regional inlinepost (ufs-community#364)
  correct benchmark diag_tables for coupled model configurations; move bm_ic directory out of inputdata directory; use aws ec2 for CI test; auto-rt fixes (ufs-community#426)
@junwang-noaa junwang-noaa deleted the regional_inlinepost branch April 20, 2021 18:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Baseline Updates Current baselines will be updated. Waiting for Reviews The PR is waiting for reviews from associated component PR's.
Projects
None yet
9 participants