Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ocean/ice product generation for GFS and GEFS #2286

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
a8e12f6
wip
aerorahul Dec 29, 2023
b862705
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 8, 2024
9f62533
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 8, 2024
9f4533a
work cice output and ice filenames
aerorahul Jan 10, 2024
d7e00b2
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 10, 2024
90fab15
wip
aerorahul Jan 12, 2024
e9de66f
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 17, 2024
3b54968
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 18, 2024
b89f539
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 22, 2024
43232fc
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 22, 2024
76d09bc
Write mediator restarts to CMEPS_RESTART/ instead of RESTART/
aerorahul Jan 22, 2024
f116a32
link ocean/ice post-processing stuff from gfs-utils
aerorahul Jan 22, 2024
33dae8f
add a rocoto job, jjob, config file and resources for ocean/ice produ…
aerorahul Jan 22, 2024
cd7b7be
add oceanice_products.yaml and declare ocean/ice grib2 directory name…
aerorahul Jan 23, 2024
c922dea
working ocean products
aerorahul Jan 31, 2024
fa610ed
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 31, 2024
552bb7b
updates to workflow generation scripts
aerorahul Feb 1, 2024
e7d8468
clean-up
aerorahul Feb 1, 2024
a4d850c
fixes
aerorahul Feb 2, 2024
c619784
update hash of gfs-utils submodule
aerorahul Feb 2, 2024
d1d415f
fix shellnorm and update CMEPS_out to only copy the last mediator res…
aerorahul Feb 2, 2024
6525773
remove ocnpost from workflow
aerorahul Feb 2, 2024
3a27424
update archive script to archive ocean/ice output and products
aerorahul Feb 2, 2024
82494f0
increase UPP memory for C768
aerorahul Feb 2, 2024
e387208
update env files for other machines
aerorahul Feb 2, 2024
c5f84aa
Apply suggestions from code review - batch 1
aerorahul Feb 5, 2024
5da925d
remove unnecessary redefinitions in predet
aerorahul Feb 5, 2024
d232b4d
fix shellcheck errors
aerorahul Feb 5, 2024
4abdd40
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 5, 2024
8eb84e7
adapt for ocean/ice output frequency
aerorahul Feb 5, 2024
2a71a27
consolidate CICE output links to COM for gfs/gdas
aerorahul Feb 5, 2024
8f2bb7c
rename {component}{prod} to {component}_{prod}. add ocean/ice produc…
aerorahul Feb 5, 2024
4a5ded9
add ocean ice prodgen for gefs
aerorahul Feb 5, 2024
f0c76f5
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 5, 2024
1aa7197
fix missing level from copy/paste in the oceanice_products.yaml
aerorahul Feb 5, 2024
26b0a9a
update metadata for date and forecast hour in the grib2 product file
aerorahul Feb 5, 2024
ce20bee
add missing key in the local task dict
aerorahul Feb 5, 2024
1b336e2
remove xsect ocean vars and add COM_OCEAN_NETCDF for netcdf products.
aerorahul Feb 7, 2024
de32912
placeholder for ocean netcdf products
aerorahul Feb 7, 2024
fdb9007
apply comments from review to replace SALTY with SALIN and remove sca…
aerorahul Feb 8, 2024
a7822c2
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 8, 2024
c54a4fa
add COM_ICE_NETCDF_TMPL for any ice netcdf products that we may have …
aerorahul Feb 9, 2024
97f99fc
perform subsetting of model native output for products
aerorahul Feb 9, 2024
7de1739
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 12, 2024
abea49d
remove not required variables from ocean native product files
aerorahul Feb 12, 2024
30fc5f7
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 12, 2024
16fd4ee
remove sst_h from ice native netcdf products. fix gefs dependency for…
aerorahul Feb 12, 2024
2f4e17b
missed a couple dependencies for gefs products
aerorahul Feb 12, 2024
0579eb6
fix undefined variable in forecast_postdet.sh that impacted RUN=gdas
aerorahul Feb 12, 2024
c9fde0e
update the dependency for prepoceanobs for updated filename
aerorahul Feb 12, 2024
9930d05
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
rename {component}{prod} to {component}_{prod}. add ocean/ice product…
…s for RUN=gfs in cycled
  • Loading branch information
aerorahul committed Feb 5, 2024
commit 8f2bb7c730b3e70b5b16d43ace8bb71bb375f6de
2 changes: 1 addition & 1 deletion workflow/applications/gefs.py
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,6 @@ def get_task_names(self):
if self.nens > 0:
tasks += ['efcs']

tasks += ['atmosprod']
tasks += ['atmos_prod']

return {f"{self._base['CDUMP']}": tasks}
10 changes: 8 additions & 2 deletions workflow/applications/gfs_cycled.py
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ def get_task_names(self):

if self.do_upp:
gdas_tasks += ['atmupp']
gdas_tasks += ['atmosprod']
gdas_tasks += ['atmos_prod']

if self.do_wave and 'gdas' in self.wave_cdumps:
if self.do_wave_bnd:
Expand Down Expand Up @@ -207,9 +207,15 @@ def get_task_names(self):

gfs_tasks += ['atmanlupp', 'atmanlprod', 'fcst']

if self.do_ocean:
gfs_tasks += ['ocean_prod']

if self.do_ice:
gfs_tasks += ['ice_prod']

if self.do_upp:
gfs_tasks += ['atmupp']
gfs_tasks += ['atmosprod']
gfs_tasks += ['atmos_prod']

if self.do_goes:
gfs_tasks += ['goesupp']
Expand Down
6 changes: 3 additions & 3 deletions workflow/applications/gfs_forecast_only.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ def get_task_names(self):
if self.do_upp:
tasks += ['atmupp']

tasks += ['atmosprod']
tasks += ['atmos_prod']

if self.do_goes:
tasks += ['goesupp']
Expand All @@ -127,10 +127,10 @@ def get_task_names(self):
tasks += ['awips_20km_1p0deg', 'awips_g2', 'fbwind']

if self.do_ocean:
tasks += ['oceanprod']
tasks += ['ocean_prod']

if self.do_ice:
tasks += ['iceprod']
tasks += ['ice_prod']

if self.do_wave:
if self.do_wave_bnd:
Expand Down
14 changes: 7 additions & 7 deletions workflow/rocoto/gefs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -159,29 +159,29 @@ def efcs(self):

return task

def atmosprod(self):
def atmos_prod(self):
atm_master_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_MASTER_TMPL"], {'MEMDIR': 'mem#member#'})
deps = []
data = f'{atm_master_path}/{self.cdump}.t@Hz.master.grb2f#fhr#'
dep_dict = {'type': 'data', 'data': data, 'age': 120}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

atmosprod_envars = self.envars.copy()
atmos_prod_envars = self.envars.copy()
postenvar_dict = {'ENSMEM': '#member#',
'MEMDIR': 'mem#member#',
'FHRLST': '#fhr#',
}
for key, value in postenvar_dict.items():
atmosprod_envars.append(rocoto.create_envar(name=key, value=str(value)))
atmos_prod_envars.append(rocoto.create_envar(name=key, value=str(value)))

resources = self.get_resource('atmos_products')

task_name = f'atmosprod_mem#member#_f#fhr#'
task_name = f'atmos_prod_mem#member#_f#fhr#'
task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
'envars': atmosprod_envars,
'envars': atmos_prod_envars,
'cycledef': 'gefs',
'command': f'{self.HOMEgfs}/jobs/rocoto/atmos_products.sh',
'job_name': f'{self.pslot}_{task_name}_@H',
Expand All @@ -191,13 +191,13 @@ def atmosprod(self):

fhr_var_dict = {'fhr': ' '.join([str(fhr).zfill(3) for fhr in
self._get_forecast_hours('gefs', self._configs['atmos_products'])])}
fhr_metatask_dict = {'task_name': 'atmosprod_#member#',
fhr_metatask_dict = {'task_name': 'atmos_prod_#member#',
'task_dict': task_dict,
'var_dict': fhr_var_dict
}

member_var_dict = {'member': ' '.join([str(mem).zfill(3) for mem in range(0, self.nmem + 1)])}
member_metatask_dict = {'task_name': 'atmosprod',
member_metatask_dict = {'task_name': 'atmos_prod',
'task_dict': fhr_metatask_dict,
'var_dict': member_var_dict
}
Expand Down
48 changes: 24 additions & 24 deletions workflow/rocoto/gfs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ def prep(self):
gfs_enkf = True if self.app_config.do_hybvar and 'gfs' in self.app_config.eupd_cdumps else False

deps = []
dep_dict = {'type': 'metatask', 'name': 'gdasatmosprod', 'offset': f"-{timedelta_to_HMS(self._base['cycle_interval'])}"}
dep_dict = {'type': 'metatask', 'name': 'gdasatmos_prod', 'offset': f"-{timedelta_to_HMS(self._base['cycle_interval'])}"}
deps.append(rocoto.add_dependency(dep_dict))
data = f'{atm_hist_path}/gdas.t@Hz.atmf009.nc'
dep_dict = {'type': 'data', 'data': data, 'offset': f"-{timedelta_to_HMS(self._base['cycle_interval'])}"}
Expand Down Expand Up @@ -1014,13 +1014,13 @@ def _upptask(self, upp_run="forecast", task_id="atmupp"):

return task

def atmosprod(self):
def atmos_prod(self):
return self._atmosoceaniceprod('atmos')

def oceanprod(self):
def ocean_prod(self):
return self._atmosoceaniceprod('ocean')

def iceprod(self):
def ice_prod(self):
return self._atmosoceaniceprod('ice')

def _atmosoceaniceprod(self, component: str):
Expand Down Expand Up @@ -1058,7 +1058,7 @@ def _atmosoceaniceprod(self, component: str):
cycledef = 'gdas_half,gdas' if self.cdump in ['gdas'] else self.cdump
resources = self.get_resource(component_dict['config'])

task_name = f'{self.cdump}{component}prod#{varname1}#'
task_name = f'{self.cdump}{component}_prod#{varname1}#'
task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
Expand Down Expand Up @@ -1349,7 +1349,7 @@ def _get_awipsgroups(cdump, config):
def awips_20km_1p0deg(self):

deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand Down Expand Up @@ -1390,7 +1390,7 @@ def awips_20km_1p0deg(self):
def awips_g2(self):

deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand Down Expand Up @@ -1431,7 +1431,7 @@ def awips_g2(self):
def gempak(self):

deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1454,7 +1454,7 @@ def gempak(self):

def gempakmeta(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1477,7 +1477,7 @@ def gempakmeta(self):

def gempakmetancdc(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1500,7 +1500,7 @@ def gempakmetancdc(self):

def gempakncdcupapgif(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand Down Expand Up @@ -1639,7 +1639,7 @@ def vminmon(self):

def tracker(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1662,7 +1662,7 @@ def tracker(self):

def genesis(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1685,7 +1685,7 @@ def genesis(self):

def genesis_fsu(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1708,7 +1708,7 @@ def genesis_fsu(self):

def fit2obs(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand Down Expand Up @@ -1773,7 +1773,7 @@ def metp(self):

def mos_stn_prep(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1796,7 +1796,7 @@ def mos_stn_prep(self):

def mos_grd_prep(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1819,7 +1819,7 @@ def mos_grd_prep(self):

def mos_ext_stn_prep(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand All @@ -1842,7 +1842,7 @@ def mos_ext_stn_prep(self):

def mos_ext_grd_prep(self):
deps = []
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

Expand Down Expand Up @@ -2160,7 +2160,7 @@ def arch(self):
dep_dict = {'type': 'task', 'name': f'{self.cdump}genesis_fsu'}
deps.append(rocoto.add_dependency(dep_dict))
# Post job dependencies
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmosprod'}
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}atmos_prod'}
deps.append(rocoto.add_dependency(dep_dict))
if self.app_config.do_wave:
dep_dict = {'type': 'task', 'name': f'{self.cdump}wavepostsbs'}
Expand All @@ -2171,12 +2171,12 @@ def arch(self):
dep_dict = {'type': 'task', 'name': f'{self.cdump}wavepostbndpnt'}
deps.append(rocoto.add_dependency(dep_dict))
if self.app_config.do_ocean:
if self.app_config.mode in ['forecast-only']:
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}oceanprod'}
if self.cdump in ['gfs']:
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}ocean_prod'}
deps.append(rocoto.add_dependency(dep_dict))
if self.app_config.do_ice:
if self.app_config.mode in ['forecast-only']:
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}iceprod'}
if self.cdump in ['gfs']:
dep_dict = {'type': 'metatask', 'name': f'{self.cdump}ice_prod'}
deps.append(rocoto.add_dependency(dep_dict))
# MOS job dependencies
if self.cdump in ['gfs'] and self.app_config.do_mos:
Expand Down
Loading