Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ocean/ice product generation for GFS and GEFS #2286

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
a8e12f6
wip
aerorahul Dec 29, 2023
b862705
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 8, 2024
9f62533
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 8, 2024
9f4533a
work cice output and ice filenames
aerorahul Jan 10, 2024
d7e00b2
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 10, 2024
90fab15
wip
aerorahul Jan 12, 2024
e9de66f
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 17, 2024
3b54968
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 18, 2024
b89f539
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 22, 2024
43232fc
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 22, 2024
76d09bc
Write mediator restarts to CMEPS_RESTART/ instead of RESTART/
aerorahul Jan 22, 2024
f116a32
link ocean/ice post-processing stuff from gfs-utils
aerorahul Jan 22, 2024
33dae8f
add a rocoto job, jjob, config file and resources for ocean/ice produ…
aerorahul Jan 22, 2024
cd7b7be
add oceanice_products.yaml and declare ocean/ice grib2 directory name…
aerorahul Jan 23, 2024
c922dea
working ocean products
aerorahul Jan 31, 2024
fa610ed
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Jan 31, 2024
552bb7b
updates to workflow generation scripts
aerorahul Feb 1, 2024
e7d8468
clean-up
aerorahul Feb 1, 2024
a4d850c
fixes
aerorahul Feb 2, 2024
c619784
update hash of gfs-utils submodule
aerorahul Feb 2, 2024
d1d415f
fix shellnorm and update CMEPS_out to only copy the last mediator res…
aerorahul Feb 2, 2024
6525773
remove ocnpost from workflow
aerorahul Feb 2, 2024
3a27424
update archive script to archive ocean/ice output and products
aerorahul Feb 2, 2024
82494f0
increase UPP memory for C768
aerorahul Feb 2, 2024
e387208
update env files for other machines
aerorahul Feb 2, 2024
c5f84aa
Apply suggestions from code review - batch 1
aerorahul Feb 5, 2024
5da925d
remove unnecessary redefinitions in predet
aerorahul Feb 5, 2024
d232b4d
fix shellcheck errors
aerorahul Feb 5, 2024
4abdd40
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 5, 2024
8eb84e7
adapt for ocean/ice output frequency
aerorahul Feb 5, 2024
2a71a27
consolidate CICE output links to COM for gfs/gdas
aerorahul Feb 5, 2024
8f2bb7c
rename {component}{prod} to {component}_{prod}. add ocean/ice produc…
aerorahul Feb 5, 2024
4a5ded9
add ocean ice prodgen for gefs
aerorahul Feb 5, 2024
f0c76f5
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 5, 2024
1aa7197
fix missing level from copy/paste in the oceanice_products.yaml
aerorahul Feb 5, 2024
26b0a9a
update metadata for date and forecast hour in the grib2 product file
aerorahul Feb 5, 2024
ce20bee
add missing key in the local task dict
aerorahul Feb 5, 2024
1b336e2
remove xsect ocean vars and add COM_OCEAN_NETCDF for netcdf products.
aerorahul Feb 7, 2024
de32912
placeholder for ocean netcdf products
aerorahul Feb 7, 2024
fdb9007
apply comments from review to replace SALTY with SALIN and remove sca…
aerorahul Feb 8, 2024
a7822c2
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 8, 2024
c54a4fa
add COM_ICE_NETCDF_TMPL for any ice netcdf products that we may have …
aerorahul Feb 9, 2024
97f99fc
perform subsetting of model native output for products
aerorahul Feb 9, 2024
7de1739
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 12, 2024
abea49d
remove not required variables from ocean native product files
aerorahul Feb 12, 2024
30fc5f7
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 12, 2024
16fd4ee
remove sst_h from ice native netcdf products. fix gefs dependency for…
aerorahul Feb 12, 2024
2f4e17b
missed a couple dependencies for gefs products
aerorahul Feb 12, 2024
0579eb6
fix undefined variable in forecast_postdet.sh that impacted RUN=gdas
aerorahul Feb 12, 2024
c9fde0e
update the dependency for prepoceanobs for updated filename
aerorahul Feb 12, 2024
9930d05
Merge branch 'develop' into feature/ocean_ice_output
aerorahul Feb 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add ocean ice prodgen for gefs
  • Loading branch information
aerorahul committed Feb 5, 2024
commit 4a5ded92301b5f7844687997031cf80983c6498b
1 change: 1 addition & 0 deletions parm/config/gefs/config.oceanice_products
8 changes: 8 additions & 0 deletions parm/config/gefs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -224,6 +224,14 @@ case ${step} in
export is_exclusive=True
;;

"oceanice_products")
export wtime_oceanice_products="00:15:00"
export npe_oceanice_products=1
export npe_node_oceanice_products=1
export nth_oceanice_products=1
export memory_oceanice_products="96GB"
;;

"stage_ic")
export wtime_stage_ic="00:15:00"
export npe_stage_ic=1
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.resources
Original file line number Diff line number Diff line change
Expand Up @@ -645,7 +645,7 @@ case ${step} in
;;

"oceanice_products")
export wtime_oceanice_products="00:30:00"
export wtime_oceanice_products="00:15:00"
export npe_oceanice_products=1
export npe_node_oceanice_products=1
export nth_oceanice_products=1
Expand Down
9 changes: 9 additions & 0 deletions workflow/applications/gefs.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ def _get_app_configs(self):
if self.do_wave:
configs += ['waveinit']

if self.do_ocean or self.do_ice:
configs += ['oceanice_products']

return configs

@staticmethod
Expand All @@ -47,4 +50,10 @@ def get_task_names(self):

tasks += ['atmos_prod']

if self.do_ocean:
tasks += ['ocean_prod']

if self.do_ice:
tasks += ['ice_prod']

return {f"{self._base['CDUMP']}": tasks}
67 changes: 47 additions & 20 deletions workflow/rocoto/gefs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -160,47 +160,74 @@ def efcs(self):
return task

def atmos_prod(self):
atm_master_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_MASTER_TMPL"], {'MEMDIR': 'mem#member#'})
deps = []
data = f'{atm_master_path}/{self.cdump}.t@Hz.master.grb2f#fhr#'
dep_dict = {'type': 'data', 'data': data, 'age': 120}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)
return self._atmosoceaniceprod('atmos')

def ocean_prod(self):
return self._atmosoceaniceprod('ocean')

def ice_prod(self):
return self._atmosoceaniceprod('ice')

def _atmosoceaniceprod(self, component: str):

products_dict = {'atmos': {'config': 'atmos_products',
'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL',
'history_file_tmpl': f'{self.cdump}.t@Hz.master.grb2#dep#'},
'ocean': {'config': 'oceanice_products',
'history_path_tmpl': 'COM_OCEAN_HISTORY_TMPL',
'history_file_tmpl': f'{self.cdump}.ocean.t@Hz.6hr_avg.#dep#.nc'},
'ice': {'config': 'oceanice_products',
'history_path_tmpl': 'COM_ICE_HISTORY_TMPL',
'history_file_tmpl': f'{self.cdump}.ice.t@Hz.6hr_avg.#dep#.nc'}}

atmos_prod_envars = self.envars.copy()
component_dict = products_dict[component]
config = component_dict['config']
history_path_tmpl = component_dict['history_path_tmpl']
history_file_tmpl = component_dict['history_file_tmpl']

varname1, varname2, varname3 = 'grp', 'dep', 'lst'
varval1, varval2, varval3 = self._get_ufs_postproc_grps(self.cdump, self._configs[config], component=component)
var_dict = {varname1: varval1, varname2: varval2, varname3: varval3}

postenvars = self.envars.copy()
postenvar_dict = {'ENSMEM': '#member#',
'MEMDIR': 'mem#member#',
'FHRLST': '#fhr#',
}
'COMPONENT': component}
for key, value in postenvar_dict.items():
atmos_prod_envars.append(rocoto.create_envar(name=key, value=str(value)))
postenvars.append(rocoto.create_envar(name=key, value=str(value)))

resources = self.get_resource('atmos_products')
resources = self.get_resource(component_dict['config'])

history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl])
deps = []
data = f'{history_path}/{history_file_tmpl}'
dep_dict = {'type': 'data', 'data': data, 'age': 120}
deps.append(rocoto.add_dependency(dep_dict))
dependencies = rocoto.create_dependency(dep=deps)

task_name = f'atmos_prod_mem#member#_f#fhr#'
task_name = f'{component}_prod_mem#member#_f#fhr#'
task_dict = {'task_name': task_name,
'resources': resources,
'dependency': dependencies,
'envars': atmos_prod_envars,
'envars': postenvars,
'cycledef': 'gefs',
'command': f'{self.HOMEgfs}/jobs/rocoto/atmos_products.sh',
'command': f'{self.HOMEgfs}/jobs/rocoto/{config}.sh',
'job_name': f'{self.pslot}_{task_name}_@H',
'log': f'{self.rotdir}/logs/@Y@m@d@H/{task_name}.log',
'maxtries': '&MAXTRIES;'
}

fhr_var_dict = {'fhr': ' '.join([str(fhr).zfill(3) for fhr in
self._get_forecast_hours('gefs', self._configs['atmos_products'])])}
fhr_metatask_dict = {'task_name': 'atmos_prod_#member#',
self._get_forecast_hours('gefs', self._configs[config])])}
fhr_metatask_dict = {'task_name': f'{component}_prod_#member#',
'task_dict': task_dict,
'var_dict': fhr_var_dict
}
'var_dict': fhr_var_dict}

member_var_dict = {'member': ' '.join([str(mem).zfill(3) for mem in range(0, self.nmem + 1)])}
member_metatask_dict = {'task_name': 'atmos_prod',
member_metatask_dict = {'task_name': f'{component}_prod',
'task_dict': fhr_metatask_dict,
'var_dict': member_var_dict
}
'var_dict': member_var_dict}

task = rocoto.create_task(member_metatask_dict)

Expand Down
2 changes: 1 addition & 1 deletion workflow/rocoto/gfs_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -1070,7 +1070,7 @@ def _atmosoceaniceprod(self, component: str):
'maxtries': '&MAXTRIES;'
}

metatask_dict = {'task_name': f'{self.cdump}{component}prod',
metatask_dict = {'task_name': f'{self.cdump}{component}_prod',
'task_dict': task_dict,
'var_dict': var_dict
}
Expand Down