Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop]: Update python docstrings and generate preliminary technical documentation #1131

Merged
merged 94 commits into from
Oct 2, 2024
Merged
Changes from 1 commit
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
b9ac737
restructure docs to have UG and Tech documentation via autodoc
gspetro Mar 14, 2023
f566a27
add UG requirements file
gspetro Mar 14, 2023
5b2f2df
update config file
gspetro Mar 14, 2023
1feac5e
update config file location
gspetro Mar 14, 2023
25239bf
minor changes
gspetro Mar 16, 2023
eff6b91
add additional modules
gspetro Mar 17, 2023
a4a4553
merge develop into autodoc
gspetro-NOAA Jul 5, 2023
8ed9658
merge develop branch into autodoc branch
gspetro-NOAA Jan 29, 2024
9592a9b
update Xlink to references.bib
gspetro-NOAA Jan 29, 2024
f2cb058
update python string to HEAD of develop
gspetro-NOAA Jan 29, 2024
e9a0214
update retrieve_data.py comments
gspetro-NOAA Jan 29, 2024
924990e
update retrieve_data.py comments; del .DS_Store
gspetro-NOAA Jan 29, 2024
1ac2815
mv files from UG to docs
gspetro-NOAA Jan 29, 2024
bf298ae
minor edis
gspetro-NOAA Jan 29, 2024
a39f4aa
minor updates/formatting
gspetro-NOAA Jan 29, 2024
3822c82
update docstrings for retrieve_data.py
gspetro-NOAA Jan 29, 2024
10084ae
add tech docs for tests; update script docstrings
gspetro-NOAA Jan 30, 2024
07ca3d7
add init.py to tests so that sphinx views tests as module
gspetro-NOAA Jan 30, 2024
b2767c1
rm __init__.py; adjust sys.path & module paths
gspetro-NOAA Jan 30, 2024
14715b5
add ush & tests files to index; configure to warn for undocumented fu…
gspetro-NOAA Feb 2, 2024
54c61fa
rename docs to doc
gspetro-NOAA Mar 1, 2024
e05e78a
merge in develop
gspetro-NOAA Mar 6, 2024
af061b8
resolve merge conflicts
gspetro-NOAA Mar 6, 2024
d5e7575
fix whitespace
gspetro-NOAA Mar 6, 2024
54a5669
add autoyaml extension
gspetro-NOAA Mar 6, 2024
90f0bf8
merge HEAD of develop
gspetro-NOAA May 25, 2024
7108c93
fix broken EMC link
gspetro-NOAA May 25, 2024
6c944bf
add mock imports
gspetro-NOAA May 25, 2024
01b7aac
Merge branch 'ufs-community:develop' into text/autodoc
gspetro-NOAA Jun 5, 2024
3c3e836
Merge branch 'ufs-community:develop' into text/autodoc
gspetro-NOAA Aug 9, 2024
f49418f
update docstrings in get_crontab_contents; make private members explicit
gspetro-NOAA Aug 20, 2024
10b56f8
add autodoc default options
gspetro-NOAA Aug 20, 2024
53819c3
add autoyaml to the doc README
gspetro-NOAA Aug 20, 2024
4b7a2e5
rm duplicative INSTALL/RUNTIME files - Issue #340
gspetro-NOAA Aug 20, 2024
cf8e0bf
update docstring for set_predef_grid_params
gspetro-NOAA Aug 20, 2024
ad8835c
update conf.py autodoc defaults
gspetro-NOAA Aug 21, 2024
e5c0a90
rm nems configure file
gspetro-NOAA Aug 21, 2024
a52be18
add ufs configure file to tech docs
gspetro-NOAA Aug 21, 2024
d721622
update docstrings for calculate_cost and check_python_version
gspetro-NOAA Aug 22, 2024
404d46d
update docstring in config_utils.py
gspetro-NOAA Aug 22, 2024
5cf30a4
update docstring in create_aqm_rc_file
gspetro-NOAA Aug 22, 2024
422391f
update docstring in create_diag_table_file
gspetro-NOAA Aug 22, 2024
fcd7ac2
update docstring for create_model_configure_file
gspetro-NOAA Aug 22, 2024
1f9b90f
update docstring for create_ufs_configure_file
gspetro-NOAA Aug 22, 2024
7956853
update docstring in generate_FV3LAM_wflow
gspetro-NOAA Aug 23, 2024
8e8522b
update docstrings in get_crontab_contents
gspetro-NOAA Aug 23, 2024
486f577
update docstrings in link_fix
gspetro-NOAA Aug 23, 2024
08bbaab
update docstring in mrms_pull_topofhour
gspetro-NOAA Aug 23, 2024
858b434
(partially) update retrieve_data docstrings
gspetro-NOAA Aug 23, 2024
24d19cc
update docstrings for run_srw_tests
gspetro-NOAA Aug 24, 2024
5f9d533
update docstrings for set_cycle_dates & set_fv3nml_ens_stoch_seeds
gspetro-NOAA Aug 24, 2024
187467b
update docstring in set_fv3nml_sfc_climo_filenames
gspetro-NOAA Aug 24, 2024
b51f10b
update docstring in set_gridparams_ESGgrid
gspetro-NOAA Aug 24, 2024
113a399
update set_gridparams_GFDLgrid docstrings
gspetro-NOAA Aug 24, 2024
b10d4f1
Merge branch 'develop' of https://github.com/ufs-community/ufs-srweat…
gspetro-NOAA Aug 26, 2024
473daa4
update full ush file set
gspetro-NOAA Aug 27, 2024
775303b
revert UFS_plot_domains changes
gspetro-NOAA Aug 27, 2024
62e9869
rm set_namelist & set_ozone_param
gspetro-NOAA Aug 27, 2024
10740c4
update ush RST files
gspetro-NOAA Aug 27, 2024
a57c7a8
update index to modules.rst
gspetro-NOAA Aug 27, 2024
f226ded
update docstrings in set_predef_grid_params and setup
gspetro-NOAA Aug 28, 2024
ee7547a
update docstrings for update_input_nml
gspetro-NOAA Aug 29, 2024
e495d25
add ush modules.rst
gspetro-NOAA Aug 29, 2024
7033661
minor docstring updates to misc files
gspetro-NOAA Aug 29, 2024
2c6eb59
minor updates to several docstrings
gspetro-NOAA Sep 3, 2024
16b7225
update misc docstrings esp in python_utils
gspetro-NOAA Sep 5, 2024
f41c27a
update misc docstrings in python_utils
gspetro-NOAA Sep 6, 2024
996ca7d
update misc python_utils docstrings
gspetro-NOAA Sep 6, 2024
2919edc
update misc docstrings in python_utils
gspetro-NOAA Sep 9, 2024
61a3fc9
update docstrings for WE2E python files; add modules.rst
gspetro-NOAA Sep 10, 2024
7fc1305
add index/modules.rst files
gspetro-NOAA Sep 10, 2024
1fdcebd
rm old index/modules.rst versions
gspetro-NOAA Sep 10, 2024
768cc2d
update docstrings in WE2E files
gspetro-NOAA Sep 10, 2024
fcc5882
update docstrings for WE2E files
gspetro-NOAA Sep 10, 2024
0660b1c
Merge branch 'ufs-community:develop' into text/autodoc
gspetro-NOAA Sep 10, 2024
2549f4a
format create_aqm_rc_file.py w/black
gspetro-NOAA Sep 11, 2024
581598a
reformat docstring for ../ush/create_aqm_rc_file.py
gspetro-NOAA Sep 11, 2024
a4b6b4e
reformat docstring in create_model_configure_file.py
gspetro-NOAA Sep 11, 2024
e6e0803
fix line length
gspetro-NOAA Sep 11, 2024
3d0fd9a
fix line length
gspetro-NOAA Sep 11, 2024
5330ab7
fix line length
gspetro-NOAA Sep 11, 2024
c037c6a
fix line length
gspetro-NOAA Sep 11, 2024
2a1d95a
fix excessive deletion in get_crontab_contents.py
gspetro-NOAA Sep 11, 2024
0f3aef6
update Gaea default data location
gspetro-NOAA Sep 12, 2024
54877a3
Merge branch 'ufs-community:develop' into text/autodoc
gspetro-NOAA Sep 17, 2024
981e83b
solve issues w/UFS_plot_domains script
gspetro-NOAA Sep 17, 2024
9ad2328
update docstrings in UFS_plot_domains
gspetro-NOAA Sep 17, 2024
9112e35
refactor UFS_plot_domains
gspetro-NOAA Sep 17, 2024
1b93bc0
add UFS_plot_domains docs
gspetro-NOAA Sep 17, 2024
69a66ee
fix minor capitalization/typos
gspetro-NOAA Sep 17, 2024
ce70c21
Update ush/create_diag_table_file.py
gspetro-NOAA Sep 17, 2024
60ddb77
change .sh to .py
gspetro-NOAA Sep 17, 2024
2ba7dbf
Merge branch 'text/autodoc' of github.com:gspetro-NOAA/ufs-srweather-…
gspetro-NOAA Sep 17, 2024
15542b7
Merge branch 'ufs-community:develop' into text/autodoc
gspetro-NOAA Sep 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update docstrings for retrieve_data.py
  • Loading branch information
gspetro-NOAA committed Jan 29, 2024
commit 3822c82f665d65a415200a5a09e061dbafbf6c78
67 changes: 33 additions & 34 deletions ush/retrieve_data.py
Original file line number Diff line number Diff line change
@@ -1,30 +1,30 @@
#!/usr/bin/env python3
# pylint: disable=logging-fstring-interpolation
"""
retrieve_data.py
====================================
This script helps users pull data from known data streams, including
URLS and HPSS (only on supported NOAA platforms), or from user-supplied
data locations on disk.

Several supported data streams are included in
parm/data_locations.yml, which provides locations and naming
``parm/data_locations.yml``, which provides locations and naming
conventions for files commonly used with the SRW App. Provide the file
to this tool via the --config flag. Users are welcome to provide their
to this tool via the ``--config`` flag. Users are welcome to provide their
own file with alternative locations and naming conventions.

When using this script to pull from disk, the user is required to
provide the path to the data location, which can include Python
templates. The file names follow those included in the --config file by
default, or can be user-supplied via the --file_name flag. That flag
templates. The file names follow those included in the ``--config`` file by
default, or can be user-supplied via the ``--file_name`` flag. That flag
takes a YAML-formatted string that follows the same conventions outlined
in the parm/data_locations.yml file for naming files.
in the ``parm/data_locations.yml`` file for naming files.

To see usage for this script:

.. code-block::

python retrieve_data.py -h

Also see the parse_args function below.
Also see the ``parse_args`` function below.
"""

import argparse
Expand Down Expand Up @@ -125,9 +125,9 @@ def download_file(url):
on disk.

Arguments:
url url to file to be downloaded
url: url to file to be downloaded

Return:
Returns:
boolean value reflecting state of download.
"""

Expand Down Expand Up @@ -185,20 +185,20 @@ def fill_template(template_str, cycle_date, templates_only=False, **kwargs):
and return the resulting string.

Arguments:
template_str a string containing Python templates
cycle_date a datetime object that will be used to fill in
template_str: a string containing Python templates
cycle_date: a datetime object that will be used to fill in
date and time information
templates_only boolean value. When True, this function will only
templates_only: boolean value. When True, this function will only
return the templates available.

Keyword Args:
ens_group a number associated with a bin where ensemble
Keyword Arguments:
ens_group: a number associated with a bin where ensemble
members are stored in archive files
fcst_hr an integer forecast hour. string formatting should
fcst_hr: an integer forecast hour. string formatting should
be included in the template_str
mem a single ensemble member. should be a positive integer value
mem: a single ensemble member. should be a positive integer value

Return:
Returns:
filled template string
"""

Expand Down Expand Up @@ -298,14 +298,14 @@ def get_file_templates(cla, known_data_info, data_store, use_cla_tmpl=False):

Arguments:

cla command line arguments Namespace object
known_data_info dict from data_locations yaml file
data_store string corresponding to a key in the
cla: command line arguments Namespace object
known_data_info: dict from data_locations yaml file
data_store: string corresponding to a key in the
known_data_info dict
use_cla_tmpl boolean on whether to check cla for templates
use_cla_tmpl: boolean on whether to check cla for templates

Returns:
file_templates a list of file templates
file_templates: a list of file templates
"""

file_templates = known_data_info.get(data_store, {}).get("file_names")
Expand Down Expand Up @@ -345,18 +345,17 @@ def get_requested_files(cla, file_templates, input_locs, method="disk", **kwargs
writeable.

Arguments:
cla: Namespace object containing command line arguments
file_templates: a list of file templates
input_locs: A string containing a single data location, either a url or disk path, or a list of paths/urls.
method: Choice of disk or download to indicate protocol for retrieval

cla Namespace object containing command line arguments
file_templates a list of file templates
input_locs A string containing a single data location, either a url or disk path, or a list of paths/urls.
method Choice of disk or download to indicate protocol for retrieval

Keyword args:
members a list integers corresponding to the ensemble members
check_all boolean flag that indicates all urls should be checked for all files
Keyword Arguments:
members: a list integers corresponding to the ensemble members
check_all: boolean flag that indicates all urls should be checked for all files

Returns:
unavailable a list of locations/files that were unretrievable
unavailable a list of locations/files that were unretrievable
"""

members = kwargs.get("members", "")
Expand Down Expand Up @@ -447,8 +446,8 @@ def hsi_single_file(file_path, mode="ls"):
whether the file_path was found.

Arguments:
file_path path on HPSS
mode the hsi command to run. ls is default. may also
file_path: path on HPSS
mode: the hsi command to run. ls is default. may also
pass "get" to retrieve the file path

"""
Expand Down