WRES Scripts Usage Guide

Table of Contents

• Overview
  • Where can I find the scripts?
  • What are the general usage instructions?
    • Flags
    • Required information
    • Examples (using bash)
      • If posting custom evaluation data sets to the WRES web-service, be sure to prepare the YAML declaration accordingly
• Troubleshooting
  • Missing Python packages

Overview

The WRES team has created Python and bash scripts that allow for easier interaction with a WRES wres-service implementation, such as the Central OWP WRES (COWRES) hosted at the National Water Center in Alabama. Both scripts employ the same command-line options.

Before reading the below, note the following:

• Any WRES web-service must have a URL through which the service is accessed. In the instructions below, that WRES web-service URL is referred to as [web-service URL].
• Any WRES web-service should use TLS/SSL to secure the endpoint, meaning that HTTPS should be used. A certificate to authenticate the service, typically a .pem file, will need to be created. That certificate is referred to as the [web-service CA .pem] in the instructions below.

Where can I find the scripts?

In the WRES repository:

• bash using curl
• Python using the requests library

What are the general usage instructions?

Python:

python path/to/wres_caller.py [-h] [-f FILENAME] [-u HOST] [-c CERT] [-o OUTPUT] [-l OBSERVED] [-p PREDICTED] [-b BASELINE] [-k] [-s]

Bash:

./path/to/wres_caller.sh [-h] [-f FILENAME] [-u HOST] [-c CERT] [-o OUTPUT] [-l OBSERVED] [-p PREDICTED] [-b BASELINE] [-k] [-s]

Flags

Options	Description
-f, --filename	Declaration filename
-u, --host	The web-service URL (without the http/https prefix); defaults to the WRES_HOST_NAME environment variable.
-c, --cert	The web-service CA .pem file to authenticate the WRES instance; defaults to WRES_CA_FILE environment variable.
-o, --output	Directory where output is to be written in relation to directory calling the script from.
-l, --observed	Data to post for the observed sources either one file or a directory. Gzip compression is recommended.
-p, --predicted	Data to post for the predicted sources either one file or a directory. Gzip compression is recommended.
-b, --baseline	Data to post for the baseline sources either one file or a directory. Gzip compression is recommended.
-k, --keep_input	Instruct COWRES to not remove posted data after evaluation is completed.
-s, --silent	Suppresses additional logging produced by this script.

The following flag is only available in the Python version of the script:

Options	Description
-h, --help	Lists the available options for the script.

Required information

• -f, --filename: All executions are tied to a evaluation declaration.
• -u, --host: The URL of the WRES web-service you want to leverage (defaults to WRES_HOST_NAME environment variable if not defined).
• -c, --cert: The TLS certificate authority .pem used for authenticating that web-service (defaults to WRES_CA_FILE environment variable). For example, cowres_dod_ca_chain.pem is the certificate file used for accessing the COWRES, and that is available through the WRES User Support VLab project.

Examples (using bash)

• Basic use-case: Will execute the provided testEvaluation.yml declaration and download output to the current directory:

  ./wres_caller.sh --cert [web-service CA .pem] --host [web-service URL] -f testEvaluation.yml

• Complex use-case: Will execute the provided testEvaluation.yml declaration, uploading data for observed, predicted, and baseline, and storing the results in a specific location:

  ./wres_caller.sh --cert [web-service CA .pem] --host [web-service URL] -f testEvaluation.yml --observed path/to/observed_data_directory --predicted path/to/predicted_file.csv --baseline path/to/baseline_file_tarred_and_gzipped.tgz --output directory/to/store/output

If posting custom evaluation data sets to the WRES web-service, be sure gzip the files prior to posting

The WRES is able to read gzipped files and gzipped tarballs. Any file to be posted to the COWRES should be gzipped prior to posting in order to save bandwidth and save time waiting for the data to be sent and processed. Furthermore, it is recommended that groups of files be tarred, and then gzipped, prior to sending so that only one file needs to be posted. The WRES can parse through a tarball in order to read individual files appropriately.

If posting custom evaluation data sets to the WRES web-service, be sure to prepare the YAML declaration accordingly

The sources entries for data to be posted must be omitted. For more information, see Posting timeseries data directly to a WRES web‐service as inputs for a WRES job#Prepare the declaration by omitting the sources for data to be sent directly.

Troubleshooting

Missing Python packages

Depending where you are using this script from, you may not have the required Python libraries to use it. If you have the ability to pip install <library> onto your machine then you can do that, but if you are on a machine where you do not have those permissions then here is a guide on how to set up a virtual Python environment to use the script.

Execution of the wres_caller.py script requires the Python libraries requests and urllib3. To create your run environment, required only the first time, do the following:

  $ python -m venv ./venv

To activate your environment and prepare it for executing the script, do the following:

  $ source venv/bin/activate
  (venv) $ pip install requests
  (venv) $ pip install urllib3==1.26.6

You can then run the script with a -h argument to list options:

  (venv) $ python [path]/wres_caller.py -h

Note that the install of a specific version of urllib3 may not be required. To test if it is required, run the script with -h before installing urllib3. If it works, then you should be good to go. Also note that the same @venv@ can be employed for future executions of the script without needing to install the packages again.