First draft docs update goes with NOAA-OWP#502

mattw-nws · Mar 29, 2023 · ab36ce3 · ab36ce3
1 parent f6768d0
commit ab36ce3
Show file tree

Hide file tree

Showing 2 changed files with 106 additions and 69 deletions.
diff --git a/doc/DEPENDENCIES.md b/doc/DEPENDENCIES.md
@@ -237,23 +237,10 @@ Version `0.3.0` or greater must be installed.
 
 ### Setup
 
-The dependency is handled as a Git Submodule, located at `extern/t-route`.   To initialize the submodule after cloning the repo:
-```sh
-git submodule update --init extern/t-route
-```
-Git _should_ take care of checking out the commit for the required version automatically (assuming latest upstream changes have been fetched), so it should be possible to also use the command above to sync future updates to the required version.
-
-Once the submodule is fetched, the routing module must installed in a suitable environment.
-
-One supported option is to create a `virtualenv` environment at `.venv` in the project root and activate this environment for any simulations using routing.  
-
-See the [installing t-route section here](PYTHON_ROUTING.md#installing-t-route) for more details.
-
-### Enabling Routing in Simulations
+See the [installing t-route section](PYTHON_ROUTING.md#installing-t-route) for more details.
 
-To do this, include the `-DNGEN_ACTIVATE_ROUTING:BOOL=true` option when running the `cmake` build on the command line to generate the build system.  An appropriate `routing_config.yaml` must be passed to the NGen realization config.  More info can be found in the [python routing documentation](PYTHON_ROUTING.md#routing-config)
+Be sure to build ngen with Python and Routing support as discussed there, and if using a virtual environment, make sure it is activated when running ngen, e.g.:
 
-Before executing any simulation, be sure to activate the virtual environment.
 ```sh
-source .venv/bin/activate
+source venv/bin/activate
 ```
diff --git a/doc/PYTHON_ROUTING.md b/doc/PYTHON_ROUTING.md
@@ -1,61 +1,51 @@
 # Python Routing
-- [Python Routing](#python-routing)
-- [Summary](#summary)
-- [Installing t-route](#installing-t-route)
-- [Using t-route with ngen](#using-t-route-with-ngen)
-  - [Routing Config](#routing-config)
 
-
-
-# Summary
-
-This describes how to use the Python-based t-route routing module with ngen. 
+## Summary
 
 [t-route](https://github.com/NOAA-OWP/t-route) is the routing framework developed by NOAA-OWP.
 
 See [Setting up t-route source](DEPENDENCIES.md#t-route) for details on aquiring the t-route submodule.
 You will also need to [set up pybind11](DEPENDENCIES.md#pybind11) to use t-route.
 
-# Installing t-route
-These steps will cover installing the t-route package into a virtual environment in the ngen project.
-From the project root (if this virutal environment exists, you may skip this step.)
+## Installing t-route
 
-```sh
-mkdir .venv
-python3 -m venv .venv
-```
+Since t-route is set of Python modules, it will need to be installed in the Python environment ngen will be running with. Below are recommended steps to accomplish this: 
 
-Activate the virtual environment and update a couple of tools.
+1. From the ngen project root (if this virtual environment exists, you may skip this step.)
 
 ```sh
-source .venv/bin/activate
-pip install -U pip setuptools cython dask
+mkdir venv
+python3 -m venv venv
 ```
 
-Install the routing driver modules.
+2. Activate the virtual environment and install/update a few prerequisites:
 
 ```sh
-pip install -e ngen/extern/t-route/src/ngen_routing/
-pip install -e ngen/extern/t-route/src/nwm_routing/
+source venv/bin/activate
+pip install -U pip deprecated pyarrow geopandas tables
 ```
 
-Next, we need to build some python extension modules that the routing package requires.  A convience script is located in the t-route
-package to help with this.
+3. Obtain the t-route software and [compile and install it](https://github.com/NOAA-OWP/t-route#usage-and-testing). This should install the modules in your activated venv from step 2. The t-route source can be downloaded and placed anywhere, it is only important that the Python modules are installed in the right environment/site.
+
+## Installation Caveats
 
-NOTE t-route extension modules rely on netcdf fortran, and thus they need to be compiled with the same fortran compiler that compiled
-the netcdf library.  In the example below, `libnetcdff` was provided by the el7 package `netcdf-fortran-openmpi-static-4.2-16.el7.x86_64`
-which was compiled with the openmpi fortran compiler.  So we have to set the `FC` environment variable appropriately before executing the script.  By default, gfortran is the selected fortran compiler.
+### Compilers and Libraries
 
+The t-route extension modules rely on netcdf fortran, and thus they need to be compiled with the same fortran compiler that compiled
+the netcdf library.  For example, if `libnetcdff` was provided by the RHEL7 package `netcdf-fortran-openmpi-static-4.2-16.el7.x86_64`
+which was compiled with the openmpi fortran compiler, you will have to set the `FC` environment variable appropriately before executing the t-route `compiler.sh` script, like so:
 ```sh
-pushd ngen/extern/t-route/src/python_routing_v02
-F90=mpif90 ./compiler.sh
-popd
+FC=mpif90 ./compiler.sh
 ```
-This should compile all extension modules and `pip install -e` the various namespace package modules for the t-route framework and routing modules.
+More generally speaking, the t-route compiler script is going to compile and link C and Fortran code, which will end up being run inside the ngen process--so you should make sure that the compilers, include paths, library paths, etc. are properly prepared in environment variables the same way that they were when you compiled ngen, so that there are not conflicting dependencies when they are run together.
+
+### Default installation is in development mode (impacts macOS)
 
-[Additional documentation for configuration and dependencies of t-route](https://github.com/NOAA-OWP/t-route#configuration-and-dependencies).  
+The compiler.sh script will install the Python modules with `-e`. On macOS, you may need to re-install the modules in t-route's `src` directory directly after running `compiler.sh`.
+
+[Additional documentation for configuration and dependencies of t-route](https://github.com/NOAA-OWP/t-route#configuration-and-dependencies).
 
-# Using t-route with ngen
+## Using t-route with ngen
   * Create the build directory including the options to activate Python and Routing: 
 
       * Activate Python flag with `-DNGEN_ACTIVATE_PYTHON:BOOL=ON`
@@ -65,29 +55,89 @@ This should compile all extension modules and `pip install -e` the various names
       * An example create build directory command with the above two options activated:
 
     ```sh
-      cmake -DCMAKE_BUILD_TYPE=Debug -B cmake-build-debug -DNGEN_ACTIVATE_PYTHON:BOOL=ON -DNGEN_ACTIVATE_ROUTING:BOOL=ON .
+      cmake -B cmake_build -DNGEN_ACTIVATE_PYTHON:BOOL=ON -DNGEN_ACTIVATE_ROUTING:BOOL=ON .
     ```  
 
   * Unit tests for the Routing_Py_Adapter class can then be built and run from the main directory with the following two commands:
 
     ```sh
-      cmake --build cmake-build-debug --target test_routing_pybind
+      cmake --build cmake_build --target test_routing_pybind
       ./cmake-build-debug/test/test_routing_pybind
-    ```  
-  * An [example realization config](../data/example_bmi_multi_realization_config_w_routing.json) with routing inputs.
-
-## Routing Config
-
-t-route uses a yaml input file for configuring the routing setup.  An [example configuration](../data/ngen_routing.yaml) file is included in the example data.  In this file, the following keys should be set appropriately:
-```yaml
-supernetwork_parameters:
-    title_string: "Ngen"
-    #Below will change with new catchment route link
-    geo_file_path: "<path_to_hydrofabric>/waterbody-params.json"
-    #CHANGE BELOW WITH NEW NGEN HYDRO FABRIC DATA
-     ngen_nexus_file: "<path_to_hydrofabric>/flowpath_edge_list.json"
-#ngen output files
-forcing_parameters:
-    nexus_input_folder: "<path_to_ngen_output>"
-    nexus_file_pattern_filter: "nex-*"
+    ```
+  * An [example realization config](../data/gauge_01073000/example_bmi_multi_realization_config_w_routing.json) with routing inputs.
+
+### Realization Config
+
+To enable routing in a simulation realization config file, a `routing` block should appear with a path to the t-route configuration file at the same level as the `time` configuration, like so:
+
+```JSON
+...
+"time": {
+    "start_time": "2015-12-01 0:00:00",
+    "end_time": "2015-12-30 23:00:00",
+    "output_interval": 3600
+},
+"routing": {
+    "t_route_config_file_with_path": "./data/gauge_01073000/routing_config.yaml"
+}
+...
+```
+
+### Routing Config
+
+t-route uses a yaml input file for configuring the routing setup, see the [t-route repo documentation](https://github.com/NOAA-OWP/t-route#configuration) for more information.  An [example configuration](../data/gauge_01073000/routing_config.yaml) file is included in the example data.
+
+### Configuration considerations for t-route with ngen
+
+Output from ngen is currently created on an hourly basis and in files per nexus, which is different from t-route's native processing expectations. To account for this, currently t-route preprocesses the ngen nexus output CSV files before running. To ensure this happens correctly, these settings *must* be correct in the configuration YAML:
+
+```YAML
+    # These examples assume a 720h (30 day) simulation:
+    forcing_parameters:
+        # t-route's internal timestep in seconds
+        dt                          : 300
+        # ngen's timestep divided by t-route's (e.g. 3600/300)
+        qts_subdivisions            : 12 
+        # total simulation t-route timesteps (e.g. 12 per hour, 288 per day)
+        nts                         : 8640
+        # number of external (ngen) timesteps
+        max_loop_size               : 720
+        # The location to find the nex-* CSV files
+        qlat_input_folder           : ./ 
+        nexus_input_folder          : ./
+        # The glob pattern to match nexus output files - MUST NOT CHANGE!
+        qlat_file_pattern_filter    : "nex-*"
+        nexus_file_pattern_filter   : "nex-*"
+        # A directory where the temporary *.parquet files will be stored
+        binary_nexus_file_folder    : /tmp
 ```
+IMPORTANT: See the #known-issues below!
+
+## Running t-route separately with ngen output
+
+In some cases it may be useful to run the routing step separately. To do so, after installing t-route in your environment as described above, execute it directly this way:
+
+```sh
+python -m nwm_routing -V4 -f /path/to/routing_config.yaml
+```
+
+
+This is particularly useful if a long simulation completes in ngen but fails in t-route. Running routing this way will also often give more detailed error messages, if you are experiencing problems during the routing phase.
+
+## Known issues
+
+### Cleanup of `*.parquet` files required
+
+Running t-route with ngen `nex-*.csv` input will generate hourly files with names matching `*.parquet` in the directory specified by `binary_nexus_file_folder` but it *does not remove them after the simulation compeltes*, and the presence of these files will prevent t-route from running. To run a simulation a second time, you will need to manually remove the created `*.parquet` files.
+
+### Bug in multiprocessing on macOS
+
+It is not currently possible to use multiprocessing in t-route on macOS as part of an ngen simulation directly. To use routing on macOS, either:
+
+1. Run t-route in ngen with the `routing` block in the realization config and ensure that the t-route configuration specifies `cpu_pool: 1` to disable multiprocessing,
+
+OR
+
+2. Run t-route separately, after the ngen simulation as described above.
+
+At present, running within ngen with `cpu_pool` > 1 will result in spawning many additional ngen processes, consuming lots of resources and likely corrupting your output! See #505 .