Merge latest version pysteps-v2 into steps_blending branch (#237)

* Update docstrings * More cleanup * Cleanup imports * Cleanup imports * More cleanup * Update docstrings * Update references Mention the work of Ravuri et al (2021, Nature) as an example of work using cGANs to generate ensembles * Clean up page * Reprojection functionality (#236) * Added Lesley's reprojection module to this branch * Added compatibility for three-dimensional xarrays * Add commentary to reprojection util * Changes to make reprojection of KNMI data possible * Changes after Daniele's review * Add dependencies * Changes to importers, see issue #215 * Add tests * Fix some issues * documentation * Fixes for tests * Set requirements again * Some fixes * Changes to nwp_importers after Carlos' response * Remove wrong example script * Remove rasterio dependencies from lists * First try to prevent testing error * Changes Daniele and fix knmi nwp importer * Add rasterio to tox.ini * Aesthetics * rasterio import test * Add rasterio to the test dependencies * Reset try-except functionality for rasterio import * Fix for failing test on windows python 3.6 * add importerskip rasterio Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be> * Revert "Merge branch 'steps_blending' into pysteps-v2" (#239) This reverts commit 2c639f8, reversing changes made to bccb8fc. Co-authored-by: ned <daniele.nerini@meteoswiss.ch> Co-authored-by: dnerini <daniele.nerini@gmail.com> Co-authored-by: Wout Dewettinck <wout.dewettinck@ugent.be>
pySTEPS · dnerini · Jan 14, 2022 · Apr 3, 2019 · Apr 5, 2019 · Jul 13, 2021
commit 82a9d1896f8dbbd031a8b9b40dde33812b715ea1
diff --git a/.github/workflows/test_pysteps.yml b/.github/workflows/test_pysteps.yml
@@ -2,7 +2,7 @@ name: Test Pysteps
 
 env:
   MINIMAL_DEPENDENCIES: cython numpy jsmin jsonschema matplotlib netCDF4 opencv pillow pyproj scipy dask
-  OPTIONAL_DEPENDENCIES: pyfftw cartopy h5py PyWavelets pandas scikit-image
+  OPTIONAL_DEPENDENCIES: pyfftw cartopy h5py PyWavelets pandas scikit-image rasterio
   TEST_DEPENDENCIES: pytest pytest-cov
 
 on:

diff --git a/.gitignore b/.gitignore
@@ -85,6 +85,3 @@ venv.bak/
 
 # mypy
 .mypy_cache/
-
-# example data
-examples/radar/
diff --git a/doc/requirements.txt b/doc/requirements.txt
@@ -1,4 +1,4 @@
-# Additional requeriments related to the documentation build only
+# Additional requirements related to the documentation build only
 sphinx
 numpydoc
 sphinxcontrib.bibtex

diff --git a/doc/source/pysteps_reference/utils.rst b/doc/source/pysteps_reference/utils.rst
@@ -16,3 +16,4 @@ Implementation of miscellaneous utility functions.
 .. automodule:: pysteps.utils.spectral
 .. automodule:: pysteps.utils.tapering
 .. automodule:: pysteps.utils.transformation
+.. automodule:: pysteps.utils.reprojection
diff --git a/doc/source/references.bib b/doc/source/references.bib
@@ -198,6 +198,17 @@ @ARTICLE{RC2011
   YEAR=2011
 }
 
+@ARTICLE{Ravuri2021,
+  AUTHOR = "Ravuri, Suman and Lenc, Karel and Willson, Matthew and Kangin, Dmitry and Lam, Remi and Mirowski, Piotr and Fitzsimons, Megan and Athanassiadou, Maria and Kashem, Sheleem and Madge, Sam and Prudden, Rachel and Mandhane, Amol and Clark, Aidan and Brock, Andrew and Simonyan, Karen and Hadsell, Raia and Robinson, Niall and Clancy, Ellen and Arribas, Alberto and Mohamed, Shakir",
+  JOURNAL = "Nature",
+  TITLE = "Skilful precipitation nowcasting using deep generative models of radar",
+  VOLUME = 597,
+  NUMBER = 7878,
+  PAGES = "672--677",
+  YEAR = 2011,
+  DOI = "10.1038/s41586-021-03854-z",
+}
+
 @ARTICLE{RCW2011,
   AUTHOR = "E. Ruzanski and V. Chandrasekar and Y. Wang",
   TITLE = "The {CASA} Nowcasting System",

diff --git a/doc/source/user_guide/machine_learning_pysteps.rst b/doc/source/user_guide/machine_learning_pysteps.rst
@@ -12,12 +12,12 @@ Before starting the comparison, you need to ask yourself what is the objective o
 To achieve objective 1, it is sufficient to produce a single deterministic nowcast that filters out the unpredictable small-scale precipitation features.
 However, this will create a nowcast that will become increasingly smooth over time.
 
-To achieve objective 2, you need to produce a probabilistic or an ensemble nowcast (several ensemble members and realizations).
+To achieve objective 2, you need to produce a probabilistic or an ensemble nowcast (several ensemble members or realizations).
 
-In weather forecasting (and nowcasting) we usually want to achieve both goals because it is impossible to predict the evolution of a chaotic system with 100% accuracy, especially space-time precipitation fields and thunderstorms!
+In weather forecasting (and nowcasting), we usually want to achieve both goals because it is impossible to predict the evolution of a chaotic system with 100% accuracy, especially space-time precipitation fields and thunderstorms!
 
 Machine learning and pysteps offer several methods to produce both deterministic and probabilistic nowcasts.
-Therefore, if you want to compare machine learning-based nowcasts to simpler extrapolation-based models, you need to select the right method.
+Therefore, if you want to compare machine learning-based nowcasts to simpler extrapolation-based models, you need to select the right method and verification measure.
 
 1. Deterministic nowcasting
 --------------------------------------------
@@ -32,17 +32,17 @@ This is best explained by the decomposition of the mean squared error (MSE):
 
 :math:`MSE = bias^2 + Var`
 
-All deterministic machine learning algorithms that minimize the MSE (or a variation) will also inevitably minimize the variance of nowcast fields.
-This is a natural attempt to filter out the unpredictable precipitation features, which would otherwise increase the variance (and the MSE).
+All deterministic machine learning algorithms that minimize the MSE (or a related measure) will also inevitably minimize the variance of nowcast fields.
+This is a natural attempt to filter out the unpredictable evolution of precipitation features, which would otherwise increase the variance (and the MSE).
 The same principle holds for convolutional and/or deep neural network architectures, which also produce smooth nowcasts.
 
-Therefore, it is better to avoid directly comparing an error-minimization machine learning nowcast to a variance-preserving radar extrapolation, as produced by the module :py:mod:`pysteps.nowcasts.extrapolation`. Instead, you should use the pysteps ensemble mean.
+Therefore, it is better to avoid directly comparing an error-minimization machine learning nowcast to a variance-preserving radar extrapolation, as produced by the module :py:mod:`pysteps.nowcasts.extrapolation`. Instead, you should use compare with the mean of a sufficiently large ensemble.
 
 A deterministic equivalent of the ensemble mean can be approximated using the modules :py:mod:`pysteps.nowcasts.sprog` or :py:mod:`pysteps.nowcasts.anvil`.
-Another possibility, but more computationally demanding, is to average many ensemble members generated by the modules :py:mod:`pysteps.nowcasts.steps` or :py:mod:`pysteps.nowcasts.sseps`.
+Another possibility, but more computationally demanding, is to average many ensemble members generated by the modules :py:mod:`pysteps.nowcasts.steps` or :py:mod:`pysteps.nowcasts.linda`.
 
 Still, even by using the pysteps ensemble mean, it is not given that its variance will be the same as the one of machine learning predictions.
-Possible solutions are to:
+Possible solutions to this:
 
 #. use a normalized MSE (NMSE) or another score accounting for differences in the variance between prediction and observation.
 #. decompose the field with a Fourier (or wavelet) transform to compare features at the same spatial scales.
@@ -54,7 +54,7 @@ A good deterministic comparison of a deep convolutional machine learning neural
 
 Probabilistic machine learning regression methods can be roughly categorized into:
 
-a. Quantile-based methods, such as quantile regression, quantile random forests and quantile neural networks.
+a. Quantile-based methods, such as quantile regression, quantile random forests, and quantile neural networks.
 b. Ensemble-based methods, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs).
 
 Quantile-based machine learning nowcasts are interesting, but can only estimate the probability of exceedance at a given point (see e.g. :cite:`FSNBG2019`).
@@ -63,7 +63,7 @@ To estimate areal exceedance probabilities, for example above catchments, or to
 
 Generative machine learning methods are similar to the pysteps ensemble members. Both are designed to produce an ensemble of possible realizations that preserve the variance of observed radar fields.
 
-A proper probabilistic verification of generative machine learning models against pysteps would be an interesting research direction.
+A proper probabilistic verification of generative machine learning models against pysteps is an interesting research direction which was recently undertake in the work of :cite:`Ravuri2021`.
 
 Summary
 -------
@@ -85,19 +85,19 @@ The comparison of methods from different types should only be done carefully and
      - Pysteps
      - Verification
    * - Deterministic (variance-preserving)
-     - SRGAN (Wang et al., 2018), Others?
+     - SRGAN, Others?
      - :py:mod:`pysteps.nowcasts.extrapolation` (any optical flow method)
      - MSE, RMSE, MAE, ETS, etc
    * - Deterministic (error-minimization)
      - Classical ANNs, (deep) CNNs, random forests, AdaBoost, etc
-     - :py:mod:`pysteps.nowcasts.sprog`, :py:mod:`pysteps.nowcasts.anvil` or ensemble mean of :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.sseps`
+     - :py:mod:`pysteps.nowcasts.sprog`, :py:mod:`pysteps.nowcasts.anvil` or ensemble mean of :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.linda`
      - MSE, RMSE, MAE, ETS, etc or better normalized scores, etc
    * - Probabilistic (quantile-based)
      - Quantile ANN, quantile random forests, quantile regression
-     - :py:mod:`pysteps.nowcasts.lagrangian_probability` or probabilities derived from :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.sseps`
+     - :py:mod:`pysteps.nowcasts.lagrangian_probability` or probabilities derived from :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.linda`
      - Reliability diagram (predicted vs observed quantile), probability integral transform (PIT) histogram
    * - Probabilistic (ensemble-based)
-     - GANs, VAEs, etc
-     - Ensemble and probabilities derived from :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.sseps`
+     - GANs (:cite:`Ravuri2021`), VAEs, etc
+     - Ensemble and probabilities derived from :py:mod:`pysteps.nowcasts.steps`/:py:mod:`~pysteps.nowcasts.linda`
      - Probabilistic verification: reliability diagrams, continuous ranked probability scores (CRPS), etc.
        Ensemble verification: rank histograms, spread-error relationships, etc
diff --git a/environment_dev.yml b/environment_dev.yml
@@ -31,3 +31,4 @@ dependencies:
   - scikit-image
   - pandas
   - xarray
+  - rasterio
diff --git a/examples/read_netcdf.py b/examples/read_netcdf.py
diff --git a/examples/test_decomposition_NWP.py b/examples/test_decomposition_NWP.py
diff --git a/pysteps/blending/__init__.py b/pysteps/blending/__init__.py
-Original file line number
+Diff line change
@@ Expand Up / @@ -31,3 +31,4 @@ dependencies: @@
       - scikit-image
       - pandas
       - xarray
+      - rasterio