Update to steps_blending branch

* First basic functions to implement STEPS blending * Add compute of blend means,sigmas and recompose * pysteps.io with xarray (pySTEPS#219) * Add xarray dependency * MCH importer returns an xarray Dataset * Remove plot lines * Remove import * Adapt readers to xarray format * Rewrite as more general decorator * Add missing metadata * Adapt io tests * Mrms bounding box (pySTEPS#222) * Fix bounding box coordinates * Add missing metadata * Import xarray DataArray Ignore quality field * Black * Do not hardcode metadata * Address review comments by ruben * Add a legacy option to the io functions A "legacy" options is added to revert back the importers and readers behavior to version 1. This is a temporary solution to allow the examples, and other functions, to run as usual (v1.*). Hopefully, this is will allow a easier transition into version 2 during the development process and will allow testing functions that were not updated to v2. * Fix compatibility problems with tests Many of the tests were updated to use the legacy data structures (v1). The tests that still contains issues, were tagged with a TODO message and they are skipped. This will allow incremental changes to be tested in the new v2. IMPORTANT: once the v2 branch is stable, we may remove the legacy compatibility from the code base and the tests. * Update dependencies * Ignore plugins test Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> * Add blend_optical_flow * changes to steps blending procedure - weights according to adjusted BPS2006 method * changes to blending procedures - adjust weights from original BPS2006 method * Determine spatial correlation of NWP model forecast * First attempt to make correlations and thus weights lead time dependent (in progress..) * Change back to original BPS2006 blending formulation and add regression of skill values to climatological values for weights determination * Reformat code with Black * Skill score script imports climatological correlation-values from file now * Small changes to skill score script * Add skill score tests and an interface * Add skill score tests and an interface * Small change to docstring * Bom import xarray (pySTEPS#228) * Add import_bom_rf3 using xarray * Add tests to xarray version * Fix mrms importer tests * Pass **kwargs to internal functions * Add nwp_importers to read bom nwp sample data * Add bom nwp data to source file * Add tests for bom_nwp reader * Fix pystepsrc Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> * Functions to store and compute climatological weights (pySTEPS#231) * Implement the functions get_default_weights, save_weights, calc_clim_weights. These functions are used to evolve the weights in the scale- and skill-dependent blending with NWP in the STEPS blending algorithm. The current weights, based on the correlations per cascade level, are regressed towards these climatological weights in the course of the forecast. These functions save the current and compute the climatological weights (a running mean of the weights of the past n days, where typically n=30). First daily averages are stored and these are then averaged over the running window of n days. * Add tests for pysteps climatological weight io and calculations. * Add path_workdir to outputs section in pystepsrc file and use it as a default path to store/retrieve blending weights. * Minor changes to docstrings, changes to skill scores and testing scripts * Completed documentation for blending clim module, cleanup. Co-authored-by: RubenImhoff <r.o.imhoff@live.nl> Co-authored-by: Carlos Velasco <carlos.velasco@bom.gov.au> Co-authored-by: ned <daniele.nerini@meteoswiss.ch> Co-authored-by: Andres Perez Hortal <16256571+aperezhortal@users.noreply.github.com> Co-authored-by: Ruben Imhoff <Ruben.Imhoff@deltares.nl> Co-authored-by: Carlos Velasco <cvelascof@gmail.com> Co-authored-by: Lesley De Cruz <lesley.decruz+git@gmail.com>
RubenImhoff · Aug 23, 2021 · eec8eed · eec8eed
1 parent d477120
commit eec8eed
Show file tree

Hide file tree

Showing 56 changed files with 2,443 additions and 354 deletions.
diff --git a/.github/workflows/test_pysteps.yml b/.github/workflows/test_pysteps.yml
@@ -8,9 +8,13 @@ env:
 on:
   # Triggers the workflow on push or pull request events but only for the master branch
   push:
-    branches: [ master ]
+    branches:
+      - master
+      - pysteps-v2
   pull_request:
-    branches: [ master ]
+      branches:
+        - master
+        - pysteps-v2
 
 jobs:
   unit_tests:

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,6 +1,6 @@
 repos:
 -   repo: https://github.com/psf/black
-    rev: 21.6b0
+    rev: 21.7b0
     hooks:
     - id: black
       language_version: python3
diff --git a/environment.yml b/environment.yml
@@ -13,3 +13,4 @@ dependencies:
   - pillow
   - pyproj
   - scipy
+  - xarray
diff --git a/environment_dev.yml b/environment_dev.yml
@@ -30,3 +30,4 @@ dependencies:
   - cartopy>=0.18
   - scikit-image
   - pandas
+  - xarray
diff --git a/examples/LK_buffer_mask.py b/examples/LK_buffer_mask.py
@@ -55,7 +55,7 @@
 
 # Read the radar composites
 importer = io.get_method(importer_name, "importer")
-R, quality, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, quality, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 del quality  # Not used
 
@@ -210,7 +210,7 @@
 )
 
 # Read and convert the radar composites
-R_o, _, metadata_o = io.read_timeseries(fns, importer, **importer_kwargs)
+R_o, _, metadata_o = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 R_o, metadata_o = conversion.to_rainrate(R_o, metadata_o)
 
 # Compute Spearman correlation

diff --git a/examples/advection_correction.py b/examples/advection_correction.py
@@ -60,7 +60,7 @@
 
 # Read the radar composites
 importer = io.get_method(importer_name, "importer")
-R, __, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, __, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to mm/h
 R, metadata = conversion.to_rainrate(R, metadata)

diff --git a/examples/anvil_nowcast.py b/examples/anvil_nowcast.py
@@ -48,7 +48,7 @@
 # Read the input time series
 importer = io.get_method(importer_name, "importer")
 rainrate_field, quality, metadata = io.read_timeseries(
-    filenames, importer, **importer_kwargs
+    filenames, importer, legacy=True, **importer_kwargs
 )
 
 # Convert to rain rate (mm/h)
@@ -118,8 +118,8 @@
     date, root_path, path_fmt, fn_pattern, fn_ext, timestep=5, num_next_files=3
 )
 
-refobs_field, quality, metadata = io.read_timeseries(
-    filenames, importer, **importer_kwargs
+refobs_field, _, metadata = io.read_timeseries(
+    filenames, importer, legacy=True, **importer_kwargs
 )
 
 refobs_field, metadata = utils.to_rainrate(refobs_field[-1], metadata)

diff --git a/examples/data_transformations.py b/examples/data_transformations.py
@@ -58,7 +58,7 @@
 
 # Read the radar composites
 importer = io.get_method(importer_name, "importer")
-Z, _, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+Z, _, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Keep only positive rainfall values
 Z = Z[Z > metadata["zerovalue"]].flatten()

diff --git a/examples/optical_flow_methods_convergence.py b/examples/optical_flow_methods_convergence.py
@@ -62,7 +62,7 @@
 # Read the reference radar composite
 importer = io.get_method(importer_name, "importer")
 reference_field, quality, metadata = io.read_timeseries(
-    fns, importer, **importer_kwargs
+    fns, importer, legacy=True, **importer_kwargs
 )
 
 del quality  # Not used

diff --git a/examples/plot_cascade_decomposition.py b/examples/plot_cascade_decomposition.py
@@ -30,7 +30,7 @@
 filename = os.path.join(
     root_path, "20160928", "201609281600_fmi.radar.composite.lowest_FIN_SUOMI1.pgm.gz"
 )
-R, _, metadata = io.import_fmi_pgm(filename, gzipped=True)
+R, _, metadata = io.import_fmi_pgm(filename, gzipped=True, legacy=True)
 
 # Convert to rain rate
 R, metadata = conversion.to_rainrate(R, metadata)

diff --git a/examples/plot_ensemble_verification.py b/examples/plot_ensemble_verification.py
@@ -56,7 +56,7 @@
 
 # Read the data from the archive
 importer = io.get_method(importer_name, "importer")
-R, _, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, _, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to rain rate
 R, metadata = conversion.to_rainrate(R, metadata)
@@ -142,7 +142,7 @@
 )
 
 # Read the observations
-R_o, _, metadata_o = io.read_timeseries(fns, importer, **importer_kwargs)
+R_o, _, metadata_o = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to mm/h
 R_o, metadata_o = conversion.to_rainrate(R_o, metadata_o)

diff --git a/examples/plot_extrapolation_nowcast.py b/examples/plot_extrapolation_nowcast.py
@@ -48,7 +48,7 @@
 
 # Read the radar composites
 importer = io.get_method(importer_name, "importer")
-Z, _, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+Z, _, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to rain rate
 R, metadata = conversion.to_rainrate(Z, metadata)
@@ -113,7 +113,7 @@
     num_next_files=n_leadtimes,
 )
 # Read the radar composites
-R_o, _, metadata_o = io.read_timeseries(fns, importer, **importer_kwargs)
+R_o, _, metadata_o = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 R_o, metadata_o = conversion.to_rainrate(R_o, metadata_o, 223.0, 1.53)
 
 # Compute fractions skill score (FSS) for all lead times, a set of scales and 1 mm/h

diff --git a/examples/plot_noise_generators.py b/examples/plot_noise_generators.py
@@ -34,7 +34,9 @@
 # Import the example radar composite
 root_path = rcparams.data_sources["mch"]["root_path"]
 filename = os.path.join(root_path, "20160711", "AQC161932100V_00005.801.gif")
-R, _, metadata = io.import_mch_gif(filename, product="AQC", unit="mm", accutime=5.0)
+R, _, metadata = io.import_mch_gif(
+    filename, product="AQC", unit="mm", accutime=5.0, legacy=True
+)
 
 # Convert to mm/h
 R, metadata = conversion.to_rainrate(R, metadata)

diff --git a/examples/plot_optical_flow.py b/examples/plot_optical_flow.py
@@ -47,7 +47,7 @@
 
 # Read the radar composites
 importer = io.get_method(importer_name, "importer")
-R, quality, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, quality, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 del quality  # Not used
 

diff --git a/examples/plot_steps_nowcast.py b/examples/plot_steps_nowcast.py
@@ -51,7 +51,7 @@
 
 # Read the data from the archive
 importer = io.get_method(importer_name, "importer")
-R, _, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, _, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to rain rate
 R, metadata = conversion.to_rainrate(R, metadata)

diff --git a/examples/probability_forecast.py b/examples/probability_forecast.py
@@ -74,7 +74,7 @@
 date = datetime.strptime("201607112100", "%Y%m%d%H%M")
 fns = io.find_by_date(date, root, fmt, pattern, ext, timestep, num_prev_files=2)
 importer = io.get_method(importer_name, "importer")
-precip, __, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+precip, __, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 precip, metadata = utils.to_rainrate(precip, metadata)
 # precip[np.isnan(precip)] = 0
 
@@ -124,7 +124,7 @@
 fns = io.find_by_date(
     date, root, fmt, pattern, ext, timestep, num_next_files=nleadtimes
 )
-obs, __, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+obs, __, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 obs, metadata = utils.to_rainrate(obs, metadata)
 obs[np.isnan(obs)] = 0
 

diff --git a/examples/rainfarm_downscale.py b/examples/rainfarm_downscale.py
@@ -33,7 +33,7 @@
 root_path = rcparams.data_sources["mch"]["root_path"]
 filename = os.path.join(root_path, "20160711", "AQC161932100V_00005.801.gif")
 precip, _, metadata = io.import_mch_gif(
-    filename, product="AQC", unit="mm", accutime=5.0
+    filename, product="AQC", unit="mm", accutime=5.0, legacy=True
 )
 
 # Convert to mm/h

diff --git a/examples/thunderstorm_detection_and_tracking.py b/examples/thunderstorm_detection_and_tracking.py
@@ -63,7 +63,7 @@
     date, root_path, path_fmt, fn_pattern, fn_ext, timestep, num_next_files=20
 )
 importer = io.get_method(importer_name, "importer")
-R, _, metadata = io.read_timeseries(fns, importer, **importer_kwargs)
+R, _, metadata = io.read_timeseries(fns, importer, legacy=True, **importer_kwargs)
 
 # Convert to reflectivity (it is possible to give the a- and b- parameters of the
 # Marshall-Palmer relationship here: zr_a = and zr_b =).

diff --git a/pysteps/blending/__init__.py b/pysteps/blending/__init__.py
@@ -0,0 +1,4 @@
+# -*- coding: utf-8 -*-
+"""Methods for blending NWP model(s) with nowcasts."""
+
+from pysteps.blending.interface import get_method