Merge branch 'develop' into calibrate-impact-functions

Author: peanutfun

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,36 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior/error:
+1. 
+
+Code example:
+```python
+# Your code here
+```
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Climada Version:** [Version or branch]
+
+**System Information (please complete the following information):**
+ - Operating system and version: [e.g. Ubuntu 22.04, macOS 14.3.1, Windows 10]
+ - Python version: [e.g. 3.10] 
+ (to obtain this information execute > import sys  >print(sys.version))
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/code-feature-request.md

@@ -0,0 +1,20 @@
+---
+name: Code feature request
+about: Suggest an idea to improve the code
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.readthedocs.yml

@@ -5,8 +5,13 @@ build:
   tools:
     python: "mambaforge-4.10"
 
+  # Append fixed Python version to requirements
+  jobs:
+    pre_create_environment:
+      - echo "  - python=3.9" >> requirements/env_climada.yml
+
 conda:
-  environment: requirements/env_docs.yml
+  environment: requirements/env_climada.yml
 
 python:
   install:

AUTHORS.md

@@ -33,3 +33,5 @@
 * Kam Lam Yeung
 * Sarah Hülsen
 * Timo Schmid
+* Luca Severino
+* Samuel Juhel

CHANGELOG.md

@@ -10,16 +10,65 @@ Code freeze date: YYYY-MM-DD
 
 ### Dependency Changes
 
-### Added
-
 ### Changed
 
+- Centroids complete overhaul. Most function should be backward compatible. Internal data is stored in a geodataframe attribute. Raster are now stored as points, and the meta attribute is removed. Several methds were deprecated or removed. [#787](https://github.com/CLIMADA-project/climada_python/pull/787)
+- Improved error messages produced by `ImpactCalc.impact()` in case impact function in the exposures is not found in impf_set [#863](https://github.com/CLIMADA-project/climada_python/pull/863)
+
 ### Fixed
 
+- Avoid an issue where a Hazard subselection would have a fraction matrix with only zeros as entries by throwing an error [#866](https://github.com/CLIMADA-project/climada_python/pull/866)
+
+### Added
+
+- climada.hazard.centroids.centr.Centroids.get_area_pixel
+- climada.hazard.centroids.centr.Centroids.get_dist_coast
+- climada.hazard.centroids.centr.Centroids.get_elevation
+- climada.hazard.centroids.centr.Centroids.get_meta
+- climada.hazard.centroids.centr.Centroids.get_pixel_shapes
+- climada.hazard.centroids.centr.Centroids.to_crs
+- climada.hazard.centroids.centr.Centroids.to_default_crs
+- climada.hazard.centroids.centr.Centroids.write_csv
+- climada.hazard.centroids.centr.Centroids.write_excel
+
 ### Deprecated
 
+- climada.hazard.centroids.centr.Centroids.from_lat_lon
+- climada.hazard.centroids.centr.Centroids.def set_area_pixel
+- climada.hazard.centroids.centr.Centroids.def set_area_approx
+- climada.hazard.centroids.centr.Centroids.set_dist_coast
+- climada.hazard.centroids.centr.Centroids.empty_geometry_points
+- climada.hazard.centroids.centr.Centroids.set_meta_to_lat_lon
+- climada.hazard.centroids.centr.Centroids.set_lat_lon_to_meta
+
 ### Removed
 
+- climada.hazard.base.Hazard.clear
+- climada.hazard.base.Hazard.raster_to_vector
+- climada.hazard.base.Hazard.read_mat
+- climada.hazard.base.Hazard.reproject_raster
+- climada.hazard.base.Hazard.set_vector
+- climada.hazard.base.Hazard.vector_to_raster
+- climada.hazard.centroids.centr.Centroids.calc_pixels_polygons
+- climada.hazard.centroids.centr.Centroids.check
+- climada.hazard.centroids.centr.Centroids.clear
+- climada.hazard.centroids.centr.Centroids.equal
+- climada.hazard.centroids.centr.Centroids.from_mat
+- climada.hazard.centroids.centr.Centroids.from_base_grid
+- climada.hazard.centroids.centr.Centroids.read_excel
+- climada.hazard.centroids.centr.Centroids.read_hdf5
+- climada.hazard.centroids.centr.Centroids.read_mat
+- climada.hazard.centroids.centr.Centroids.set_elevation
+- climada.hazard.centroids.centr.Centroids.set_geometry_points
+- climada.hazard.centroids.centr.Centroids.set_lat_lon
+- climada.hazard.centroids.centr.Centroids.set_raster_file
+- climada.hazard.centroids.centr.Centroids.set_raster_from_pnt_bounds
+- climada.hazard.centroids.centr.Centroids.set_vector_file
+- climada.hazard.centroids.centr.Centroids.values_from_raster_files
+- climada.hazard.centroids.centr.Centroids.values_from_vector_files
+- climada.hazard.centroids.centr.generate_nat_earth_centroids
+- `requirements/env_docs.yml`. The regular environment specs are now used to build the online documentation [#687](https://github.com/CLIMADA-project/climada_python/pull/687)
+
 ## 4.1.1
 
 Release date: 2024-02-21

climada/conf/climada.conf

@@ -67,7 +67,7 @@
         "cache_db": "{local_data.system}/.downloads.db",
         "cache_enabled": true,
         "cache_dir": "{local_data.system}/.apicache",
-        "supported_hazard_types": ["river_flood", "tropical_cyclone", "storm_europe", "relative_cropyield", "wildfire", "earthquake", "flood"],
-        "supported_exposures_types": ["litpop", "crop_production", "ssp_population"]
+        "supported_hazard_types": ["river_flood", "tropical_cyclone", "storm_europe", "relative_cropyield", "wildfire", "earthquake", "flood", "hail"],
+        "supported_exposures_types": ["litpop", "crop_production", "ssp_population", "crops"]
     }
 }

climada/engine/forecast.py

@@ -31,6 +31,7 @@
 from matplotlib.ticker import PercentFormatter, ScalarFormatter
 from matplotlib.colors import ListedColormap, BoundaryNorm
 import cartopy.crs as ccrs
+from matplotlib import colormaps as cm
 import pyproj
 import shapely
 from cartopy.io import shapereader
@@ -88,7 +89,7 @@
 warnprob_colors_extended = np.repeat(warnprob_colors, 10, axis=0)
 CMAP_WARNPROB = ListedColormap(warnprob_colors_extended)
 # colors for impact forecast
-color_map_pre = plt.get_cmap("plasma", 90)
+color_map_pre = cm.get_cmap("plasma").resampled(90)
 impact_colors = color_map_pre(np.linspace(0, 1, 90))
 white_extended = np.repeat([[255 / 255, 255 / 255, 255 / 255, 1]], 10, axis=0)
 impact_colors_extended = np.append(white_extended, impact_colors, axis=0)

climada/engine/impact_calc.py

@@ -112,7 +112,7 @@ def impact(self, save_mat=True, assign_centroids=True,
         apply_deductible_to_mat : apply deductible to impact matrix
         apply_cover_to_mat : apply cover to impact matrix
         """
-        # check for compability of exposures and hazard type
+        # check for compatibility of exposures and hazard type
         if all(name not in self.exposures.gdf.columns for
                name in ['if_', f'if_{self.hazard.haz_type}',
                         'impf_', f'impf_{self.hazard.haz_type}']):
@@ -121,14 +121,28 @@ def impact(self, save_mat=True, assign_centroids=True,
                 f"for hazard type {self.hazard.haz_type} in exposures."
                 )
 
-        # check for compability of impact function and hazard type
+        # check for compatibility of impact function and hazard type
         if not self.impfset.get_func(haz_type=self.hazard.haz_type):
             raise AttributeError(
                 "Impact calculation not possible. No impact functions found "
                 f"for hazard type {self.hazard.haz_type} in impf_set."
                 )
 
         impf_col = self.exposures.get_impf_column(self.hazard.haz_type)
+        known_impact_functions = self.impfset.get_ids(haz_type=self.hazard.haz_type)
+
+        # check for compatibility of impact function id between impact function set and exposure
+        if not all(self.exposures.gdf[impf_col].isin(known_impact_functions)):
+            unknown_impact_functions = list(self.exposures.gdf[
+                    ~self.exposures.gdf[impf_col].isin(known_impact_functions)
+                ][impf_col].drop_duplicates().astype(int).astype(str))
+            raise ValueError(
+                f"The associated impact function(s) with id(s) "
+                f"{', '.join(unknown_impact_functions)} have no match in impact function set for"
+                f" hazard type \'{self.hazard.haz_type}\'.\nPlease make sure that all exposure "
+                "points are associated with an impact function that is included in the impact "
+                "function set.")
+
         exp_gdf = self.minimal_exp_gdf(impf_col, assign_centroids, ignore_cover, ignore_deductible)
         if exp_gdf.size == 0:
             return self._return_empty(save_mat)

climada/engine/test/test_cost_benefit.py

@@ -35,16 +35,15 @@
 from climada.test import get_test_file
 
 
-HAZ_TEST_MAT = get_test_file('atl_prob_no_name', file_format='matlab')
 ENT_TEST_MAT = get_test_file('demo_today', file_format='MAT-file')
-
+HAZ_TEST_TC :Path = get_test_file('test_tc_florida')
 
 class TestSteps(unittest.TestCase):
     """Test intermediate steps"""
     def test_calc_impact_measures_pass(self):
         """Test _calc_impact_measures against reference value"""
-        self.assertTrue(HAZ_TEST_MAT.is_file(), "{} is not a file".format(HAZ_TEST_MAT))
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        self.assertTrue(HAZ_TEST_TC.is_file(), "{} is not a file".format(HAZ_TEST_TC))
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
 
         self.assertTrue(ENT_TEST_MAT.is_file(), "{} is not a file".format(ENT_TEST_MAT))
         entity = Entity.from_mat(ENT_TEST_MAT)
@@ -230,7 +229,7 @@ def test_cb_one_meas_fut_pass(self):
 
     def test_calc_cb_no_change_pass(self):
         """Test _calc_cost_benefit without present value against reference value"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_mat(ENT_TEST_MAT)
         entity.measures._data['TC'] = entity.measures._data.pop('XX')
         for meas in entity.measures.get_measure('TC'):
@@ -267,7 +266,7 @@ def test_calc_cb_no_change_pass(self):
 
     def test_calc_cb_change_pass(self):
         """Test _calc_cost_benefit with present value against reference value"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_mat(ENT_TEST_MAT)
         entity.measures._data['TC'] = entity.measures._data.pop('XX')
         for meas in entity.measures.get_measure('TC'):
@@ -438,7 +437,7 @@ def test_norm_value(self):
 
     def test_combine_fut_pass(self):
         """Test combine_measures with present and future"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -498,7 +497,7 @@ def test_combine_fut_pass(self):
 
     def test_combine_current_pass(self):
         """Test combine_measures with only future"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -538,7 +537,7 @@ def test_combine_current_pass(self):
 
     def test_apply_transf_current_pass(self):
         """Test apply_risk_transfer with only future"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -588,7 +587,7 @@ def test_apply_transf_current_pass(self):
 
     def test_apply_transf_cost_fact_pass(self):
         """Test apply_risk_transfer with only future annd cost factor"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -636,7 +635,7 @@ def test_apply_transf_cost_fact_pass(self):
 
     def test_apply_transf_future_pass(self):
         """Test apply_risk_transfer with present and future"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -692,7 +691,7 @@ def test_apply_transf_future_pass(self):
 
     def test_remove_measure(self):
         """Test remove_measure method"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -720,7 +719,7 @@ class TestCalc(unittest.TestCase):
     def test_calc_change_pass(self):
         """Test calc with future change"""
         # present
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.exposures.gdf.rename(columns={'impf_': 'impf_TC'}, inplace=True)
         entity.check()
@@ -777,7 +776,7 @@ def test_calc_change_pass(self):
 
     def test_calc_no_change_pass(self):
         """Test calc without future change"""
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         entity = Entity.from_excel(ENT_DEMO_TODAY)
         entity.check()
         entity.exposures.ref_year = 2018
@@ -808,7 +807,7 @@ class TestRiskFuncs(unittest.TestCase):
     def test_impact(self):
         ent = Entity.from_excel(ENT_DEMO_TODAY)
         ent.check()
-        hazard = Hazard.from_mat(HAZ_TEST_MAT)
+        hazard = Hazard.from_hdf5(HAZ_TEST_TC)
         impact = ImpactCalc(ent.exposures, ent.impact_funcs, hazard).impact()
         return impact
 

climada/engine/test/test_forecast.py

@@ -110,14 +110,12 @@ def test_Forecast_plot(self):
             HAZ_DIR.joinpath('storm_europe_cosmoe_forecast_vmax_testfile.nc'),
             run_datetime=dt.datetime(2018,1,1),
             event_date=dt.datetime(2018,1,3))
-        haz1.centroids.lat += 0.6
-        haz1.centroids.lon -= 1.2
+        haz1.centroids.gdf.geometry = haz1.centroids.gdf.geometry.translate(-1.2, 0.6)
         haz2 = StormEurope.from_cosmoe_file(
             HAZ_DIR.joinpath('storm_europe_cosmoe_forecast_vmax_testfile.nc'),
             run_datetime=dt.datetime(2018,1,1),
             event_date=dt.datetime(2018,1,3))
-        haz2.centroids.lat += 0.6
-        haz2.centroids.lon -= 1.2
+        haz2.centroids.gdf.geometry = haz2.centroids.gdf.geometry.translate(-1.2, 0.6)
         #exposure
         data = {}
         data['latitude'] = haz1.centroids.lat