Use a static grid, asymmetric region, and xarray.testing for raster tests

[maxrjones]

This issue was originally for discussing improvements to the PyGMT test suite and is now dedicated to tracking specific improvements:

1. Use xarray.testing rather than pygmt.grdinfo
2. Use static_earth_relief rather than earth_relief
3. Use asymmetric regions to facilitate identification of lat/lon errors

If anyone wants to help, please edit this comment to add your username next to the module or post a comment below to avoid redundant work.

• Add static_earth_relief to cache (Add small grid based on earth_relief_01d V2.3 gmtserver-admin#139)
• Add function to load static_earth_relief.nc as an xarray.DataArray (Add load_static_earth_relief function for internal testing #1727)
• test_grd2cpt.py (static_earth_relief) (@willschlitzer Add static_earth_relief to grd2cpt tests #1736)
• test_grd2xyz.py (static_earth_relief) (@willschlitzer Add static_earth_relief to grd2xyz tests #1737)
• test_grdclip.py (xarray.testing) (@meghanrjones, Use xarray.testing rather than pygmt.grdinfo in grdclip tests #1573)
• test_grdclip.py (static_earth_relief) (@willschlitzer Add static_earth_relief to grdclip tests #1738)
• text_grdcontour.py (static_earth_relief) (@willschlitzer Change test_grdcontour.py to use static_earth_relief #1741)
• test_grdcut.py (xarray.testing) (@meghanrjones, Use xarray.testing rather than pygmt.grdinfo in grdcut tests #1574)
• test_grdcut.py (static_earth_relief) (@willschlitzer Change test_grdcut.py to use static_earth_relief #1742)
• test_grdfill.py (xarray.testing) (@willschlitzer, Use xarray.testing rather than pygmt.grdinfo in grdfill tests #1677)
• test_grdfill.py (static_earth_relief) (@willschlitzer Change test_grdfill.py to use static_earth_relief #1743)
• test_grdfilter.py (xarray.testing and static_earth_relief) (@willschlitzer, Change test_grdfilter.py to use static_earth_relief #1678)
• test_grdgradient.py (xarray.testing and static_earth_relief) (@willschlitzer, Change test_grdgradient.py to use static_earth_relief #1679)
• test_grdinfo.py (static_earth_relief) (@willschlitzer Change test_grdinfo.py to use static_earth_relief #1746)
• test_grdlandmask.py (xarray.testing) (@willschlitzer, Use xarray.testing rather than pygmt.grdinfo in grdlandmask tests #1680)
• test_grdproject.py (xarray.testing and static_earth_relief) (@willschlitzer Change test_grdproject.py to use static_earth_relief #1683)
• test_grdsample.py (xarray.testing and static_earth_relief) (@willschlitzer, Change test_grdsample.py to use static_earth_relief #1681)
• test_grdtrack.py (static_earth_relief) (@willschlitzer, Change test_grdtrack.py to use static_earth_relief #1762)
• test_grdview.py (static_earth_relief) (@willschlitzer Change test_grdview.py to use static_earth_relief #1752)
• test_grdvolume.py (static_earth_relief) (@willschlitzer Change test_grdvolume.py to use static_earth_relief #1747)
• test_makecpt.py (static_earth_relief) (@willschlitzer Change test_makecpt.py to use colorbar instead of grdimage #1748)
• test_xyz2grd.py (xarray.testing and static_earth_relief) (@willschlitzer Use xarray.testing rather than pygmt.grdinfo in xyz2grd tests #1682)
• pygmt.clib.session example (static_earth_relief) (@willschlitzer Change session.py to use static_earth_relief #1761)

[maxrjones]

Thanks @willschlitzer for getting started on these tests! As @weiji14 mentioned in #593 (comment), we may want to work to address the likely failures due to upcoming @earth_relief failures during this process of migrating away from grdinfo. I have a few questions/comments regarding the tests:

1. Should we continue using the @earth_relief remote datasets for grid processing/plotting tests at all? We could instead create a xarray.DataArray with a subset of the V2.1 SRTM15+ data using a new helper function (e.g., pygmt.helpers.testing.static_earth_relief(). Probably no one wants to write out the data for 360 x 180 grid, so this would require determining whether any of the tests using global grids could use a smaller subset. The tests that use @earth_relief are listed below for context. For small grids (10x10), it is ~5% quicker to create a grid than use the remote data and could mean less maintenance work for future SRTM15+ updates.

2. I think we should move away from regions where the latitudes and longitudes could be switched without notice (e.g., region = [-5, 5, -5, 5]

3. We may need to adjust the rtol on assert_allclose for some tests - I'll post more on this later but accidentally opened this issue before working out the precision factors.

Processing/plotting tests using load_earth_relief:

Global pixel registration

• test_grdcut.py - load_earth_relief(registration="pixel")
• test_grd2cpt.py - load_earth_relief()
• test_grdfilter.py - load_earth_relief(registration="pixel”)

Global gridline registration

• test_grdimage.py - load_earth_relief(registration="gridline")
• test_grdinfo.py - load_earth_relief(registration="gridline")
• test_grdcontour.py - load_earth_relief(registration="gridline”)
• test_makecpt.py - load_earth_relief(registration="gridline”)

Symmetric subset

• test_grdgradient.py - load_earth_relief(resolution="01d", region=[-5, 5, -5, 5])
• test_grdproject.py - load_earth_relief(resolution="01d", region=[-5, 5, -5, 5])
• test_grdclip.py - load_earth_relief(resolution="01d", region=[-5, 5, -5, 5]
• test_grdsample.py - load_earth_relief(resolution="01d", region=[-5, 5, -5, 5], registration=“pixel”)

Other subset

• test_grd2xyz.py - load_earth_relief(resolution="01d", region=[-1, 1, 3, 5]
• test_grdtrack.py - load_earth_relief(registration="gridline").sel(lat=slice(-49, -42), lon=slice(-118, -107))
• test_grdvolume.py - load_earth_relief(resolution="01d", region=[-100, -95, 34, 39])
• test_grdfill.py - load_earth_relief(registration="pixel", region=[125, 130, -25, -20])

[maxrjones]

These are the tests that can be expected to fail after the @earth_relief update to V2.3 (n=45):

FAILED ../pygmt/clib/session.py::pygmt.clib.session.Session
FAILED ../pygmt/clib/session.py::pygmt.clib.session.Session.virtualfile_from_grid
FAILED ../pygmt/tests/test_datasets_earth_relief.py::test_earth_relief_01d
FAILED ../pygmt/tests/test_datasets_earth_relief.py::test_earth_relief_01d_with_region
FAILED ../pygmt/tests/test_datasets_earth_relief.py::test_earth_relief_30m
FAILED ../pygmt/tests/test_grd2cpt.py::test_grd2cpt
FAILED ../pygmt/tests/test_grdclip.py::test_grdclip_outgrid
FAILED ../pygmt/tests/test_grdclip.py::test_grdclip_no_outgrid
FAILED ../pygmt/tests/test_grdcontour.py::test_grdcontour
FAILED ../pygmt/tests/test_grdcontour.py::test_grdcontour_labels
FAILED ../pygmt/tests/test_grdcontour.py::test_grdcontour_slice
FAILED ../pygmt/tests/test_grdcontour.py::test_grdcontour_file
FAILED ../pygmt/tests/test_grdcontour.py::test_grdcontour_interval_file_full_opts
FAILED ../pygmt/tests/test_grdcut.py::test_grdcut_file_in_file_out
FAILED ../pygmt/tests/test_grdcut.py::test_grdcut_file_in_dataarray_out
FAILED ../pygmt/tests/test_grdcut.py::test_grdcut_dataarray_in_file_out
FAILED ../pygmt/tests/test_grdcut.py::test_grdcut_dataarray_in_dataarray_out
FAILED ../pygmt/tests/test_grdfill.py::test_grdfill_file_out
FAILED ../pygmt/tests/test_grdfilter.py::test_grdfilter_dataarray_in_dataarray_out
FAILED ../pygmt/tests/test_grdfilter.py::test_grdfilter_dataarray_in_file_out
FAILED ../pygmt/tests/test_grdfilter.py::test_grdfilter_file_in_dataarray_out
FAILED ../pygmt/tests/test_grdfilter.py::test_grdfilter_file_in_file_out
FAILED ../pygmt/tests/test_grdgradient.py::test_grdgradient_outgrid
FAILED ../pygmt/tests/test_grdgradient.py::test_grdgradient_no_outgrid
FAILED ../pygmt/tests/test_grdinfo.py::test_grdinfo
FAILED ../pygmt/tests/test_grdinfo.py::test_grdinfo_file
FAILED ../pygmt/tests/test_grdinfo.py::test_grdinfo_region
FAILED ../pygmt/tests/test_grdproject.py::test_grdproject_file_out
FAILED ../pygmt/tests/test_grdproject.py::test_grdproject_no_outgrid
FAILED ../pygmt/tests/test_grdtrack.py::test_grdtrack_input_dataframe_and_dataarray
FAILED ../pygmt/tests/test_grdtrack.py::test_grdtrack_input_csvfile_and_dataarray
FAILED ../pygmt/tests/test_grdtrack.py::test_grdtrack_input_dataframe_and_ncfile
FAILED ../pygmt/tests/test_grdtrack.py::test_grdtrack_input_csvfile_and_ncfile
FAILED ../pygmt/tests/test_grdtrack.py::test_grdtrack_without_outfile_setting
FAILED ../pygmt/tests/test_grdview.py::test_grdview_with_perspective_and_zscale
FAILED ../pygmt/tests/test_grdview.py::test_grdview_with_perspective_and_zsize
FAILED ../pygmt/tests/test_grdview.py::test_grdview_with_cmap_for_perspective_surface_plot
FAILED ../pygmt/tests/test_grdview.py::test_grdview_on_a_plane
FAILED ../pygmt/tests/test_grdview.py::test_grdview_on_a_plane_with_colored_frontal_facade
FAILED ../pygmt/tests/test_grdview.py::test_grdview_with_perspective_and_zaxis_frame
FAILED ../pygmt/tests/test_grdview.py::test_grdview_surface_plot_styled_with_contourpen
FAILED ../pygmt/tests/test_grdview.py::test_grdview_surface_mesh_plot_styled_with_meshpen
FAILED ../pygmt/tests/test_grdview.py::test_grdview_on_a_plane_styled_with_facadepen
FAILED ../pygmt/tests/test_makecpt.py::test_makecpt_truncated_zlow_zhigh
FAILED ../pygmt/tests/test_makecpt.py::test_makecpt_continuous

[willschlitzer]

I think this is a good idea. Having a standardized grid should make it easier to write tests, and will save us time in the future. If nothing else, we should move grids to be over-land, as I assume land data is less likely to change instead of ocean data.

[willschlitzer]

@meghanrjones Should we work to change the region arguments for the tests that have symmetric or global regions right now, or wait until the earth_relief update comes and we also (hopefully) move to a local xarray file?

[maxrjones]

I think it would be better to update the region at the same as migrating to a locally defined xarray.DataArray to reduce reviewer work. If we chose to use a function that creates a DataArray rather than the remote files, we do not need to wait for the earth_relief update since we can create the grid based on the contents of SRTM15+V2.1.

[willschlitzer]

Are we planning on using preset DataArrays rather than grids from load_earth_relief? Now that the new Earth relief files have been pushed, it seems like we should decide how this issue will be handled.

[weiji14]

I quite like @meghanrjones's idea at #1684 (comment) to have a pygmt.helpers.testing.static_earth_relief() function which could be a near drop-in replacement for existing tests using pygmt.datasets.load_earth_relief(). Or at least some function in pygmt.helpers.testing which loads a standard grid we can re-use across all tests.

There are some unit tests that run on an actual file (e.g. in test_grdimage.py) rather than an xarray.DataArray grid, so I'm thinking whether it's worth caching an @earth_relief_01d_p_v2.3 file (and possibly the gridline-registered version) at https://github.com/GenericMappingTools/gmtserver-admin/tree/master/cache which pygmt.helpers.testing.static_earth_relief() would pull from. Alternatively, we could generate a purely synthetic grid, even though I feel that testing on semi-real data is better.

[maxrjones]

There are some unit tests that run on an actual file (e.g. in test_grdimage.py) rather than an xarray.DataArray grid, so I'm thinking whether it's worth caching an @earth_relief_01d_p_v2.3 file (and possibly the gridline-registered version) at https://github.com/GenericMappingTools/gmtserver-admin/tree/master/cache which pygmt.helpers.testing.static_earth_relief() would pull from. Alternatively, we could generate a purely synthetic grid, even though I feel that testing on semi-real data is better.

We could also use @tut_bathy.nc for some tests rather than @earth_relief_01d. It's a bit smaller (11 KB versus 114 KB) and wouldn't require a new cache file.

gmt grdinfo @tut_bathy.nc
gmt grdimage @tut_bathy.nc -B -png test

Title: ETOPO5 global topography
Command: grdreformat -fg bermuda.grd bermuda.nc=ns
Remark: /home/elepaio5/data/grids/etopo5.i2
Gridline node registration used [Geographic grid]
Grid file format: ns = GMT netCDF format (16-bit integer), CF-1.7
x_min: -66 x_max: -60 x_inc: 0.0833333333333 (5 min) name: Longitude n_columns: 73
y_min: 30 y_max: 35 y_inc: 0.0833333333333 (5 min) name: Latitude n_rows: 61
v_min: -5475 v_max: -89 name: Topography [m]
scale_factor: 1 add_offset: 0
format: classic
Default CPT: 

[willschlitzer]

There are some unit tests that run on an actual file (e.g. in test_grdimage.py) rather than an xarray.DataArray grid, so I'm thinking whether it's worth caching an @earth_relief_01d_p_v2.3 file (and possibly the gridline-registered version) at https://github.com/GenericMappingTools/gmtserver-admin/tree/master/cache which pygmt.helpers.testing.static_earth_relief() would pull from. Alternatively, we could generate a purely synthetic grid, even though I feel that testing on semi-real data is better.

We could also use @tut_bathy.nc for some tests rather than @earth_relief_01d. It's a bit smaller (11 KB versus 114 KB) and wouldn't require a new cache file.

This still seems like it covers a pretty big area/has a lot of data; wouldn't it speed up our plotting tests more to cache a smaller grid file (something like 10 x10 degree squadre at 30 minute resolution)?

[weiji14]

There are some unit tests that run on an actual file (e.g. in test_grdimage.py) rather than an xarray.DataArray grid, so I'm thinking whether it's worth caching an @earth_relief_01d_p_v2.3 file (and possibly the gridline-registered version) at https://github.com/GenericMappingTools/gmtserver-admin/tree/master/cache which pygmt.helpers.testing.static_earth_relief() would pull from. Alternatively, we could generate a purely synthetic grid, even though I feel that testing on semi-real data is better.

We could also use @tut_bathy.nc for some tests rather than @earth_relief_01d. It's a bit smaller (11 KB versus 114 KB) and wouldn't require a new cache file.

This still seems like it covers a pretty big area/has a lot of data; wouldn't it speed up our plotting tests more to cache a smaller grid file (something like 10 x10 degree squadre at 30 minute resolution)?

Ok, how about we make a pygmt.helpers.testing.static_relief() based on a subset of the @tut_bathy.nc? Further to Meghan's point, we probably want a rectangular sample grid (e.g. 4 x 6 pixels) so that we don't get accidental errors with longitude and latitude mixed up.

[willschlitzer]

There are some unit tests that run on an actual file (e.g. in test_grdimage.py) rather than an xarray.DataArray grid, so I'm thinking whether it's worth caching an @earth_relief_01d_p_v2.3 file (and possibly the gridline-registered version) at https://github.com/GenericMappingTools/gmtserver-admin/tree/master/cache which pygmt.helpers.testing.static_earth_relief() would pull from. Alternatively, we could generate a purely synthetic grid, even though I feel that testing on semi-real data is better.

We could also use @tut_bathy.nc for some tests rather than @earth_relief_01d. It's a bit smaller (11 KB versus 114 KB) and wouldn't require a new cache file.

This still seems like it covers a pretty big area/has a lot of data; wouldn't it speed up our plotting tests more to cache a smaller grid file (something like 10 x10 degree squadre at 30 minute resolution)?

Ok, how about we make a pygmt.helpers.testing.static_relief() based on a subset of the @tut_bathy.nc? Further to Meghan's point, we probably want a rectangular sample grid (e.g. 4 x 6 pixels) so that we don't get accidental errors with longitude and latitude mixed up.

That works for me.