Skip to content

lookup_linke_turbidity speed improvement, mat to h5 #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 30, 2018
Merged

lookup_linke_turbidity speed improvement, mat to h5 #442

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
6ed6d6d
lookup_linke_turbidity speed improvement (Issue #437)
cedricleroy Mar 28, 2018
759e22f
Fix old references to .mat LT file in docstrings / rst doc
cedricleroy Mar 28, 2018
f23483d
tables is pytables in conda
cedricleroy Mar 29, 2018
1e45484
mention .mat to .h5 in whatsnew
cedricleroy Mar 29, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ install:
- cmd: conda info -a

# install depenencies
- cmd: conda create -n test_env --yes --quiet python=%PYTHON_VERSION% pip numpy scipy pandas nose pytest pytz ephem numba siphon -c conda-forge
- cmd: conda create -n test_env --yes --quiet python=%PYTHON_VERSION% pip numpy scipy pytables pandas nose pytest pytz ephem numba siphon -c conda-forge
- cmd: activate test_env
- cmd: python --version
- cmd: conda list
Expand Down
1 change: 1 addition & 0 deletions ci/requirements-py27.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies:
- python=2.7
- numpy
- scipy
- pytables
- pandas
- pytz
- ephem
Expand Down
1 change: 1 addition & 0 deletions ci/requirements-py34.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies:
- python=3.4
- numpy
- scipy
- pytables
- pandas
- pytz
- ephem
Expand Down
1 change: 1 addition & 0 deletions ci/requirements-py35.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies:
- python=3.5
- numpy
- scipy
- pytables
- pandas
- pytz
- ephem
Expand Down
1 change: 1 addition & 0 deletions ci/requirements-py36.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ dependencies:
- python=3.6
- numpy
- scipy
- pytables
- pandas
- pytz
#- ephem
Expand Down
1 change: 1 addition & 0 deletions docs/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ dependencies:
- mock # needed for local python 2.7 builds
- numpy=1.11.2
- scipy
- tables
- pandas=0.19.1
- pytz
- ephem
Expand Down
12 changes: 6 additions & 6 deletions docs/sphinx/source/clearsky.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,24 +136,24 @@ the year. You could run it in a loop to create plots for all months.

In [1]: import os

In [1]: import scipy.io
In [1]: import tables

In [1]: pvlib_path = os.path.dirname(os.path.abspath(pvlib.clearsky.__file__))

In [1]: filepath = os.path.join(pvlib_path, 'data', 'LinkeTurbidities.mat')

In [1]: mat = scipy.io.loadmat(filepath)
In [1]: filepath = os.path.join(pvlib_path, 'data', 'LinkeTurbidities.h5')

# data is in units of 20 x turbidity
In [1]: linke_turbidity_table = mat['LinkeTurbidity'] # / 20. # crashes on rtd
In [1]: lt_h5_file = tables.open_file(filepath)

In [1]: def plot_turbidity_map(month, vmin=1, vmax=100):
...: plt.figure();
...: plt.imshow(linke_turbidity_table[:, :, month-1], vmin=vmin, vmax=vmax);
...: plt.imshow(lt_h5_file.root.LinkeTurbidity[:, :, month-1], vmin=vmin, vmax=vmax);
...: plt.title('Linke turbidity x 20, ' + calendar.month_name[month]);
...: plt.colorbar(shrink=0.5);
...: plt.tight_layout();

In [1]: lt_h5_file.close()

@savefig turbidity-1.png width=10in
In [1]: plot_turbidity_map(1)

Expand Down
6 changes: 3 additions & 3 deletions docs/sphinx/source/whatsnew/v0.5.2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ API Changes

Enhancements
~~~~~~~~~~~~
*
* Improve clearsky.lookup_linke_turbidity speed, changing .mat source
file to .h5 (:issue:`437`)

Bug fixes
~~~~~~~~~
Expand Down Expand Up @@ -42,5 +43,4 @@ Contributors
* Will Holmgren
* KonstantinTr
* Anton Driesse


* Cedric Leroy
26 changes: 15 additions & 11 deletions pvlib/clearsky.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ def ineichen(apparent_zenith, airmass_absolute, linke_turbidity,
def lookup_linke_turbidity(time, latitude, longitude, filepath=None,
interp_turbidity=True):
"""
Look up the Linke Turibidity from the ``LinkeTurbidities.mat``
Look up the Linke Turibidity from the ``LinkeTurbidities.h5``
data file supplied with pvlib.

Parameters
Expand All @@ -165,18 +165,18 @@ def lookup_linke_turbidity(time, latitude, longitude, filepath=None,
longitude : float

filepath : None or string, default None
The path to the ``.mat`` file.
The path to the ``.h5`` file.

interp_turbidity : bool, default True
If ``True``, interpolates the monthly Linke turbidity values
found in ``LinkeTurbidities.mat`` to daily values.
found in ``LinkeTurbidities.h5`` to daily values.

Returns
-------
turbidity : Series
"""

# The .mat file 'LinkeTurbidities.mat' contains a single 2160 x 4320 x 12
# The .h5 file 'LinkeTurbidities.h5' contains a single 2160 x 4320 x 12
# matrix of type uint8 called 'LinkeTurbidity'. The rows represent global
# latitudes from 90 to -90 degrees; the columns represent global longitudes
# from -180 to 180; and the depth (third dimension) represents months of
Expand All @@ -194,18 +194,15 @@ def lookup_linke_turbidity(time, latitude, longitude, filepath=None,
# 1st column: 179.9583 W, 2nd column: 179.875 W

try:
import scipy.io
import tables
except ImportError:
raise ImportError('The Linke turbidity lookup table requires scipy. '
raise ImportError('The Linke turbidity lookup table requires tables. '
'You can still use clearsky.ineichen if you '
'supply your own turbidities.')

if filepath is None:
pvlib_path = os.path.dirname(os.path.abspath(__file__))
filepath = os.path.join(pvlib_path, 'data', 'LinkeTurbidities.mat')

mat = scipy.io.loadmat(filepath)
linke_turbidity_table = mat['LinkeTurbidity']
filepath = os.path.join(pvlib_path, 'data', 'LinkeTurbidities.h5')

latitude_index = (
np.around(_linearly_scale(latitude, 90, -90, 0, 2160))
Expand All @@ -214,7 +211,14 @@ def lookup_linke_turbidity(time, latitude, longitude, filepath=None,
np.around(_linearly_scale(longitude, -180, 180, 0, 4320))
.astype(np.int64))

lts = linke_turbidity_table[latitude_index][longitude_index]
lt_h5_file = tables.open_file(filepath)
try:
lts = lt_h5_file.root.LinkeTurbidity[latitude_index, longitude_index, :]
except IndexError:
raise IndexError('Latitude should be between 90 and -90, '
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a test that asserts that this exception is raised under invalid inputs.

Also, I confirmed that the code below does in fact execute the finally block despite a new exception being raised.

try:
    raise IndexError
except IndexError:
    raise IndexError('blah')
finally:
    print('finally')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually did write a test for that, but then realized it was already covered.

For the try except finally, I was hoping open_file to come with a context manager, but it does not look like it is the case and fall back to the old school way to make sure the file get closed whatever happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my bad, thanks for reminding me...

'longitude between -180 and 180.')
finally:
lt_h5_file.close()

if interp_turbidity:
linke_turbidity = _interpolate_turbidity(lts, time)
Expand Down
Binary file renamed pvlib/data/LinkeTurbidities.mat → pvlib/data/LinkeTurbidities.h5
View file
Open in desktop
Binary file not shown.
9 changes: 9 additions & 0 deletions pvlib/test/conftest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,15 @@
requires_scipy = pytest.mark.skipif(not has_scipy, reason='requires scipy')


try:
import tables
has_tables = True
except ImportError:
has_tables = False

requires_tables = pytest.mark.skipif(not has_tables, reason='requires tables')


try:
import ephem
has_ephem = True
Expand Down
16 changes: 8 additions & 8 deletions pvlib/test/test_clearsky.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
from pvlib import atmosphere
from pvlib import irradiance

from conftest import requires_scipy
from conftest import requires_scipy, requires_tables


def test_ineichen_series():
Expand Down Expand Up @@ -158,7 +158,7 @@ def test_ineichen_altitude():
assert_frame_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity():
times = pd.date_range(start='2014-06-24', end='2014-06-25',
freq='12h', tz='America/Phoenix')
Expand All @@ -171,7 +171,7 @@ def test_lookup_linke_turbidity():
assert_series_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity_leapyear():
times = pd.date_range(start='2016-06-24', end='2016-06-25',
freq='12h', tz='America/Phoenix')
Expand All @@ -184,7 +184,7 @@ def test_lookup_linke_turbidity_leapyear():
assert_series_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity_nointerp():
times = pd.date_range(start='2014-06-24', end='2014-06-25',
freq='12h', tz='America/Phoenix')
Expand All @@ -195,7 +195,7 @@ def test_lookup_linke_turbidity_nointerp():
assert_series_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity_months():
times = pd.date_range(start='2014-04-01', end='2014-07-01',
freq='1M', tz='America/Phoenix')
Expand All @@ -206,7 +206,7 @@ def test_lookup_linke_turbidity_months():
assert_series_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity_months_leapyear():
times = pd.date_range(start='2016-04-01', end='2016-07-01',
freq='1M', tz='America/Phoenix')
Expand All @@ -217,7 +217,7 @@ def test_lookup_linke_turbidity_months_leapyear():
assert_series_equal(expected, out)


@requires_scipy
@requires_tables
def test_lookup_linke_turbidity_nointerp_months():
times = pd.date_range(start='2014-04-10', end='2014-07-10',
freq='1M', tz='America/Phoenix')
Expand Down Expand Up @@ -448,7 +448,7 @@ def test_simplified_solis_nans_series():
assert_frame_equal(expected, out)


@requires_scipy
@requires_tables
def test_linke_turbidity_corners():
"""Test Linke turbidity corners out of bounds."""
months = pd.DatetimeIndex('%d/1/2016' % (m + 1) for m in range(12))
Expand Down