Python 3 fixes, testing on both python2 and 3, not yet on CI, more st…

…rict tests enforcing same output for many functions between version

.gitignore

@@ -4,7 +4,6 @@
 .Spotlight-V100
 .Trashes
 .DS_Store
-*.ms
 *.buf
 *.hdr
 templates

.travis.yml

@@ -3,10 +3,10 @@ os:
   - linux
   # - osx  # Note, travis doesn't support python builds on osx
 python:
-  - "2.7_with_system_site_packages"
+  - "2.7"
   # - "3.3"
   # - "3.4"
-  # - "3.5"
+  - "3.5"
 # virtualenv:
 #     system_site_packages: true
 sudo: false
@@ -15,13 +15,6 @@ sudo: false
 env:
   - OBSPY_VERSION=1.0.1
 
-addons:
-  apt:
-    source:
-      - lucid
-    packages:
-      - pandoc
-
 # before install:
 #   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update        ; fi
 #   - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew install pandoc; fi
@@ -109,20 +102,21 @@ install:
   # - pip install -r requirements.txt
   - |
       if [[ "${py:0:1}" == '3' ]]; then
-        OPENCV="-c https://conda.anaconda.org/menpo opencv3"
+        OPENCV="-c menpo opencv3=3.1.0"
       else
         OPENCV="opencv"
       fi
   - conda install $OPENCV
-  - pip install coveralls
+  - conda install -c janschulz pypandoc=1.1.3
+  - conda install -c menpo coveralls=1.1
+  - conda install -c auto geographiclib=1.34
   - pip install pytest-cov
-  - pip install geographiclib
-  - pip install pypandoc
   # Copy opencv cv2.so file to path
   # - cp /usr/lib/pymodules/python2.7/cv2.so /home/travis/build/calum-chamberlain/EQcorrscan/.
   # current pyimgur stable release has a py2.6 incompatibility
-  - pip install https://github.com/megies/PyImgur/archive/py26.zip
-  - pip install obspy==$OBSPY_VERSION
+  # - pip install https://github.com/megies/PyImgur/archive/py26.zip
+  - conda install -c obspy obspy=$OBSPY_VERSION
+  # - pip install obspy==$OBSPY_VERSION
   - pip freeze
   - conda list
   # done installing dependencies

CHANGES.md

@@ -23,6 +23,15 @@ requires you to specify the station and channel to plot.
 * Add tests for stacking - PWS test needs more checks.
 * Add many examples to doc-strings, not complete though.
 * Change docs to have one page per function.
+* Python 3.5 testing underway, all tests pass, but only testing about 40% of
+codebase.
+* Use tox for testing.
+* Add io functions to match_filter to simplify detection handling including
+writing detections to catalog and to text file.
+* Stricter match_filter testing to enforce exactly the same result with a
+variety of systems.
+* Add hack to template_gen tutorial to fix differences in sorting between python 3.x
+and python 2.
 
 ## 0.1.1
 * Cope with events not always having time_errors in them in eventtoSfile;

README.md

@@ -10,10 +10,7 @@
 
 # Installation
 Installation has been tested on both OSX and Linux (Ubuntu), and now
-Windows systems.  Currently we only support python 2.7, but plan to extend
-this coverage in the near future - almost everything is written to work in python 3.x,
-we just need to get the testing going, work out the openCV3 install on travis and
-fix some small variable type changes.
+Windows systems.  We support Python versions 2.7, 3.3, 3.4 and 3.5.
 
 Installation for all systems should be as simple as:
 
@@ -35,18 +32,21 @@ this and support it, nevertheless, if you find any issues then let us know.
 
 If you have any issues installing please let me know.  
 
-You will need to install openCV (note that currently only openCV2 is tested,
-we plan on switching to openCV3 by default soon once the install is smoothed,
-this will ease transitions to python3.x) separately using (on Linux):
+You will need to install openCV (note that openCV versions 2 and 3 work for
+Python 2.7, but only openCV version 3 works for Python 3.x, therefore we
+recommend installing openCV 3). If you are running 64-Bit Linux,
+Windows or OSX, or 32-Bit Windows, you can simplify your install by running:
 
 ```bash
-apt-get install python-opencv
+conda install -c menpo opencv3=3.1.0
 ```
 
-Or, for Mac users, this is available on Macports or other similar package managers.
-
-For Windows users, you should follow the instructions [here](http://docs.opencv.org/3.1.0/d5/de5/tutorial_py_setup_in_windows.html#gsc.tab=0),
-note that you need to copy the cv2.pyd file.
+Otherwise, if you are running 32-Bit Linux, or 32-Bit OSX installation
+instructions can be found
+[here for ubuntu](http://www.pyimagesearch.com/2015/07/20/install-opencv-3-0-and-python-3-4-on-ubuntu/)
+and [here for OSX](http://www.pyimagesearch.com/2015/06/15/install-opencv-3-0-and-python-2-7-on-osx/).
+Note these two links are Python dependent and you will need to change your pip
+and python versions appropriate to your system.
 
 For those who want to run the GUIs (in very early development) you will need to
 install tk, on Windows and OSX this is usually pre-installed, on Linux you
@@ -56,6 +56,18 @@ may need to run:
 apt-get install python-tk
 ```
 
+*A note for Ubuntu 12.04 users and python 3.x*
+You will need the python3.x-dev libraries to install openCV if installing from
+source.  Getting these is a little difficult...  They are available by doing the
+following:
+```bash
+add-apt-repository ppa:fkrull/deadsnakes
+apt-get update
+apt-get install python3.x-dev
+```
+Note you will likely need root privileges for these actions, and you will need
+to replace the *x* with your version number.
+
 ## Updates
 
 If you want to be kept informed about releases, bug-tracking and enhancements

eqcorrscan/core/match_filter.py

@@ -72,7 +72,14 @@ def __init__(self, template_name, detect_time,
 
     def __repr__(self):
         """Simple print."""
-        return "DETECTION()"
+        print_str = ' '.join(['template name=', self.template_name, '\n',
+                              'detection time=', str(self.detect_time), '\n',
+                              'number of channels=', str(self.no_chans), '\n',
+                              'channels=', str(self.chans), '\n',
+                              'detection value=', str(self.detect_val), '\n',
+                              'threshold=', str(self.threshold), '\n',
+                              'detection type=', str(self.typeofdet)])
+        return "DETECTION(" + print_str + ")"
 
     def __str__(self):
         """Full print."""
@@ -82,6 +89,98 @@ def __str__(self):
                               str(self.chans)])
         return print_str
 
+    def write(self, fname, append=True):
+        """Write to file, will append if append==True and file exists
+
+        :type fname: str
+        :param fname: Full path to file to open and write to.
+        :type append: bool
+        :param appaned: Set to true to append to an existing file, if True \
+            and file doesn't exist, will create new file and warn.  If False
+            will overwrite old files.
+        """
+        import os
+        if append and os.path.isfile(fname):
+            f = open(fname, 'a')
+        else:
+            f = open(fname, 'w')
+            header = '; '.join(['Template name', 'Detection time (UTC)',
+                                'Number of channels', 'Channel list',
+                                'Detection value', 'Threshold',
+                                'Detection type'])
+            f.write(header + '\n')  # Write a header for the file
+        print_str = '; '.join([self.template_name, str(self.detect_time),
+                               str(self.no_chans), str(self.chans),
+                               str(self.detect_val), str(self.threshold),
+                               self.typeofdet])
+        f.write(print_str + '\n')
+        f.close()
+
+
+def read_detections(fname):
+    """Read detections from a file to a list of DETECTION objects.
+
+    :type fname: str
+    :param fanme: File to read from, must be a file written to by \
+        DETECTION.write.
+
+    :returns: list of DETECTION
+
+    .. note:: Does not return DETECTIONS containing events
+    """
+    from obspy import UTCDateTime
+    import ast
+    f = open(fname, 'r')
+    detections = []
+    for index, line in enumerate(f):
+        if index == 0:
+            continue  # Skip header
+        detection = line.rstrip().split('; ')
+        detection[1] = UTCDateTime(detection[1])
+        detection[2] = int(detection[2])
+        detection[3] = ast.literal_eval(detection[3])
+        detection[4] = float(detection[4])
+        detection[5] = float(detection[5])
+        detections.append(DETECTION(template_name=detection[0],
+                                    detect_time=detection[1],
+                                    no_chans=detection[2],
+                                    detect_val=detection[4],
+                                    threshold=detection[5],
+                                    typeofdet=detection[6],
+                                    chans=detection[3]))
+    f.close()
+    return detections
+
+
+def write_catalog(detections, fname, format="QUAKEML"):
+    """Write events contained within detections to a catalog file.
+
+    :type detections: list
+    :param detections: list of eqcorrscan.core.match_filter.DETECTION
+    :type fname: str
+    :param fname: Name of the file to write to
+    :type format: str
+    :param format: File format to use, see obspy.core.event.Catalog.write \
+        for supported formats.
+    """
+    catalog = get_catalog(detections)
+    catalog.write(filename=fname, format=format)
+
+
+def get_catalog(detections):
+    """Extract catalog from detections.
+
+    :type detections: list
+    :param detections: list of eqcorrscan.core.match_filter.DETECTION
+
+    :returns: obspy.core.event.Catalog
+    """
+    from obspy.core.event import Catalog
+    catalog = Catalog()
+    for detection in detections:
+        catalog.append(detection.event)
+    return catalog
+
 
 def extract_from_stream(stream, detections, pad=2.0, length=30.0):
     r"""Extract a list of detections from a stream.

eqcorrscan/tests/core_test.py

@@ -56,6 +56,23 @@ def test_normal_normxcorr2(self):
         ccc = normxcorr2(template, image)
         self.assertNotEqual(ccc.max(), 1.0)
 
+    def test_set_normxcorr2(self):
+        """Check that correlations output are the same irrespective of version.
+        """
+        import numpy as np
+        from eqcorrscan.core.match_filter import normxcorr2
+        from obspy import read
+        import os
+        testing_path = os.path.join(os.path.abspath(os.path.dirname(__file__)),
+                                    'test_data')
+        template = read(os.path.join(testing_path, 'test_template.ms'))
+        template = template[0].data.astype(np.float32)
+        image = read(os.path.join(testing_path, 'test_image.ms'))
+        image = image[0].data.astype(np.float32)
+        ccc = normxcorr2(template, image)[0]
+        expected_ccc = np.load(os.path.join(testing_path, 'test_ccc.npy'))
+        self.assertTrue((ccc == expected_ccc).all())
+
     def test_perfect_template_loop(self):
         """Check that perfect correlations are carried through.
         """

eqcorrscan/tests/find_peaks_test.py

@@ -0,0 +1,29 @@
+"""
+Functions for testing the utils.findpeaks functions
+"""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from __future__ import unicode_literals
+import unittest
+
+
+class TestTutorialScripts(unittest.TestCase):
+    def test_main_find_peaks(self):
+        """Test find_peaks2_short"""
+        from eqcorrscan.utils.findpeaks import find_peaks2_short
+        import numpy as np
+        import os
+        testing_path = os.path.join(os.path.abspath(os.path.dirname(__file__)),
+                                    'test_data')
+        expected_ccc = np.load(os.path.join(testing_path, 'test_ccc.npy'))
+        peaks = find_peaks2_short(arr=expected_ccc, thresh=0.2, trig_int=10,
+                                  debug=0, starttime=False, samp_rate=200.0)
+        expected_peaks = np.load(os.path.join(testing_path, 'test_peaks.npy'))
+        self.assertTrue((np.array(peaks) == expected_peaks).all())
+
+if __name__ == '__main__':
+    """
+    Run tests
+    """
+    unittest.main()

eqcorrscan/tests/sac_template_test.py → eqcorrscan/tests/template_gen_test.py

@@ -8,9 +8,9 @@
 import unittest
 
 
-class TestSACTemplateGeneration(unittest.TestCase):
+class TestTemplateGeneration(unittest.TestCase):
     """Test the reading a writing of pick info."""
-    def test_template_gen(self):
+    def test_sac_template_gen(self):
         """Test template generation."""
         from eqcorrscan.core.template_gen import from_sac
         import glob
@@ -34,7 +34,8 @@ def test_template_gen(self):
                 template = from_sac(sac_files, lowcut=2.0, highcut=8.0,
                                     samp_rate=samp_rate, filt_order=4,
                                     length=length,
-                                    swin='all', prepick=0.1, debug=0, plot=False)
+                                    swin='all', prepick=0.1, debug=0,
+                                    plot=False)
                 for tr in template:
                     self.assertEqual(len(tr.data), length * samp_rate)
             else:
@@ -45,5 +46,33 @@ def test_template_gen(self):
                                         swin='all', prepick=0.1, debug=0,
                                         plot=False)
 
+    def test_tutorial_template_gen(self):
+        """Test template generation from tutorial, uses from_client method.
+
+        Checks that the tutorial generates the templates we expect it to!
+        """
+        from obspy import read
+        from eqcorrscan.tutorials.template_creation import mktemplates
+        import os
+        import numpy as np
+
+        testing_path = os.path.join(os.path.abspath(os.path.dirname(__file__)),
+                                    'test_data')
+        mktemplates(plot=False)
+        for template_no in range(4):
+            template = read('tutorial_template_' + str(template_no) + '.ms')
+            expected_template = read(os.path.join(testing_path,
+                                                  'tutorial_template_' +
+                                                  str(template_no) + '.ms'))
+            for tr in template:
+                expected_tr = expected_template.select(station=tr.stats.
+                                                       station,
+                                                       channel=tr.stats.
+                                                       channel)[0]
+                self.assertTrue((expected_tr.data.astype(np.float32) ==
+                                 tr.data.astype(np.float32)).all())
+            del(template)
+            os.remove('tutorial_template_' + str(template_no) + '.ms')
+
 if __name__ == '__main__':
     unittest.main()

eqcorrscan/tests/test_data/expected_tutorial_detections.txt

@@ -0,0 +1,24 @@
+Template name; Detection time (UTC); Number of channels; Channel list; Detection value; Threshold; Detection type
+tutorial_template_3.ms; 2016-01-04T00:13:23.498399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.03043; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T00:13:39.398399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 4.00244; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T00:45:58.148399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 5.0; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T01:21:42.748399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.53477; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T01:48:52.948399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 3.67236; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T02:17:23.898399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.20337; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T03:30:07.898399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 1.84586; 1.78002929688; corr
+tutorial_template_3.ms; 2016-01-04T09:22:14.048399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.95605; 1.78002929688; corr
+tutorial_template_1.ms; 2016-01-04T01:57:21.398399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.50073; 1.81201171875; corr
+tutorial_template_1.ms; 2016-01-04T02:01:03.098399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 1.88281; 1.81201171875; corr
+tutorial_template_1.ms; 2016-01-04T02:10:47.898399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 5.0; 1.81201171875; corr
+tutorial_template_2.ms; 2016-01-04T00:15:11.898399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 3.90381; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T00:17:02.098399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.35736; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T00:18:42.248399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.69165; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T00:23:15.148399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.11035; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T00:25:08.198399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 5.0; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T03:24:01.398399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.39648; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T09:21:42.998399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 3.37061; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T12:57:19.298399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.39505; 1.759765625; corr
+tutorial_template_2.ms; 2016-01-04T14:01:08.798399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 3.09814; 1.759765625; corr
+tutorial_template_0.ms; 2016-01-04T00:08:01.848399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 5.0; 2.13610839844; corr
+tutorial_template_0.ms; 2016-01-04T00:49:00.398399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.14001; 2.13610839844; corr
+tutorial_template_0.ms; 2016-01-04T16:29:43.298399Z; 5; [(u'BFZ', u'HZ'), (u'DVHZ', u'EZ'), (u'HOWZ', u'EZ'), (u'POWZ', u'EZ'), (u'CPWZ', u'EZ')]; 2.30841; 2.13610839844; corr