RDSE: serialization for py bindings

[breznak]

TODO

• py test for pickle

Fixes #600

[breznak]

This is waiting for @leonardbcm for the tests, but if we don't hear from him, can be merged as is

[leonardtschora]

Hi, thanks for the support again.

I tried serializing and deserializing rdse objects with pickle and it worked fine.

I have written this following piece of code :

def testPickle(self):
        """
        The pickling is successfull if pickle serializes and de-serialize the
        RDSE object. 
        Moreover, the de-serialized object shall give the same SDR than the 
        original encoder given the same scalar value to encode.
        """        
        rdse_params = RDSE_Parameters()
        rdse_params.sparsity = 0.1
        rdse_params.size = 100
        rdse_params.resolution = 0.1
        rdse_params.seed = 1997

        rdse = RDSE(rdse_params)
        filename = "RDSE_testPickle"

        try:
            with open(filename, "wb") as f:
                pickle.dump(rdse, f)
        except:
            dump_success = False
        else:
            dump_success = True

        assert(dump_success)

        try:
            with open(filename, "rb") as f:
                rdse_loaded = pickle.load(f)
        except:
            read_success = False
        else:
            read_success = True

        assert(read_success)
        value_to_encode = 69003        
        SDR_original = rdse.encode(value_to_encode)
        SDR_loaded = rdse_loaded.encode(value_to_encode)

        assert(SDR_original == SDR_loaded)

Which gives the following logs :

C:\Users\Leonard\Documents\htm.core>pytest bindings/py/tests/encoders/rdse_test.py
============================================================================================== test session starts ===============================================================================================
platform win32 -- Python 3.7.2, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- c:\users\leonard\appdata\local\programs\python\python37\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\Leonard\Documents\htm.core, inifile: setup.cfg
plugins: cov-2.7.1
collected 9 items

bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testAverageOverlap PASSED                                                                                                                               [ 11%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testConstructor PASSED                                                                                                                                  [ 22%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testDeterminism PASSED                                                                                                                                  [ 33%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testErrorChecks PASSED                                                                                                                                  [ 44%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testPickle FAILED                                                                                                                                       [ 55%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testRadiusResolution PASSED                                                                                                                             [ 66%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testRandomOverlap PASSED                                                                                                                                [ 77%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testSeed PASSED                                                                                                                                         [ 88%]
bindings/py/tests/encoders/rdse_test.py::RDSE_Test::testSparsityActiveBits PASSED                                                                                                                           [100%]

==================================================================================================== FAILURES ====================================================================================================
______________________________________________________________________________________________ RDSE_Test.testPickle ______________________________________________________________________________________________

self = <rdse_test.RDSE_Test testMethod=testPickle>

    def testPickle(self):
        """
        The pickling is successfull if pickle serializes and de-serialize the
        RDSE object.
        Moreover, the de-serialized object shall give the same SDR than the
        original encoder given the same scalar value to encode.
        """
        rdse_params = RDSE_Parameters()
        rdse_params.sparsity = 0.1
        rdse_params.size = 100
        rdse_params.resolution = 0.1
        rdse_params.seed = 1997

        rdse = RDSE(rdse_params)
        filename = "RDSE_testPickle"

        try:
            with open(filename, "wb") as f:
                pickle.dump(rdse, f)
        except:
            dump_success = False
        else:
            dump_success = True

        assert(dump_success)

        try:
            with open(filename, "rb") as f:
                rdse_loaded = pickle.load(f)
        except:
            read_success = False
        else:
            read_success = True

        assert(read_success)
        value_to_encode = 69003
        SDR_original = rdse.encode(value_to_encode)
>       SDR_loaded = rdse_loaded.encode(value_to_encode)
E       RuntimeError: CHECK FAILED: "output.size == size"

bindings\py\tests\encoders\rdse_test.py:295: RuntimeError
---------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
ERR:  CHECK FAILED: "output.size == size"  [c:\users\leonard\documents\htm.core\src\htm\encoders\randomdistributedscalarencoder.cpp line 94]
----------------------------------------------------------------- generated xml file: C:\Users\Leonard\Documents\htm.core\junit-test-results.xml -----------------------------------------------------------------
======================================================================================= 1 failed, 8 passed in 0.42 seconds =======================================================================================

The loaded encoder does not seems to be able to encoder values anymore.
I will investigate this next Monday.

[breznak]

I have written this following piece of code

the test looks good, please add it to this PR.

[leonardtschora]

Hi, while trying to push comited changes on "serialization_rdse', the permission is denied.

[breznak]

while trying to push comited changes on "serialization_rdse', the permission is denied.

yes, you don't have push access to this repo. The work flow should be:

• you fork this repo to your account
• develop the changes (you are here)
• push the branch to your repo
• open a PR (your feature branch -> this repo)
  • in this case, do not open against master here, but against serialization_rdse branch
• I'll then merge the changes.

The problem is because I've started a branch from this repo, instead of a PR from your repo which would normally occur.

[breznak]

Back to this branch.
@Leonardbcm are the tests passing for you? @brev had a similar problem with failing bindings on pickle in #603

[breznak]

With these changes, this looks good to me 👍

[breznak]

Ping @Leonardbcm , please test the code

[dkeeney]

After spending most of 2 days struggling with pickle on SP I can give you what I found out as it relates to your class.

[dkeeney]

Also add a test for saveToFile() and loadFromFile()
See end of spatial_pooler_test.py in branch sp_save.

[breznak]

Thanks @dkeeney , this is working with your fixes from #644

[breznak]

@Leonardbcm @dkeeney can you ACK the code, please? There's a remaining test, but I don't think I'll have time to write it in near future, so I'd like to merge the fixes as is, which makes the requested serialization for the encoders work.

[dkeeney]

The missing tests can be added later.
I am working on a generic Encoder Region which will need to make some changes to this encoder so I can add the tests then.

[breznak]

Thank you!

I am working on a generic Encoder Region

this will be interesting!