Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed pickling on SP #644

Merged
merged 5 commits into from
Aug 21, 2019
Merged

Fixed pickling on SP #644

merged 5 commits into from
Aug 21, 2019

Conversation

dkeeney
Copy link

@dkeeney dkeeney commented Aug 20, 2019

This PR fixes the pickling on SP and adds saveToFile( filename ) and loadFromFile(filename).
Response to Issue #641

@dkeeney
Copy link
Author

dkeeney commented Aug 20, 2019

Now that I have something that works for pickling, I need to apply this to most of the other objects in the bindings.

@breznak
Copy link
Member

breznak commented Aug 20, 2019

breznak
breznak previously approved these changes Aug 20, 2019
Copy link
Member

@breznak breznak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, thanks for demystifying the pybind11/pickle issue!

@dkeeney
Copy link
Author

dkeeney commented Aug 20, 2019

Travis is still not happy

I typed in the name of the function incorrectly :-)

@@ -38,15 +38,51 @@ def testCompute(self):
self.assertTrue( active.getSum() > 0 )


@pytest.mark.skipif(sys.version_info < (3, 6), reason="Fails for python2 with segmentation fault")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't this still valid? (we don't test on py2, but someone still may run the repo on python2)

breznak
breznak previously approved these changes Aug 20, 2019
Copy link
Member

@breznak breznak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, it's great to have a clean pickle serialization for python working!

@dkeeney
Copy link
Author

dkeeney commented Aug 20, 2019

@breznak when you have some time can you give this a review again.

I want to point out that in bindings\py\tests\algorithms\temporal_memory_test.py, line 58
I marked this py unit test as skipped. It is giving me an error that comes from Cereal. It is an error I have see before with JSON output and it may indicate some bad Cereal serialization code someplace. I think this should be looked at with a different PR. I will added an Issue for it. Issue #645

Funny how when we add more tests we seem to find more bugs.

Copy link
Member

@breznak breznak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to have py serialization for the big classes in order! 👍 And to have the new tests to verify.
Thanks a lot @dkeeney for the investigative work!!



@pytest.mark.skip(reason="Fails with rapidjson internal assertion -- indicates a bad serialization")
def testNupicTemporalMemorySavingToString(self):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, this is a new test, that fails - we think exposing an internal JSON/pybind bug. It's fine to skip in that case.
Can you file an issue with pybind11? Is there a newer release we could try?

@breznak breznak merged commit 99b76b4 into master Aug 21, 2019
@breznak breznak deleted the sp_save branch August 21, 2019 07:55
@dkeeney
Copy link
Author

dkeeney commented Aug 21, 2019

Can you file an issue with pybind11?

Let's be sure our code is good first. I know that JSON is more sensitive to errors than the BINARY mode is. This will take some time to investigate.

Is there a newer release we could try?

I checked. They are still on 1.2.2 (Feb 2017) Nothing has changed in the master branch in 3 years. The error in JSON comes from rapidjson which is the third party parser. http://rapidjson.org/
Its latest release is Aug 2016 so nothing new. But there are some more recent changes in their master.

@breznak
Copy link
Member

breznak commented Aug 21, 2019

Nothing has changed in the master branch in 3 years.

hope we didn't switch to a dead project/ I'm happy with a mature, stable codebases, but nothing in 3yrs sounds suspicious

@dkeeney
Copy link
Author

dkeeney commented Aug 21, 2019

I agree. The buggy part is the JSON encoder and until rapidjson does a new release Cereal cannot either. (Note that this is Cereal and not pybind11 we are talking about)

@dkeeney
Copy link
Author

dkeeney commented Aug 21, 2019

I am still trying to understand how the build process works...
Can you tell me what GitHub nethook triggers we are using and perhaps what plugins we have? I assume we are using nethooks to kick off the three CI's. I don't want to change anything, just need to know what the parts are.

@breznak
Copy link
Member

breznak commented Aug 22, 2019

Can you tell me what GitHub nethook triggers

I'm not sure what you mean, the CI are installed as "GH apps" and we have webhooks (all events allowed) which the CI react to: commit push, PR opened, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants