Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send task error instead of raw exception for signal #4150

Merged
merged 2 commits into from
Mar 2, 2019

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Feb 24, 2019

What do these changes do?

Exceptions are not always serializable, so we cannot send them safely. Send the RayTaskError instead.

Related issue number

Fixes #4149

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12273/
Test FAILed.

Copy link
Contributor

@istoica istoica left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me.

@pcmoritz
Copy link
Contributor

This is not currently passing tests, see https://travis-ci.com/ray-project/ray/jobs/180115893

@ericl
Copy link
Contributor Author

ericl commented Mar 1, 2019

The tests seem to only fail in python 2.

@robertnishihara
Copy link
Collaborator

robertnishihara commented Mar 2, 2019

I thought it was a cloudpickle bug, but it turns out to be a pickle bug.

In Python 2, this fails

import pickle

class A(Exception):
    def __init__(self, x):
        pass

a = A(1)

serialized = pickle.dumps(a)
pickle.loads(serialized)

It raises

Traceback (most recent call last):
  File "bug.py", line 10, in <module>
    pickle.loads(serialized)
  File "/home/ubuntu/miniconda/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/home/ubuntu/miniconda/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/home/ubuntu/miniconda/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
TypeError: __init__() takes exactly 2 arguments (1 given)

https://stackoverflow.com/questions/41808912/cannot-unpickle-exception-subclass

@ericl
Copy link
Contributor Author

ericl commented Mar 2, 2019

Oh weird. I guess we could send function_name, backtrace as a tuple and construct RayTaskError on the other side then?

@robertnishihara
Copy link
Collaborator

I made it a string instead. What do you think about that?

@ericl
Copy link
Contributor Author

ericl commented Mar 2, 2019

Sounds good to me. We can iterate on this later.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12450/
Test FAILed.

@robertnishihara robertnishihara merged commit 9950f63 into ray-project:master Mar 2, 2019
@robertnishihara robertnishihara deleted the fix-signal branch March 2, 2019 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants