[core] handle unserializable user exception

[hongchaodeng]

If I run the following script, core worker would fail to serialize the RayError and crash

(credit to @alexeykudinkin )

from time import sleep

import ray

ray.init()

@ray.remote(num_cpus=1)
class Callee:

    def bar(self):
        from tenacity import retry, stop_after_attempt

        @retry(stop=stop_after_attempt(1))
        def failing_method():
            raise ValueError("failed")

        failing_method()


@ray.remote(num_cpus=1)
class Caller:

    def __init__(self, h):
        self.callee = h

    def foo(self):
        ref = self.callee.bar.remote()
        ray.wait([ref])


callee = Callee.remote()
caller = Caller.remote(callee)

ray.wait([caller.foo.remote()])

sleep(1800)

This PR will fix this.

[jjyao]

Could you add some tests?

[jjyao]

The original cause of RayTaskError ({self.cause.__class__}) is non-serializable: {e}. Overwriting the cause to be RayError ?

[hongchaodeng]

@jjyao Is there any existing tests on testing exceptions?

[jjyao]

Code lgtm

[jjyao]

nit: this can be split into multiple lines to avoid # noqa

[hongchaodeng]

done

[hongchaodeng]

@jjyao Test and doc added. PTAL!

[aslonnie]

not sure why it ends with colon :.

and also maybe change to active tense, like:

If Ray cannot serialize the user's exception, it will convert the exception to a ``RayError``

[hongchaodeng]

Thanks! Missed some sample code before. Fixed

[angelinalg]

Made some attempts at some style corrections. Please fix if not correct.

[angelinalg]

Suggested change

	# The original cause of RayTaskError (<class '__main__.UnserializableException'>) is not serializable: cannot pickle '_thread.lock' object. Overwriting the cause to RayError.
	# The original cause of RayTaskError (<class '__main__.UnserializableException'>) isn't serializable: can't pickle '_thread.lock' object. Overwriting the cause to a RayError.

[hongchaodeng]

Yeah this is not my writings.
It's output from the program runs.
So let's not change it.

[hongchaodeng]

Actually I just see you suggest changes on the code as well. I will update that and rerun the program to reflect it.

[hongchaodeng]

@jjyao Ready for review. PTAL

[jjyao]

Move it down to after line 49. Currently its in the middle of subclass section.

[jjyao]

nit: we can just set self.args in the end of the constructor so we don't need to set it twice. basically move line 122 down to the end.

[jjyao]

One last comment

[jjyao]

Could you move the comments (line 119-121) to here as well since they are comments for self.args.

[hongchaodeng]

fixed