Plasma manager blocks on an object transfer when the object is evicted.

[stephanie-wang]

Plasma managers request object transfers from each other. These requests get queued at each Plasma manager, which sends the object data to the receiver in order. It's possible for the requested object to get evicted while the request is waiting in the queue. In this case, the sending Plasma manager will block (possibly forever) trying to get the object.

This can be seen in the multinode reconstruction tests: logs.

We should fix this by having the Plasma manager skip requests for which the object is no longer available locally. The receiving Plasma manager should be notified that the transfer is canceled, if some of the object data had already been sent.

[robertnishihara]

The blocking referred to is in this do/while loop here https://github.com/ray-project/ray/blob/master/src/plasma/plasma_manager.c#L751.

If plasma_get results in obj_buffer.data_size == -1, then the object is not present. I believe we can simply clean stuff up and return. We shouldn't have to send any message to the receiving plasma manager. The receiving plasma manager should time out after a little while.

None of the object data will have been sent already, so we don't have to handle that.

[robertnishihara]

I haven't tested it, but this may be addressed in #347.

[robertnishihara]

The previous PR was closed, but this should be addressed by #373.