Multithreading refactor for ObjectManager. #1911
Conversation
There was a problem hiding this comment.
LGTM, I went through it with Melih, @stephanie-wang: Can you also have a look?
|
Test PASSed. |
|
Test PASSed. |
| RAY_CHECK_OK(TransferCompleted(TransferQueue::TransferType::SEND)); | ||
| return chunk_status.second; | ||
| } | ||
| // If status is not okay, and this is the first chunk to be gotten for this object, |
There was a problem hiding this comment.
The comment about "this is the first chunk to be gotten" isn't correct anymore, right? Can we remove that?
There was a problem hiding this comment.
The comment seems to be correct to me. Can you explain why it's no longer correct?
There was a problem hiding this comment.
Shouldn't it be "If status is not okay, then return immediately..."? I don't think it matters which chunk of the object it is.
|
Test FAILed. |
|
Test FAILed. |
|
Test PASSed. |
* master: Handle interrupts correctly for ASIO synchronous reads and writes. (ray-project#1929) [DataFrame] Adding read methods and tests (ray-project#1712) Allow task_table_update to fail when tasks are finished. (ray-project#1927) [rllib] Contribute DDPG to RLlib (ray-project#1877) [xray] Workers blocked in a `ray.get` release their resources (ray-project#1920) Raylet task dispatch and throttling worker startup (ray-project#1912) [DataFrame] Eval fix (ray-project#1903) [tune] Polishing docs (ray-project#1846) [tune] [rllib] Automatically determine RLlib resources and add queueing mechanism for autoscaling (ray-project#1848) Preemptively push local arguments for actor tasks (ray-project#1901) [tune] Allow fetching pinned objects from trainable functions (ray-project#1895) Multithreading refactor for ObjectManager. (ray-project#1911) Add slice functionality (ray-project#1832) [DataFrame] Pass read_csv kwargs to _infer_column (ray-project#1894) Addresses missed comments from multichunk object transfer PR. (ray-project#1908) Allow numpy arrays to be passed by value into tasks (and inlined in the task spec). (ray-project#1816) [xray] Lineage cache requests notifications from the GCS about remote tasks (ray-project#1834) Fix UI issue for non-json-serializable task arguments. (ray-project#1892) Remove unnecessary calls to .hex() for object IDs. (ray-project#1910) Allow multiple raylets to be started on a single machine. (ray-project#1904) # Conflicts: # python/ray/rllib/__init__.py # python/ray/rllib/dqn/dqn.py
This PR refactors the ObjectManager to send/receive messages and create connections on the main thread. It also removes the ObjectManager queuing mechanism and instead relies on asio's internal queuing.