[xray] Log warnings for asio handlers that take too long

[stephanie-wang]

What do these changes do?

This adds warnings when an event handler takes too long to process a message from a ClientConnection. Ideally, we would also be able to do this in the future for all handlers on the event loop (e.g., timer expiration handlers).

This also changes the timer used for heartbeats to one that uses the std::chrono::steady_clock, and logs a warning if the last heartbeat was more than 500ms ago.

[atumanov]

What are the advantages of killing raylet when it missed num_heartbeats? I understand that the monitor will mark it as dead, but I don't understand why it's better to kill raylet in this case rather than letting it make progress. To make it consistent with the monitor action? Could we have the monitor mark the node as live again when it starts receiving the heartbeats from that node again?

Switching to the monotonic clock (as we've done with legacy Ray) makes sense to me.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7326/
Test PASSed.

[stephanie-wang]

@atumanov, yeah, the fatal check probably isn't that helpful. I changed it to log a warning if the last heartbeat was too long ago.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/7330/
Test PASSed.