Trajectory Execution Failed Sliently due to "Sending data through socket failed."

[shuobh]

Summary

Introduction to the issue

When executing multiple trajectories sequentially using remote control, some trajectory may fail silently due to "Sending data through socket failed.". For example, when executing A->B->C. If B failed, then C will directly be executed after A which out any error throwing.

Versions

ROS Driver version: ROS Noetic
Affected Robot Software Version(s): 5.11.1
Affected Robot Hardware Version(s): UR3e

Impact

If a trajectory is skipped, then it might cause the robot to blindly move from the end of the previous trajectory to the beginning of the next trajectory which my cause a big hazard.

Project status at point of discovered

When did you first observe the issue?
This happens by chance with relatively low probability.

Workaround Suggestion

By searching in this repo, the cause to this failure of "Sending data through socket failed." might be due to that we are not using a real time kernal. We will be testing it following https://github.com/UniversalRobots/Universal_Robots_ROS_Driver/blob/master/ur_robot_driver/doc/real_time.md. However, if this case happens, the driver should either retry, or throw an error instead of fail silently.

[shuobh]

Ran into another trajectory execution issue:

Jul 14 14:24:44 nuc-19-robot-14 bluehill-start[2271640]: [ WARN] [1657833884.788138173]: Dropping all 1 trajectory point(s), as they occur before the current time.
Jul 14 14:24:44 nuc-19-robot-14 bluehill-start[2271640]: Last point is 1.555738s in the past.

[fmauch]

I cannot quite follow what you are trying to accomplish. Could you please elaborate what (and with which timing) you are sending to the driver's interfaces?

Sending data through socket failed should not happen in a normal usage, so this would be something to further inspect.

And the comment from your second post comes from the joint trajectory controller directly and means that the trajectory sent to the controller is illegal as (as the output states) the last point in the trajectory is in the past. Timestamps of waypoints must be increasing in time.

[shuobh]

We've tried to install a realtime kernel but it failed to compile. However, by upgrading to low-latency kernel, this problem seems to be resolved as well. @fmauch Do you have any insight on the difference between low latency kernel and realtime kernel and how it might impact the driver?

[github-actions]

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

[fmauch]

We are currently looking deeper into this and hope to provide some more insight soon.

[fmauch]

The deeper look as mentioned above is now taking shape in #615.

[github-actions]

This issue has not been updated for a long time. If no further updates are added, this will be closed automatically. Comment on the issue to prevent automatic closing.

[github-actions]

This issue has been closed due to inactivity. Feel free to comment or reopen if this is still relevant.