Issue with Server-Client Setup

[abadithela]

Hi,

We're trying to get the robot controller setup but are running into a few issues (the second one is the bottleneck)

1. First, when we try to command "go_sleep" after setting up the Dockerfile, docker compose exec robonet bash -lic "go_sleep", we get the error:

WARN[0000] /home/iromlab/Desktop/bridge_data_robot/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion 
Traceback (most recent call last):
  File "/usr/local/bin/go_sleep", line 4, in <module>
    from widowx_envs.widowx_env import StateReachingWidowX
  File "/home/robonet/interbotix_ws/src/widowx_envs/widowx_envs/widowx_env.py", line 8, in <module>
    from widowx_envs.base.robot_base_env import RobotBaseEnv
  File "/home/robonet/interbotix_ws/src/widowx_envs/widowx_envs/base/robot_base_env.py", line 3, in <module>
    from widowx_envs.control_loops import Environment_Exception
  File "/home/robonet/interbotix_ws/src/widowx_envs/widowx_envs/control_loops.py", line 8, in <module>
    from widowx_envs.utils.image_utils import npy_to_gif, npy_to_mp4, resize_store
  File "/home/robonet/interbotix_ws/src/widowx_envs/widowx_envs/utils/image_utils.py", line 2, in <module>
    import moviepy.editor as mpy
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/__init__.py", line 4, in <module>
    from moviepy.audio import fx as afx
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/audio/fx/__init__.py", line 5, in <module>
    from moviepy.audio.fx.AudioDelay import AudioDelay
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/audio/fx/AudioDelay.py", line 5, in <module>
    from moviepy.audio.AudioClip import CompositeAudioClip
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/audio/AudioClip.py", line 13, in <module>
    from moviepy.audio.io.ffmpeg_audiowriter import ffmpeg_audiowrite
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/audio/io/ffmpeg_audiowriter.py", line 7, in <module>
    from moviepy.config import FFMPEG_BINARY
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/config.py", line 7, in <module>
    from moviepy.tools import cross_platform_popen_params
  File "/home/robonet/myenv/lib/python3.8/site-packages/moviepy/tools.py", line 251, in <module>
    ) -> tuple[int, int]:
TypeError: 'type' object is not subscriptable

To get around this, we changed the version of moviepy by running pip install moviepy==1.0.0. There were a few other dependency issues with scipy and scikit-image that we could resolve. We're trying to make these fixes in the Dockerfile itself instead of having to fix this each time we start the container. Is there an updated Dockerfile that addresses this?

2. Next, we try to setup the server-client architecture, with the client and the server running on the same machine. The server starts up OK but there is an issue with the client:

Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.
Waiting for robot to be ready...
WARNING:root:Failed to send message: Resource temporarily unavailable
DEBUG:root:WARNING: No res from server. reset socket.

The instructions mention that the robot should be executing some pre-determined movements but we do not see this. Your help in resolving this is greatly appreciated!

[abadithela]

For context, here is the terminal output when I start up robonet. There seems to be an issue with libusb:

robonet_iromlab  | process[wrist/wrist_stream-1]: started with pid [228]
robonet_iromlab  | [ INFO] [1737663219.092151749]: Initializing nodelet with 20 worker threads.
robonet_iromlab  | process[blue/blue_stream-1]: started with pid [230]
robonet_iromlab  | process[yellow/yellow_stream-1]: started with pid [257]
robonet_iromlab  | process[D435/D435_stream-1]: started with pid [258]
robonet_iromlab  | [ INFO] [1737663219.149209293]: Initializing nodelet with 20 worker threads.
robonet_iromlab  | [ INFO] [1737663219.149661257]: Initializing nodelet with 20 worker threads.
robonet_iromlab  | [ INFO] [1737663219.158337789]: Initializing nodelet with 20 worker threads.
robonet_iromlab  | [ INFO] [1737663219.216072405]: Camera name: wrist
robonet_iromlab  | [ INFO] [1737663219.216134543]: Provided camera_info_url: ''
robonet_iromlab  | [ INFO] [1737663219.216166266]: Publishing with frame_id: world
robonet_iromlab  | [ INFO] [1737663219.216204924]: Setting camera FPS to: 30
robonet_iromlab  | [ INFO] [1737663219.216235803]: Throttling to fps: 30
robonet_iromlab  | [ INFO] [1737663219.216271869]: Setting buffer size for capturing frames to: 1
robonet_iromlab  | [ INFO] [1737663219.216314844]: Flip horizontal image is: false
robonet_iromlab  | [ INFO] [1737663219.216352207]: Flip vertical image is: false
robonet_iromlab  | [ INFO] [1737663219.216415119]: Video start frame is: 0
robonet_iromlab  | [ INFO] [1737663219.216465220]: Video stop frame is: -1
robonet_iromlab  | [ INFO] [1737663219.216509834]: Forced image width is: 640
robonet_iromlab  | [ INFO] [1737663219.216539086]: Forced image height is: 480
robonet_iromlab  | [ INFO] [1737663219.286987727]: Camera name: yellow
robonet_iromlab  | [ INFO] [1737663219.287049294]: Provided camera_info_url: ''
robonet_iromlab  | [ INFO] [1737663219.287096153]: Publishing with frame_id: world
robonet_iromlab  | [ INFO] [1737663219.287133452]: Setting camera FPS to: 30
robonet_iromlab  | [ INFO] [1737663219.287164270]: Throttling to fps: 30
robonet_iromlab  | [ INFO] [1737663219.287197188]: Setting buffer size for capturing frames to: 1
robonet_iromlab  | [ INFO] [1737663219.287226209]: Flip horizontal image is: false
robonet_iromlab  | [ INFO] [1737663219.287256779]: Flip vertical image is: false
robonet_iromlab  | [ INFO] [1737663219.287288714]: Video start frame is: 0
robonet_iromlab  | [ INFO] [1737663219.287323971]: Video stop frame is: -1
robonet_iromlab  | [ INFO] [1737663219.287381641]: Forced image width is: 640
robonet_iromlab  | [ INFO] [1737663219.287416432]: Forced image height is: 480
robonet_iromlab  | [ INFO] [1737663219.340988457]: Camera name: blue
robonet_iromlab  | [ INFO] [1737663219.341041435]: Provided camera_info_url: ''
robonet_iromlab  | [ INFO] [1737663219.341067759]: Publishing with frame_id: world
robonet_iromlab  | [ INFO] [1737663219.341102843]: Setting camera FPS to: 30
robonet_iromlab  | [ INFO] [1737663219.341134649]: Throttling to fps: 30
robonet_iromlab  | [ INFO] [1737663219.341175603]: Setting buffer size for capturing frames to: 1
robonet_iromlab  | [ INFO] [1737663219.341207142]: Flip horizontal image is: false
robonet_iromlab  | [ INFO] [1737663219.341237952]: Flip vertical image is: false
robonet_iromlab  | [ INFO] [1737663219.341266480]: Video start frame is: 0
robonet_iromlab  | [ INFO] [1737663219.341295827]: Video stop frame is: -1
robonet_iromlab  | [ INFO] [1737663219.341319913]: Forced image width is: 640
robonet_iromlab  | [ INFO] [1737663219.341344577]: Forced image height is: 480
robonet_iromlab  | [ INFO] [1737663219.358155626]: Camera name: D435
robonet_iromlab  | [ INFO] [1737663219.358213641]: Provided camera_info_url: ''
robonet_iromlab  | [ INFO] [1737663219.358260914]: Publishing with frame_id: world
robonet_iromlab  | [ INFO] [1737663219.358306793]: Setting camera FPS to: 30
robonet_iromlab  | [ INFO] [1737663219.358345315]: Throttling to fps: 30
robonet_iromlab  | [ INFO] [1737663219.358393218]: Setting buffer size for capturing frames to: 1
robonet_iromlab  | [ INFO] [1737663219.358430015]: Flip horizontal image is: false
robonet_iromlab  | [ INFO] [1737663219.358471811]: Flip vertical image is: false
robonet_iromlab  | [ INFO] [1737663219.358511523]: Video start frame is: 0
robonet_iromlab  | [ INFO] [1737663219.358548143]: Video stop frame is: -1
robonet_iromlab  | [ INFO] [1737663219.358585743]: Forced image width is: 640
robonet_iromlab  | [ INFO] [1737663219.358625180]: Forced image height is: 480
robonet_iromlab  | [ INFO] [1737663219.670347567]: [xs_sdk] The operating mode for the 'arm' group was changed to position.
robonet_iromlab  | [ INFO] [1737663219.704306616]: [xs_sdk] The operating mode for the 'gripper' joint was changed to pwm.
robonet_iromlab  | [ INFO] [1737663219.924339744]: [xs_sdk] Interbotix 'xs_sdk' node is up!
robonet_iromlab  | [ INFO] [1737663224.629425634]:  
robonet_iromlab  | [ INFO] [1737663224.746445426]: Device with serial number 244222071650 was found.
robonet_iromlab  | 
robonet_iromlab  | [ INFO] [1737663224.746483548]: Device with physical ID 2-2-5 was found.
robonet_iromlab  | [ INFO] [1737663224.746507610]: Device with name Intel RealSense D435 was found.
robonet_iromlab  | [ INFO] [1737663224.746741219]: Device with port number 2-2 was found.
robonet_iromlab  | [ INFO] [1737663224.746759159]: Device USB type: 3.2
robonet_iromlab  | [ INFO] [1737663224.748809181]: getParameters...
robonet_iromlab  | [ INFO] [1737663224.794511053]: setupDevice...
robonet_iromlab  | [ INFO] [1737663224.794563706]: JSON file is not provided
robonet_iromlab  | [ INFO] [1737663224.794591915]: ROS Node Namespace: D435
robonet_iromlab  | [ INFO] [1737663224.794634285]: Device Name: Intel RealSense D435
robonet_iromlab  | [ INFO] [1737663224.794669505]: Device Serial No: 244222071650
robonet_iromlab  | [ INFO] [1737663224.794728183]: Device physical port: 2-2-5
robonet_iromlab  | [ INFO] [1737663224.794757001]: Device FW version: 05.16.00.01
robonet_iromlab  | [ INFO] [1737663224.794795444]: Device Product ID: 0x0B07
robonet_iromlab  | [ INFO] [1737663224.794825074]: Enable PointCloud: Off
robonet_iromlab  | [ INFO] [1737663224.794866985]: Align Depth: Off
robonet_iromlab  | [ INFO] [1737663224.794905195]: Sync Mode: Off
robonet_iromlab  | [ INFO] [1737663224.794986483]: Device Sensors: 
robonet_iromlab  | [ INFO] [1737663224.869374519]: Stereo Module was found.
robonet_iromlab  | [ INFO] [1737663224.889191371]: RGB Camera was found.
robonet_iromlab  | [ INFO] [1737663224.889276231]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
robonet_iromlab  | [ INFO] [1737663224.889321908]: num_filters: 0
robonet_iromlab  | [ INFO] [1737663224.889351392]: Setting Dynamic reconfig parameters.
robonet_iromlab  | hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
robonet_iromlab  | hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
robonet_iromlab  | hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
robonet_iromlab  | hwmon command 0x80( 5 0 0 0 ) failed (response -7= HW not ready)
robonet_iromlab  | [ INFO] [1737663227.968733680]: Done Setting Dynamic reconfig parameters.
robonet_iromlab  | [ INFO] [1737663227.969941680]: depth stream is enabled - width: 640, height: 480, fps: 30, Format: Z16
robonet_iromlab  | [ INFO] [1737663227.971266798]: color stream is enabled - width: 640, height: 480, fps: 30, Format: RGB8
robonet_iromlab  | [ INFO] [1737663227.971350728]: setupPublishers...
robonet_iromlab  | [ INFO] [1737663227.976412782]: Expected frequency for depth = 30.00000
robonet_iromlab  | [ INFO] [1737663228.025950141]: Expected frequency for color = 30.00000
robonet_iromlab  | [ INFO] [1737663228.057957811]: setupStreams...
robonet_iromlab  |  23/01 12:13:48,347 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  | [ INFO] [1737663228.398370542]: SELECTED BASE:Depth, 0
robonet_iromlab  | [ INFO] [1737663228.416542278]: RealSense Node Is Up!
robonet_iromlab  | [ WARN] [1737663228.467305374]: 
robonet_iromlab  |  23/01 12:13:48,468 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  |  23/01 12:13:48,518 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  |  23/01 12:13:48,569 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  |  23/01 12:13:48,690 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  |  23/01 12:13:48,740 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0
robonet_iromlab  |  23/01 12:13:48,891 WARNING [140021916563200] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0

Do you know what the following line means: (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Success, number: 0. For context, we are using a realsense D435. The "error: success" seems contradictory.

[abadithela]

For further context, we do not have yellow/blue cameras or a wrist camera. The usb_connector_chart.yml looks like this:

blue: ''
yellow: ''
wrist: ''
D435: '(usb-0000:00:14.0-2'

And the robonet terminal outputs the following messages:

robonet_iromlab  | [ WARN] [1737669604.357660628]: Camera calibration file /home/robonet/.ros/camera_info/blue.yaml not found.
robonet_iromlab  | [ INFO] [1737669604.358225957]: Opening VideoCapture with provider: /dev/video0
robonet_iromlab  | [ WARN:0] global ../modules/videoio/src/cap_gstreamer.cpp (1758) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Cannot identify device '/dev/video0'.
robonet_iromlab  | [ WARN:0] global ../modules/videoio/src/cap_gstreamer.cpp (888) open OpenCV | GStreamer warning: unable to start pipeline
robonet_iromlab  | [ WARN:0] global ../modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
robonet_iromlab  | [ WARN:0] global ../modules/videoio/src/cap_v4l.cpp (887) open VIDEOIO(V4L2:/dev/video0): can't open camera by index
robonet_iromlab  | [ INFO] [1737669604.652908678]: Video stream provider type detected: videodevice
robonet_iromlab  | [ INFO] [1737669604.652963024]: Backend can't provide camera FPS information
robonet_iromlab  | [ERROR] [1737669604.652992970]: Could not open the stream.

Interestingly, before the robonet container is started, the command v4l2-ctl --list-devices returns:

Intel(R) RealSense(TM) Depth Ca (usb-0000:00:14.0-2):
	/dev/video0
	/dev/video1
	/dev/video2
	/dev/video3
	/dev/video4
	/dev/video5
	/dev/media0
	/dev/media1

but when the container is up and running, it says Cannot open device /dev/video0, exiting.

[youliangtan]

Hi, if you are planning to try it on a widowx robot, you could try it on: https://github.com/rail-berkeley/manipulator_gym

It is used in some of our internal projects, and have cleaner API.

[abadithela]

OK, just for clarification, you are recommending us to use this new repo for robot control instead of this one (bridge_data_robot)? Do you think this new repo addresses the camera streaming issues over usb?

For the moment, we are not doing data collection, but we just want to use the WidowX controller. Since the issue is regarding the USB streaming and Docker interface, we were also thinking of trying to get the widowx controller ros package outside of the docker container. Would you recommend this?

[youliangtan]

@abadithela If you are not running the datacollection pipeline, the manipulator_gym is a better option. It has a simple gym API, abstract out the ros impl, and doesn't use complicated docker. Internally for camera video capture, it just does vanilla cv2.videocapture, that's where you can read the realsense img.

[abadithela]

Sounds good, thank you! We did find the manipulator_gym API easier to use. We had a few questions regarding the WidowX controller. In this repo, you implement your own custom WidowX controller, and in manipulator_gym, you use set_ee_pose_components. We had to increase the moving_time argument in set_ee_pose_components so that the end effector moves more smoothly. Was there a reason for implementing the controller differently here vs. in the manipulator_gym? Did you observe any important differences in behavior?

[youliangtan]

Afaik, the moving_time won't matter much since most of our widowx executions are using blocking control (synchronous). Feel free to open PR for potential improvements to manipulator_gym. Btw there are also gym wrappers for different usage.