What are you using ROS# for?

[MartinBischoff]

We are very curious what you use ROS# for!

Let us use this issue to present our projects, whether finished, ongoing, or first idea.

I start! Here is our first public project:

A Tutlebot 2 is teleoperated via touchpad controllers HTC Vive.
A 3d point cloud of the camera image is illustrated in VR, as well as the mapping data and the moving turtlebot. The wheels of the turtlebot are animated via JointState messages. Users can teleport to different positions in the scene.
Further, users can switch between teleoperation and simulation mode in real time. In the simulation mode, RosBridgeClient disconnects and Users control the Urdf Simulation model by applying a torque at the wheels. It feels quite similar with the VR glasses on. The only difference is that the camera image is missing and the map does not update anymore. 😏

[lucascoelhof]

Hi all,

I'm a master's student at UFMG, in Brazil. My master's thesis project is about robot swarm teleoperation, and since we had a Oculus Rift laying around we thought it was a good idea to use it. However, as you may know, Oculus Rift does not have a fully functional Linux driver and they don't have plans to officially support it, so I made numerous attempts on somehow doing something in ROS and being able to visualize it on Oculus Rift. Maybe I lost 6 months trying different stuff.
In one of my attempts, I started using Unity, and I was looking for some C# libraries for ROS and I stumbled upon ROS#. And it solved all my problems!
You guys had a URDF importer (something I haven't even thought I needed but it was extremely useful). And had also implemented rosbridge with some message types that I needed. In a few HOURS I could implement what I was trying to accomplish for months. 😄

Since then, I'm using ROS# to transfer data between Unity and ROS. From Unity I need the Oculus Poses and buttons states. And ROS sends back the positions of my dear little robots and some other visualization stuff.

[jeremyfix]

Hi,

I like to test control and visual processing algorithms on robotic platforms. While we can certainly test them on real platforms, it is clearly easier to test them on simulated robots.

To achieve this purpose, I used to work with V-REP interfaced with ROS. However, it turns that a particular project, I needed to get some realistic shading, light reflections, etc... Why my limited knowledge, I think that this is not easily achieved with VREP. I then turned to game engines like Unreal Engine and more recently Unity.

Unfortunately, I initially forgot quickly about Unity, wrongly thinking it is was not possible to use without a license. I therefore invested some time trying to achieve my purpose with Unreal Engine, especially trying to use Microsoft AirSim (yes I needed a drone as well) ; But, the Unreal Engine really requires a beast, not a computer :) And I even do not know how easily it would have been to interface with the ROS world, but definitely the time it takes to compile the engine makes the progresses at a very slow pace. More recently, colleagues suggested to use Unity and it turns out that 1) it is very quick to set up and to run even on a moderately decent hardware and 2) combined with ROS-sharp it seems to perfectly fit my needs.

So, for visually controlled robots, I can benefit from the Unity 3D capabilities for the rendering, and still develop/use algorithms on the ROS side; And this is also more funny to play with nice looking simulations :)

[Jasonthefirst]

I am a student at the KIT Karlsruhe trying to control a Kuka Robot with the Hololens. Right now we try to use the ros-sharp library but UWP is resilient. So far we only used ros-sharp urdf importer to get our robot into unity because it works better than the old urdf importer. After the import we have to delete ros-sharp because we can't compile the project.
The project goal is to control a robot virtually and only if the robot is able to do a movement without colliding in the virtual world the robot should move in the real world.

[MartinBischoff]

Hi @Jasonthefirst , thanks for sharing your project with us! We also had a small project with the Hololens this year. Unfortunately without ROS#, though 😉

Let's discuss platform requirements in issue #21. Or, alternatively, please open a new issue if more appropriate.

I find it fascinating for what different types af applications, robots and VR/AR equipment you guys use ROS#. Looking forward to hear more stories!

[MartinBischoff]

We are very proud and happy that parts of ROS# already found their way into Process Simulate, a Siemens Simulation Software for production lines.

My colleagues just sent me the link to this youtube video illustrating the import of a URDF model.

[jeremyfix]

Hi @MartinBischoff , here is a follow up with a particular example of Unity3D + RosSharp where we want to simulate underground environments explored with drones.

You can have a look to it at : https://youtu.be/XajgNfNJ1VI

Not the full code is yet provided. Only the code of the drone is available for the moment at : https://github.com/jeremyfix/ros-and-unity/tree/master/DronePrefab

[MartinBischoff]

Very impressive @jeremyfix ! Thank you for sharing this info here.
I am happy to see what you can do with ROS#. Are you using this setup to test any (autonomous) functions of the drone? Is the similarity of real and virtual camera data of importance for you here?

All: Please do not hesitate and tell us about your applications here. It needn't be as elaborated as the applications above. We are also happy to read about any planned or ongoing tryouts...

[jeremyfix]

@MartinBischoff I indeed use this setup for testing autonomous navigation in underground . This navigation is based on optic flow. For easily testing this ability , we needed to get a simulated environment with 3D shapes, textures, lightning so be as fair as possible with what we may encounter in real underground environements.

[NishanthJKumar]

Hi all,

I'm an undergraduate from Brown University in Providence, and my lab currently uses ROS# for a variety or projects including VR and AR visualization and teleoperation of a Rethink Robotics Baxter, and a Kinova Robotics Movo.

@MartinBischoff, I'm currently attempting to implement something similar to your turtlebot teleoperation project, but on a different robot. Is the Unity Project for this online somewhere, or is there a tutorial on how to integrate a depth image with the rgb camera? That'd be very helpful!

Our lab will post links to videos and images of the project soon! Excited to share more with all of you.

[MartinBischoff]

Hi @NishanthJKumar ! Thanks for sharing this info about your project here!
I am looking forward to see the videos and images!

We did not publish our code for the turtlebot application open source. It would be too much effort to to make the code readable, document, fix bugs and administrated it properly. I hope you understand.

For the 3D point cloud we proceeded as follows:

• We used opencv library to uncompress the CompressedImage data.
• We used a script like this to get the points in 3D from the two 2D images.
• We then put this data into a mesh with a simple shader that illustrates point data only.
• By only updating parts of the mesh (e.g. 20%) in each frame you can improve the performance of your application.

[MahirGulzar]

Hello everyone,

I am a Masters student in Computer Science from University of Tartu Estonia. I am currently doing an Internship in an Estonian Robotics Organization. My task here is to integrate ROS with Unity to display statistics and other operations on a Hololens device.

So far it was going good with ROS# Library but I had to abandon it. The problem arrises with sockets. The current version of ROS# doesn't work with UWP. Infact System.net.sockets is not working with UWP. After digging deep I found out that UDW supports Windows.Networking for sockets but Unity cannot recognize this. So right now trying to find a workaround to get pass this issue.

Btw I followed your thread for ROS# upgrade in which you use IProtocol to pass socket type on connection but still the issue persisted.

[kwunh]

Hello,

I'm currently working on my project where I have to implement a way to teleoperate a Baxter Robot machine using a HTC Vive on a Single Linux Ubuntu machine. Honestly a really daunting task as it's my first year doing coding but I was mainly just using java so C# is thankfully not too far-off. Anyways, I have got Unity working on my machine & learned basics of ROS, now I want to start communicating between the two.

This seems to be exactly the tool I need, so thank you so much for providing this! But I just want to make sure this is actually possible to do so on a single Linux machine as the documentation only mentions a Windows build.

I've seen posts of both people successfully and unsuccessfully using ROS# on a Linux machine... I can't get an opportunity to test this out until Monday but just want to know what other people's experiences with ROS# on a Linux machine is like. Using a Linux Ubuntu 16.04 btw. Thanks for any input! :)