[Question] Surface gripper can’t close on cuda device - advice?

[jnzhnd]

Hi all,

On my quest to use a surface gripper in a parallellized simulation, I've stumbled upon a weird issue: The gripper fails to close when using device="cuda:0" and closes when using device="cpu".

I'm using Isaac Sim 2022.2.0 and the latest version of Orbit to spawn a bunch of UR10s with surface grippers attached to pick up objects, for which I've modified the play_ik_control example that comes with Orbit. Simplified code is attached below, and the full standalone script is attached to this post.

Problem Description:

When I run with

SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cuda:0")

the gripper fails to close:

[omni.isaac.manipulators.grippers.surface_gripper] gripper didn't close successfully

This error seems to come from inside omni.isaac.surface_gripper._surface_gripper, as the close()-function I have access to in surface_gripper.py executes the way It's supposed to.

When I run with

SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cpu")

everything works. The gripper closes, and the object is picked up.

Code:

My (very abbreviated) code is as follows:

from omni.isaac.kit import SimulationApp
simulation_app = SimulationApp(config)

import torch
import time

import omni.isaac.orbit.utils.kit as kit_utils
from omni.isaac.orbit.robots.config.custom_universal_robots import UR10_CFG
from omni.isaac.orbit.robots.single_arm import SingleArmManipulator
from omni.isaac.manipulators.grippers.surface_gripper import SurfaceGripper

sim = SimulationContext(physics_dt=0.001, rendering_dt=0.01, backend="torch", device="cuda:0")

# Scene setup
kit_utils.create_ground_plane("/World/defaultGroundPlane", z_position=0.0)

robot_cfg = UR10_CFG
robot_cfg.data_info.enable_jacobian = True
robot_cfg.rigid_props.disable_gravity = True

robot = SingleArmManipulator(cfg=robot_cfg)
robot.spawn("/World/envs/env_0/Robot", translation=(0.0, 0.0, 0.0))

model_usd_path = "/home/*/Documents/Omniverse/Content/Models/My_USD/my_model.usd"
prim_utils.create_prim("/World/envs/env_0/Model_0", usd_path=model_usd_path, position=(0.485, 0.0, 0.184/2), orientation=(0.707, 0, 0.707, 0))

gripper = SurfaceGripper(
        end_effector_prim_path=f"/World/envs/env_0/Robot/ee_link", translate=0.1611, direction="x"
    )

robot.initialize("/World/envs/env_0/Robot")
gripper.initialize()

count = 0

# Simulate physics
while simulation_app.is_running():
    if count == 300:
        gripper.close()
    if count % 500 == 0:
        count = 0
        gripper.open()
    count += 1

Full script:

play_suction.py.zip

Btw, please ignore the very hacky way I've modified the Orbit IK example to pick up an object and follow a trajectory - this was only supposed to be a quick demo.

Anyway, any ideas what is going wrong? What are the differences between running surface_gripper on a gpu vs a cpu? Is it the thing I am trying to pick?

If I can provide more Information/code, I'm very happy to do so.

[jnzhnd]

After some more investigating, it looks like the ee_link of the robots (and its colliders?) are not moving with the visuals of the arm (or rather, the position of the ee_link prim is not updated during simulation), as can be seen in the screenshots below. I'm pretty sure that is the reason for my issue - but how come this only happens on the gpu?

Edit: Or maybe not entirely - I guess the visualizations of the colliders are not updated because of Fabric, see (https://forums.developer.nvidia.com/t/world-position-not-updating-during-simulation-run/245372). However, if I do the following:

ee_0_prim = RigidPrimView(prim_paths_expr="/World/envs/env_0/Robot/ee_link")
print(ee_0_prim.get_world_poses())

the pose is also not updating only when using the gpu.

Edit 2: Never mind - i was missing ee_0_prim.initialize() in betwen those two lines. Still, could it be that the surface_gripper does not update its own position because it is using the USD API?

[Mayankm96]

Yes, with GPU simulation, you don't see the debug visualizations getting updated in 2022.2.1. This will be fixed in future releases.

Additionally, with GPU simulation, we enable flatcache / fabric which bypasses USD completely. So no USD attributes are updated. You can explicitly disable flatcache using the SimulationContext and see if that fixes the issue.

There is currently no implementation of surface grippers on GPU so this may slow down things for RL-like training.

[jnzhnd]

Hey, thanks for your reply! Unfortunately, disabling flatcache via sim.get_physics_context().enable_flatcache(False) does not fix the issue. Is there anything else you could think of that I could try? The surface_gripper class in isaac sim is closed-source, so I can't see how it works under the hood.

I saw that on one of the pictures in your documentation (this one), you have what looks like a surface gripper on a UR10 in the very top left - if that was a functional example, how did you manage to implement it?

I guess worst case, I'd have to try making some kind of vacuum-like gripper implementation myself.

[Mayankm96]

We used the surface gripper with the CPU pipeline for that environment. For GPU simulation, as far as I know, a solution doesn't exist.

Not sure what a custom implementation for this would look like. I am happy to discuss it with you if you already have some ideas.

[jnzhnd]

Sure! But be aware that I'm by no means an expert (neither in orbit or in isaac sim for the matter).

Basically, we could raycast from the EE into one direction (using get_physx_scene_query_interface().raycast_closest() ?). If we hit something within a specified distance, we create a fixed joint between the EE and the object that we hit. I think this is also very similar to how the SurfaceGripper class works in isaac sim, but it probably uses some of the USD APIs behind the scenes and therefore does not work when running with fabric/flatcache.

What do you think?

[ImSubin98]

Hello! I apologize for the very simple question, but I've been struggling with this issue for two weeks. I'd like to import a UR10 into Isaac Sim and apply an artificial potential field (pathfinding algorithm) to see the results.

I'm having trouble with the part where I import the UR10 and apply the code. Could you please help me with the process? Thank you in advance.

[TerenceXue-tech]

from omni.isaac.manipulators.grippers.surface_gripper import SurfaceGripper
I’d like to know if this class is available in the latest IsaacLab version, as I couldn’t find it in my search.