SmolVLA Training on LeIsaac Datasets Does Not Produce Expected Results

[SilVerC77]

Hi,

My team is trying to use datasets collected with LeIsaac for imitation learning using SmolVLA, but we have not been able to achieve the expected results so far.

Has anyone successfully trained SmolVLA on a LeIsaac dataset for tasks such as grasping or pick-and-place?

If possible, could you share:

• Example datasets
• Training configurations
• Evaluation results
• Demo videos

Any references or successful examples would be greatly appreciated.

For reference, our team's workflow is roughly as follows:

1. Set up a real-world scene and a corresponding USD scene, and try to make them as similar as possible.
2. Collect around 50 demonstration episodes using LeIsaac.
3. The task is relatively simple: grasp a LEGO brick and lift it up.
4. Train a SmolVLA policy using the collected dataset.
5. Deploy and evaluate the policy on the real robot.

Thank you.

[EverNorif]

Have you tried evaluating it in simulation? Running inference directly on the real robot may suffer from a sim2real gap, which could lead to poor performance.

[SilVerC77]

Thank you for your reply. I will try to evaluate it in simulation.
By the way, I have another question. Do the observations defined in observations.py need to match what is available in the real world?

In my current implementation, I am using the LEGO brick coordinates directly from Isaac Sim.
However, in the real world, we obviously do not have direct access to the object's coordinates.

I am referencing the implementation from the scripts in the following folder:
./leisaac/source/leisaac/leisaac/tasks/lift_cube

[EverNorif]

@SilVerC77 In simulation, you can access more information that is not available in the real world, and you can define these as observations. The important part is which observations you use for training. Ideally, all observations used for policy training should also be available in the real world, such as camera images, joint positions, and so on.

If you did not use the LEGO brick coordinates when training the policy, I do not think this would have any impact.

[SilVerC77]

Hi,

I tried evaluating the model in Isaac Sim, but the inference results were much worse than I expected.
The evaluation results are summarized below:

Train with	Evaluate with	Result
Real	Real	Good (~10/10 successful grasps)
Real	Isaac Sim	The robot does not move at all.
Isaac Sim	Real	It mostly grasps empty air (less than 1/10 successful grasps).
Isaac Sim	Isaac Sim	Poor (less than 3/10 successful grasps).

Based on these results, it seems that:

1. A model trained on real-world data does not run properly in Isaac Sim.
2. A model trained on Isaac Sim performs poorly in both the real world and Isaac Sim.

Is this behavior expected, or could there be something wrong with my setup?

Has anyone encountered similar behavior? Any suggestions on what I should check or how to debug this would be greatly appreciated.

Thanks!