Question about ACT behavior when object is absent or significantly displaced

[buttler11-max]

Hi, thank you for sharing this excellent work. The method is impressive, and the codebase is very helpful for reproducing and adapting ACT-style policies to real robot tasks.

I am currently using Leju 4Pro with a Qiangnao dexterous hand to train an ACT policy. The task is to grasp an object and place it into a box.

After training, the policy can complete the full task when the object is placed in the demonstrated region. However, I noticed that when the object is moved too far away from the training distribution, or when the object is completely removed from the scene, the policy still executes almost the same grasping and placing motion.

I would like to ask:

1、Is this expected behavior for ACT when the training data mainly contains successful demonstrations?
2、Would you recommend adding negative demonstrations, such as “no object → stay still” or “object too far → do not execute”?
3、Is there a recommended way to make the policy more sensitive to object presence and position, for example by adding object detection, failure cases, or additional observation/reward signals?
4、In your experience, is this mainly caused by overfitting to the demonstrated action trajectory, insufficient visual conditioning, or lack of out-of-distribution data?

[kevingao1113]

Hi, glad the codebase has been helpful for your setup with Leju 4 Pro and Qiangnao hand!

1. Yes, it is expected. We also noticed it during our own testing. Since ACT is a pure behavior cloning policy, when your training set only contains trajectories like "object in region - pick - place," the model never sees what "no object" and "object too far" look like, which will lead to executing PnP motion under these OOD situations. This is inherent to behavior cloning methods in general.

2. Sure, adding negative cases helps, but keep it a minority of the dataset. A small set of "no object stay still" demos (5–10% of total data) will teach the model how to behave in such no-object situations.

3. With data-driven methods, the most direct fix is usually in the data itself. Especially for those VLA policies, they are generally already sensitive enough to visual differences as long as the dataset is well designed. Other methods, like adding masks on visual input or reward-aligned BC, may work, but they will take you more time than focusing on data.

4. I would rank narrow distribution of training data as the primary reason, which will also make the model more prone to overfitting specific spatial patterns. So I suggest adding a small set of "no object" demos and enhancing the spatial diversity of object placement in your positive demos.

It would be great if you could share more info about the quantity and diversity of the dataset and how you collected them. Wish it helps.