Use VR controllers as trackers while preserving accurate hand tracking #2794
Conversation
This gives possibility to track real world object and hands in the same time. As example to track aiming of stationary placed real world gun by attached controller to it. And in the same time still have hands skeletones in game.
…-tracker-when-it-is-not-in-hand
| (DETACHED_CONTROLLER_LEFT, "/user/detached_controller_meta/left"), | ||
| (DETACHED_CONTROLLER_RIGHT, "/user/detached_controller_meta/right"), | ||
| (FAKE_TRACKER_LEFT, "/user/fake_tracker/left"), | ||
| (FAKE_TRACKER_RIGHT, "/user/fake_tracker/right"), |
There was a problem hiding this comment.
Why not using the DETACHED_CONTROLLER_* devices?
There was a problem hiding this comment.
I'm sending motion data from the controller while treating it as a tracker — both when it's detached and when it's held in hand. That's why I decided to separate out dedicated devices for this purpose.
I also assumed that you might have plans for handling detached devices in your own way, and I didn’t want my specific implementation to conflict with your future design.
There was a problem hiding this comment.
So for example when the left controller held, the hand position is duplicated between HAND_LEFT AND FAKE_TRACKER_LEFT?
There was a problem hiding this comment.
When using controller as fake tracker mode is on i don't send HAND_LEFT motion data, Just FAKE_TRACKER motion data is sending and skeleton data for hands
There was a problem hiding this comment.
I'll propose a different solution. Let's have the client send either held controllers or detached controllers (depending in the controllers proximity to the hands), and on the server side decide if to recombine the controller positions into a single device position. Detached or held controller motions would be mutually exclusive anyways.
If i was to redesign the Tracking structure, i would add a is_left/right_detached booleans, so HAND_LEFT/RIGHT would suffice for your usecase. The problem is we can't make breaking changes yet (but we are planning to)
There was a problem hiding this comment.
Initially, I actually did just that — I was sending motion data from detached controllers into HAND_LEFT / HAND_RIGHT when the controller wasn’t held.
However, I was concerned this might interfere with your future plans for detached controllers, so I decided to isolate my functionality by wrapping this data into separate virtual devices instead.
If you're okay with the client writing detached controller data into HAND_LEFT / HAND_RIGHT when the controller is not held, I’m happy to switch back to that approach. I’ll then handle the interpretation on the server side in a way that fits my use case.
There was a problem hiding this comment.
No, use the first solution I mentioned, which is to properly implement detached controllers on the client side. On the server side you would read from both detached and held controllers, whichever is active
| FAKE_LEFT_TRACKER_ID = PathStringToHash("/user/body/left_elbow"); | ||
| FAKE_RIGHT_TRACKER_ID = PathStringToHash("/user/body/right_elbow"); |
There was a problem hiding this comment.
The reason I waited to implement support for detached controllers is because we don't have a way of redefining which tracker should be used for them. Hardcoding some specific tracker might be annoying and even surprising for the end user. I would suggest to make it configurable on the settings side.
|
The check fails seem to be unrelated to this PR, we will fix them ASAP |
|
Whenever you can, rebase your changes on top of the latest master to remove the lint errors. Please rebase and don't use merge commits |
We have a VR project — a machine gun simulator — where we use a real physical model of a machine gun. Our goal is to track where the user is aiming with the real machine gun and transfer that data into the virtual world. At the same time, we also need to track the user’s hands so that the player can see their virtual hands and accurately grab the machine gun handles.
To achieve this in Unity3D, we had to treat the VR controllers as trackers, and then extract tracking data from them in Unity accordingly.