Hand tracking on the Pico 4 seems to be broken

[talpadk]

I believe that the hand tracking on pico 4 has a bug or two, or possibly I'm using OpenXR the wrong way
I'm using:

• Godot 4.3 beta 1
• Godot XR tools for Godot 4 (4.3.1) (for the hands and there skeletons)
• Godot OpenXR Vendors plugin v3.0.0 beta 1

I have tied both the depreciated OpenXRHand and XRNode3D + XRHandModifier3D

1. With both when I hold my hands horizontal the hands point up inside VR
2. Only the OpenXRHand modifies the bones, the XRHandModifier3D does nothing
3. The OpenXRHand with bone update = "rotation only" bends my fingers in the direction of my pinky finger
4. The Pico 4 insists that the program doesn't support running with hand tracking

In the export settings I have tired bot optional and required for hand tracking

In the project settings i have enabled hand tracking and tried both with and without "Hand Interaction Profile" enabled

Some of the rotation and bending issues might be solvable by applying a rotation matrix to the data from the Pico 4 or perhaps even just swapping an axis or two.
It may of cause simply be a bug in the Pico firmware, but it would be nice if it could be worked around until they fix it (if ever).

As to why the XRHandModifier3D doesn't modify the bones I wouldn't rule out a misconfiguration on my part

Oh and last but not least: Thanks for the plugin quite amazing how relatively easy it is to code VR applications using it

[talpadk]

If I apply the following Euler rotations to the Armature nodes:
Left hand: [0, 90, -90]
Right hand: [90,90,0]

The hands are oriented correctly inside VR.
(The fingers still bends the wrong way)

Which gives the rotation matrices
Left:
[ 0.0000000, 0.0000000, 1.0000000;
-1.0000000, 0.0000000, 0.0000000;
-0.0000000, -1.0000000, 0.0000000 ]

Right:
[ 0.0000000, 0.0000000, 1.0000000;
1.0000000, 0.0000000, -0.0000000;
-0.0000000, 1.0000000, 0.0000000 ]

So it seems like the coordinates from the Pico 4 should be rearranged slightly

And for the left hand two of them used as negative values

[talpadk]

If I use bone update full on the left hand that uses the depreciated OpenXRHand it renders in the correct orientation without any rotation of the armature.

But then it distorts the hand mesh

[dsnopek]

Thanks for the report!

Is this a regression (meaning, it works in Godot 4.2, but is broken in 4.3-beta1)?

If there is a bug here, it's likely in Godot itself, rather than this extension. The extension is really only responsible for the loader and Android manifest changes - basic OpenXR hand tracking support is in Godot.

[talpadk]

I will move the bug report to the Godot tracker instead.

Should the part about the Pico not picking up on the application supporting hand tracking remain here or is that also a Godot issue?

OpenXRHand behaves the same in Godot 4.2.2 (I can't select to only use rotational transform but other than that)
XRNode3D + XRHandModifier3D isn't available as far as I can tell in 4.2.2

And thanks for your help this far

[dsnopek]

I will move the bug report to the Godot tracker instead.

I think it's OK to keep it here until we figure out what's going on. If this is a matter of adding something to the Android manifest or requesting a Pico-specific permission, then the fix will be here. But if it's anything else, then the fix will need to be in Godot.

[BastiaanOlij]

Try and see if you can reproduce the issue in this demo: https://github.com/godotengine/godot-demo-projects/tree/master/xr/openxr_hand_tracking_demo

For me when I tried that on my PICO 4 it was working correctly.

Do note that the PICO 4 only provides hand tracking data if you are not using a controller, it does not support controller inferred hand tracking.

[BastiaanOlij]

Also seeing the mention of 3.0.0 beta 1 being used, I am not sure if we added the PICO changes in that yet. We added additional manifest entries in beta 2 and moved to the Khronos loader in beta 3 adding additional required manifest entries.

[talpadk]

Got stuck way too long on vendors beta 3 requires godot beta 3, drained all the fun out of it, will have to try reporting back later.

[ckschim]

I was trying to get hand tracking working on my PICO 4 as well and was facing the issues descibed by @talpadk.

Versions

• Godot 4.3 stable
• Godot OpenXR Vendors plugin v3.0.1

Issues

PICO says controllers are needed

I had to manually add <meta-data android:name="handtracking" android:value="1" /> to the Android Manifest.xml to let the PICO now that no controllers are needed for the application and go with the hand tracking.

Orientation of hands wrong

In my own small test app I was facing the described issues 1-3 so far as well, but in the hand tracking demo it is working fine.