Add OpenXR fb_render_model extension wrapper and OpenXRFBRenderModel node

[devloglogan]

Adds an extension wrapper for the OpenXR XR_FB_render_model extension and corresponding node implementing its usage. This extension provides access to gltf models of controllers/keyboards at runtime.

The extension wrapper caches a list of valid paths to these models (accessible with function get_valid_paths() in the OpenXRFBRenderModel node or the same function in the extension wrapper singleton). One of these paths can be assigned to the render_model_path property of the OpenXRFBRenderModel node, which will trigger that model to be loaded and accessible via the nodes get_render_model() function. The only other time the model is loaded is on startup via the OpenXrInterface session_begun signal.

At present (tested on Quest 2 and Quest 3), it seems only the controller models are able to be successfully loaded. The virtual keyboard requires the XR_META_virtual_keyboard extension to be enabled, but I believe the other two should be available. Unsure if there's another dependency missing for these. The controller models have been added to the demo main scene.

[m4gr3d]

Looks great! I've added some minor feedback to address.

[Malcolmnixon]

This works well for me!

[BastiaanOlij]

Looks really good at an early glance.

I'm concerned about the requirement of the rotation of the grip pose to position the render model, that should match up or the spec should provide information about correct placement. If it doesn't that's going to be bad and bite us down the road if other vendors follow suit but do properly position the meshes.

On that subject, exposing the render models to the user should take a more global approach and not be FB centric. Here I'm unfortunately not at liberty to provide more detail yet, but we're going to back ourselves into a corner if we do not take a wider view.

[devloglogan]

I'm concerned about the requirement of the rotation of the grip pose to position the render model

To be clear, the render model isn't being repositioned by anything other than the setting of the grip pose. The hacky rotation is only there for repositioning the hand tablet correctly from one pose to the other. Would this not be expected? We could just set the pose for the controllers to be grip by default and reposition the hand tablet in the editor instead.

[dsnopek]

We could just set the pose for the controllers to be grip by default and reposition the hand tablet in the editor instead.

Yes, I think we should do exactly this!

[devloglogan]

Alright, I changed the default controller poses to grip and added some (hard coded) render_model_path suggestions

[dsnopek]

@BastiaanOlij:

On that subject, exposing the render models to the user should take a more global approach and not be FB centric. Here I'm unfortunately not at liberty to provide more detail yet, but we're going to back ourselves into a corner if we do not take a wider view.

If a vendor neutral OpenXR extension is added to the spec, we'd implement that in Godot itself. So, personally, I don't think having the Meta-specific one here in 'godot_openxr_vendors' paints us into any corners. Folks not specifically targeting Meta headsets won't use their vendor extensions (they'll wait for the vendor neutral one), and Meta's extension will keep working on Meta devices even after the vendor neutral one exists.

[m4gr3d]

One minor comment to address but otherwise it looks good!

[devloglogan]

Updated based on the reparenting / model path comments, created an issue for the keyboard models.

[BastiaanOlij]

Just an intermediate update from me as I'm waiting on hearing back before we can make an informed decision on how to proceed here. Until I can share updates, best to press pause, hope to have info in the next couple days.

[BastiaanOlij]

If a vendor neutral OpenXR extension is added to the spec, we'd implement that in Godot itself. So, personally, I don't think having the Meta-specific one here in 'godot_openxr_vendors' paints us into any corners. Folks not specifically targeting Meta headsets won't use their vendor extensions (they'll wait for the vendor neutral one), and Meta's extension will keep working on Meta devices even after the vendor neutral one exists.

It's not that simple. The following scenario is fairly likely:

• we implement the meta extension as is
• in the near future we implement the official solution but there is no guarantee Meta will immediately support it as they already have a solution
• our users need to build something that uses the Meta solution on Quest, and the official solution on everything else, the most likely scenario is simply checking which one is available.
• Meta adds supports for the official solution but keeps the Meta solution around for backwards compatibility, both are now reported as available
• Existing Godot apps potentially will break as they try to activate both Meta and official solutions at the same time.

So we have a mandate to at least discuss whether we can come up with a solution that staves off this scenario (which is very likely due to reasons I'm not at liberty to share), or whether we should forge ahead and deal with the fallout later.

[devloglogan]

@BastiaanOlij rebased!

[m4gr3d]

@BastiaanOlij @dsnopek From our discussions, it's likely we'll move forward with this approach as a v0 api. Can you review in that context.

@devloglogan Can you rebase the PR for review.

[devloglogan]

Rebased! Per Fredia's request I've swapped the model selector on the OpenXRFbRenderModel node to be an enum type. After doing that, it seemed pointless storing path names and exposing them, so I removed that functionality.

[devloglogan]

@m4gr3d requests added!

[dsnopek]

This is looking great to me!

I wonder if the "classes" directory should really be "scene" because this is a node and that's what Godot names the directory that holds nodes? If we end up with some RefCounted classes it may be nice to have those in a different directory. But what we have here is certainly fine for now :-)

[m4gr3d]

One thing we keep forgetting to do, can you update the CHANGES.md doc and reference the new feature.

[BastiaanOlij]

Ok Logan, just starting. Getting everything to compile and to a point where I thought I could run things took a lot longer than expected, and I ran into a number of snags I'm still looking into. Part of it was me getting familiar with things again (haven't had a deep dive into the vendors plugin), part is that we may have lost some instructions or that a clean build is throwing up some issues.

So there is more to come. So far it looks good, I'm mostly worried about some flow things around the model loading at the wrong time and in the wrong scenario.

Also my apologies that it took so long before I could really sink my teeth into. There are things happening on the core of OpenXR where doing something a little more elaborate to make this a bit more future proof may have been a necessary thing.

But looking at some structural differences and the fact that things will take longer than I hoped, I'm now of the belief that this is too soon and that an FB specific implementation as you've done here, is the right course of action. I also do not believe it likely that other vendors will come out with their own extension which would require us to think this through a little more.

That will mean that in the near future users who implement this extension and are planning to release games beyond the Meta platform, will likely need to refactor their code. We will deal with that, when the time is right.

[m4gr3d]

@BastiaanOlij Looks like all your feedback on the PR have been addressed. Are you good with merging the current implementation?

[BastiaanOlij]

@m4gr3d I haven't gotten anywhere near actually testing the latest version, as I was discussing with David the other day I am getting fairly concerned with this, especially as the testing I did get around doing last week show that a few things have snuck in that are causing issues and right now the plugin is in a broken state for me.

Purely based on the code and the discussion around this however, I am happy. But I would like to actually take it for a drive before giving my final thumbs up.

[BastiaanOlij]

OK, so thanks to some motivational help from Fredia I managed to get things working today. There are some issues unrelated to this PR but the big issues that got me stuck and ready to throw in the proverbial towel, seemed to have been solved.

The only one that worries me but is likely again unrelated to this PR, is that the Godot IDE hung while I was looking around at the setup.

There is one question that should be down below for what happens when controllers are let go and some suggestions for improving the demo. I'm happy for this to be merged and that issue to be raised as a separate issue to be handled in a follow up PR.

[BastiaanOlij]

I'm not sure about the best implementation but the one thing I was missing is control/signals related to the controller no longer being tracked.

Right now in the demo you react to the openxr_fb_render_model_loaded signal, and hide the hand tracking but that seems to be too restrictive, and once we use proper hand tracking, it should actually show the hand properly holding the controller (and thus verifying the positioning is correct).

Off course this could be a limitation of the demo, and possibly the logic I'm looking for shouldn't be part of this but should be part of the general hand tracking implementation that is in the core of Godot.

If the latter, we should probably do a follow up PR that adds hand tracking into the demo with the hand meshes Malcolm has (I plan to do a version of this during the week for the godot-demo-projects repo).

But if there is more to handle here, especially if tracking of controllers stops and the controllers should be hidden, that is worth looking into before we merge this.

[devloglogan]

My initial thought: this extension exists to provide 3D models, so our implementation should just expose those 3D models, and any further logic should be done by end users (or at least not here). It's possible users may want to drop static controller models in other areas, not just use them for the tracked controllers.

[dsnopek]

I agree with that. There's also more models than just controllers, this can also be used for physics keyboards (for the tracked keyboard feature) and the advanced virtual keyboard (which isn't tracking anything at all).

EDIT: Or could be used for those things once we add the extra code to enable them. :-) But I think we'd still want to use this same node for those.

[m4gr3d]

@devloglogan Can you rebase the PR so we can proceed to merge it!

[devloglogan]

@m4gr3d rebased