OpenXR: Virtual Reality Rendering

[johannesvollmer]

Hi! This project sounds awesome. I just discovered it and wanted to talk about XR support.

One could assume that this could be implemented as an external module. However, the OpenXR library creates one or more render targets (for Vulkan/GL/...) based on the physical XR hardware. Therefore, the XR module needs to talk to the rendering module. As a consequence, a few generic changes to the rendering module may possibly be required, changes that are most probably unspecific to OpenXR. That's why it's reasonable to talk about this now already.

I'd like to look into OpenXR, which already has Rust bindings. However, the architecture should probably be prototyped in collaboration, as I don't know all the bevy stuff yet. I'll read all the bevy documentation and then come back to this issue.

I'd be happy to hear your thoughts to this! Cheers

[cart]

I would love to support XR. I've never personally written code in that space, but I have a general idea about a few ways it could integrate with bevy. Does OpenXR interact directly with render apis (ex vulkan / opengl) or is it abstract (in a way that we could use the bevy render api directly)?

If it needs direct vulkan access, then we can either:

1. build a wgpu extension that exposes the relevant apis
2. build a second bevy render backend that uses an api (like vulkan) directly

my preference would be to do this through wgpu to reduce complexity on our end (and also improve the wgpu ecosystem).

[erlend-sh]

Probably should be done upstream: gfx-rs/gfx#3219

[johannesvollmer]

Yeah, it needs direct vulkan access. It should really be done upstream.

This means that bevy should only expose the gfx XR API after gfx adds support

[StarArawn]

I want to point out once gfx-hal supports openXR you'd have to create a wgpu native(web wont get this) extension as well. Keep in mind we'd need some way of turning on/off wgpu native extensions in bevy. Current extensions can be found here:
https://wgpu.rs/doc/wgpu/struct.Features.html

[IceSentry]

gfx-rs/gfx#3219 mentions that the api could potentially be based on WebXR which means it could also target the web without needing a native and web extension

[ambeeeeee]

The WGPU API will be based on WebXR, the gfx one is going to be based on OpenXR.

[jpryne]

Is this a work in progress? Can I study the code, perhaps even make a pull request?

[ndarilek]

I could be wrong about this, but I've been in the community for a while and haven't seen anyone actively moving it forward. I imagine you could take it if you wanted. I, personally, am interested in this if only because I hope it'll get us better spatial audio support, which I think is a bigger deal in VR than it is in most desktop/mobile games. Be prepared to draw up an RFC first, though, as that seems to be the direction we're headed in. Alternatively, you may be able to knock together a renderer similar to what bevy_webgl2 did, though at some point there'll probably be a bureaucratic hurdle to overcome. :)

[agrande]

I've been working on this for a while, continuing what @amberkowalski started on the gfx side of things, but I'm new to Rust, Vulkan and OpenXR. Here's what I have so far: https://github.com/agrande/gfx/tree/gfx-xr. Not quite there yet, but I'm actively working on it.
I don't feel experienced enough to lead the effort, but since no one seemed to be doing it, and I really want to have VR in bevy, I decided to give it a go. Maybe we should open a channel on the discord to coordinate the efforts and make sure we all have the same level of information, as several people seem interested in the topic...

[SafariMonkey]

I am not experienced in OpenXR, but I have some OpenVR experience. I would like to see a channel or other place for discussion if that's possible, as I am also in the position of wanting to contribute, but not having enough experience that I feel able to lead the effort. If we can coordinate, we can hopefully make some progress without too much duplication of effort.

[agrande]

Ok, so now we have an XR channel on discord to coordinate the effort 🙌

[johannesvollmer]

Nice. I'm afraid I can't help as I currently do not have access to a vr headset anymore