Description
An Interactive Regions API would give the ability to define interactive regions within an immersive WebXR experience, allowing the OS (or the browser) to highlight these regions when gazed upon, providing users with intuitive visual feedback in a privacy conscious way.
The WebXR experience would be responsible for creating interactive regions and attaching them to an active WebXR session, and optionally listening for 'click' events on these regions.
Background
Hover effects are useful and ubiquitous on the web but they raise serious privacy concerns if they are driven directly by a user's gaze (e.g. via eye tracking on an HMD) in a naive way. To strike a balance between usability and user privacy, Apple introduced a privacy-preserving method with its visonOS wherein elements are visually highlighted when a user looks at them, but crucially the highlighting is done only by the OS, without any applications having knowledge of the highlighting state.
Applications must be able to specify the areas to highlight, 'interactive regions', to the OS. In the case of the Safari browser, the WebKit engine generates these regions on the 2D page from various pieces of webpage markup and styles. The webpage thus determines where these interactive regions are.
Extending this concept into the 3D world of WebXR would be a logical progression. While WebXR doesn't rely on traditional 2D DOM elements which a web engine could automatically identify and highlight, immersive experiences have their own set of 3D elements that users can interact with that would greatly benefit from being able see highlighted like they can see elsewhere in their interactions on the platform. As the browser can't automatically identify regions, the web developer would have to explicitly supply them.
References:
WWDC Safari talk, discussion of 2D web interactive regions:
https://developer.apple.com/videos/play/wwdc2023/10279/?time=288
Code:
An example of what usage of Interactive Regions would look like:
// Engine object
const position = { x: 1.0, y: 1.5, z: -3.0 };
const orientation = { x: 0, y: 0, z: 0, w: 1 };
Const scale = { x: 0.5, y: 0.2, z: 0.1 };
Let floatingButtonObj = new Engine.Button3D('floatingButton', position, orientation, scale);
// create an Interactive Region
let interactiveButtonRegion = new XRInteractiveRegion();
interactiveButtonRegion.regionId = 'floatingButton';
interactiveButtonRegion.transform = new XRRigidTransform(position, orientation);
interactiveButtonRegion.width = 0.5; // meters
interactiveButtonRegion.height = 0.2;
interactiveButtonRegion.style = new XRRegionAppearanceAttributes({ highlightRadius: 0.1 });
xrSession.interactiveRegions.setSpace(xrSpace);
// Add region
xrSession.interactiveRegions.addRegion(interactiveButtonRegion);
// Handle events
// new event
xrSession.addEventListener('interactiveregionclick', (event) => {
if (event.regionId === 'floatingButton') {
// Handle interaction
floatingButtonObj.click();
}
});
// Or could use 'select'
xrSession.addEventListener('select', (event) => {
let source = event.inputSource;
let frame = event.frame;
let targetRayPose = frame.getPose(source.targetRaySpace, myRefSpace);
// If the interaction source has an associated region
if (source.interactionRegion) {
if (source.interactionRegion.id === 'floatingButton') {
// Handle interaction...
}
} else {
let targetObject = findTargetUsingRay(targetRay.transform.matrix);
// ...
}
});Prototype:
.