XR Caption is an innovative solution for enhancing accessibility in mixed reality environments. This project enables real-time subtitles to appear in augmented reality (AR) settings, making information more accessible for individuals with hearing impairments. The project was prototyped at FIU ShellHacks for Microsoft's Mixed Reality Challenge and leverages cutting-edge AI and AR technologies.
- Real-Time Subtitles: Adds AI-generated captions to mixed reality experiences.
- Accessibility Focus: Designed to help the hearing impaired "hear with their eyes."
- Classroom Integration: Enhances educational settings by improving accessibility.
- Open-Source AR Headset: Utilizes a 3D-printed design that fits smartphones to reflect captions directly into the user’s view.
- AI: Built using OpenAI Whisper for real-time transcription and language processing.
- Web Framework: Developed with Three.js for rendering AR experiences.
- Hardware: An open-source, 3D-printed AR headset compatible with smartphones.
Caption: The open-source AR headset displaying captions.
Caption: Classroom demonstration of AI-powered captioning.
Experience XR Caption in action:
Test XR Caption live with the following interactive demos:
This project was developed as part of the FIU ShellHacks for Microsoft's Mixed Reality Challenge. Special thanks to OpenAI Whisper, Three.js, and the open-source AR community for enabling this innovation.
If referencing this project, please use:
Jay Rosen – XR Caption: Subtitles for Mixed Reality and Accessibility.
DOI: 10.5281/zenodo.14739260