| title | description | author | ms.author | ms.date | ms.topic | ms.localizationpriority | keywords |
|---|---|---|---|---|---|---|---|
Unreal Development Overview |
Overview of mixed reality development using Unreal Engine 4 |
hferrone |
v-hferrone |
08/04/2020 |
article |
high |
Unreal, Unreal Engine 4, UE4, HoloLens, HoloLens 2, streaming, remoting, mixed reality, development, getting started, features, new project, emulator, documentation, guides, features, holograms, game development, mixed reality headset, windows mixed reality headset, virtual reality headset |
Getting started with mixed reality applications is a significant task. New concepts, platforms, and hardware can seem like barriers. However, if you're an Unreal developer you're in luck. Support for Windows Mixed Reality (VR) and HoloLens 2 (AR) is now included in Unreal Engine's newest release. This update includes:
- Mixed Reality UX Tools plugin support
- OpenXR support
- App Remoting from a desktop app
- Better performance
- Mixed reality capture
- Initial support for Azure Spatial Anchors
If you're new to Unreal development, don't jump in blind. Explore the Unreal tutorial series and look for assets in the Unreal marketplace. You can also find support in the mixed reality forums. These resources are your links to the community of builders and problem solvers in today's mixed reality market.
Important
Take a look at our porting guide if you have an existing Unreal project that you want to bring over to immersive headsets such as the Reverb G2.
Use the following checkpoints to bring your Unreal games and applications into the world of mixed reality. If you haven't explored the Designing Holograms sample application, we recommend downloading it to familiarize yourself with the basics of Mixed Reality UX.
The Mixed Reality Toolkit for Unreal is a set of components designed to speed up your development in Unreal. Each component includes plugins, samples, and documentation for setting up immersive experiences.
- UX Tools for Unreal is the first component to be released and is currently only supported on HoloLens 2. The component plugin includes code, blueprints, and example assets of common UX features for input simulation, hand interaction actors, press-able button components, manipulator components and follow behavior components.
By the end of this section, you'll have a basic understanding of the Mixed Reality Toolkit, a properly configured development environment for Mixed Reality apps, and a working MRTK project in Unreal.
| Checkpoint | Outcome |
|---|---|
| Install the latest tools | Download and install the latest version of Unreal Engine and set up your project for mixed reality |
| HoloLens 2 tutorial series | Dive into beginner level MRTK tutorials for HoloLens 2 hardware |
There are several key features of mixed reality development that our tutorial series doesn't cover. These building blocks are available as standalone features and through the Mixed Reality Toolkit. You might not need all of them at once, but we recommend exploring early on. After diving into the core building blocks listed below, you'll have a toolbox full of features you can integrate into your Mixed Reality projects.
Note
You can dive into the UX Tools for Unreal GitHub repository for more details.
Other key features that play a role in mixed reality applications are available without any extra packages or setup. These features can be added to Unreal projects with or without MRTK installed. After diving into these more advanced capabilities, you'll be able to build more complex Mixed Reality apps.
| Feature | Capabilities |
|---|---|
| HoloLens camera | Capture Mixed Reality and real-world visual content from your app running on a HoloLens device |
| QR codes | Render QR codes as holograms using a coordinate system at each code's real-world position |
| WinRT | Create a separate binary with WinRT code that can be consumed by Unreal’s build system |
If you're a first-time Unreal app developer for HoloLens, you'll need to download supporting files from the Epic Launcher. Once you have those files installed, you're ready to deploy from either the Unreal editor or the Device Portal.
At this point in your development journey, you might be looking to add services or for a helping hand with commercial deployment. Integrating Azure Cloud Services and Dynamics 365 features can level up your projects in a major way. We've compiled a few starting points for you to explore and expand your Mixed Reality knowledge.
A developer's job is never done, especially when learning a new tool or SDK. The following sections can take you into areas beyond the beginner level material you've already completed, along with helpful resources if you get stuck. Resources aren't in any sequential order, so feel free to jump around and explore!
If you want to test out your application on a HoloLens device while still in development, you can stream it directly from your PC by using either the Unreal editor or a packaged Windows executable file.
If you're looking to debug the application with Visual Studio, follow these instructions.
Developing for mixed reality comes with performance checkpoints that depend on the platform. A HoloLens 2 app must run at 60 frames per second for holograms to appear stable and responsive. Luckily, we have performance recommendations for upgrading performance in your Unreal applications.
| HoloLens 2 Feature | Earliest Supported Unreal Engine Version |
|---|---|
| ARM64 support | 4.23 |
| Streaming from a PC | 4.23 |
| Spatial mapping | 4.23 |
| Hand and joint tracking | 4.23 |
| Eye tracking | 4.23 |
| Voice input | 4.23 |
| Spatial anchors | 4.23 |
| Camera access | 4.23 |
| QR codes | 4.23 |
| Spatial audio | 4.23 |
| Spectator Screen support for streaming | 4.24 |
| Planar LSR over streaming | 4.24 |
| Sample apps (HoloLens2Example and Mission AR) | 4.24 |
| Mobile multi-View: Performance hits 60 fps | 4.25 |
| Third camera render | 4.25 |
| Streaming from a packaged desktop app | 4.25.1 |
| Azure Spatial Anchors for HoloLens 2 (beta) | 4.25 |
| OpenXR support (beta) | 4.25 |
| UX Tools support (0.8) | 4.25 |
| Developer docs & tutorials | 4.25 |
[!div class="nextstepaction"] Install the tools