| title | description | author | ms.author | ms.date | ms.topic | ms.localizationpriority | keywords |
|---|---|---|---|---|---|---|---|
Unreal Development Overview |
Overview of mixed reality development using Unreal Engine 4 |
hferrone |
v-hferrone |
08/04/2020 |
article |
high |
Unreal, Unreal Engine 4, UE4, HoloLens, HoloLens 2, streaming, remoting, mixed reality, development, getting started, features, new project, emulator, documentation, guides, features, holograms, game development |
Getting started with mixed reality applications is a big task. New concepts, platforms, and cutting edge hardware can seem like barriers. However, if you're an Unreal developer you're in luck. Support for Windows Mixed Reality (VR) and HoloLens 2 (AR) is now included in Unreal Engine's newest release. This update includes:
- Mixed Reality UX Tools plugin support
- OpenXR support
- App Remoting from a desktop app
- Better performance
- Mixed reality capture
- Initial support for Azure Spatial Anchors
If you're new to Unreal development don't jump in blind. Explore the Unreal tutorial series to get up to speed and look for assets and support in the Unreal marketplace and mixed reality forums. These resources are your links to the community of builders and problem solvers in todays mixed reality market.
Important
Take a look at our porting guides if you have an existing Unreal project that you want to bring over to HoloLens 2 or immersive headsets such as the Reverb G2.
Use the following checkpoints to bring your Unreal games and applications into the world of mixed reality.
The Mixed Reality Toolkit for Unreal is a set of components designed to speed up your development in Unreal. Each component includes plugins, samples, and documentation for setting up immersive experiences.
- UX Tools for Unreal is the first component to be released and is currently only supported on HoloLens 2. The component plugin includes code, blueprints, and example assets of common UX features for input simulation, hand interaction actors, press-able button components, manipulator components and follow behavior components.
By the end of this section, you'll have a basic understanding of the Mixed Reality Toolkit, a properly configured development environment for Mixed Reality apps, and a working MRTK project in Unreal.
| Checkpoint | Outcome |
|---|---|
| Install the latest tools | Download and install the latest Unity package and setup your project for mixed reality |
| HoloLens 2 tutorial series | Dive into beginner level MRTK tutorials for HoloLens 2 hardware |
There are several key features of mixed reality development that our tutorial series doesn't cover. These building blocks are available as standalone features and through the Mixed Reality Toolkit. You might not need all of them at once, but we recommend exploring early on. After diving into the core building blocks listed below, you'll have a toolbox full of features you can integrate into your Mixed Reality projects.
Note
You can dive into the UX Tools for Unreal GitHub repository for more details.
Other key features that play a role in mixed reality applications are available without any extra packages or setup. These features can be added to Unreal projects with or without MRTK installed. After diving into these more advanced capabilities, you'll be able to build more complex Mixed Reality apps.
| Feature | Capabilities |
|---|---|
| HoloLens camera | Capture Mixed Reality and real-world visual content from your app running on a HoloLens device |
| QR codes | Render QR codes as holograms using a coordinate system at each code's real-world position |
| WinRT | Create a separate binary with WinRT code that can be consumed by Unreal’s build system |
If this is your first time creating or deploying an Unreal app for HoloLens, you'll need to download supporting files from the Epic Launcher. Once you have those files installed, you're ready to deploy from either the Unreal editor or the Device Portal.
At this point in your development journey you might be looking to add services or for a helping hand with commercial deployment. Integrating Azure Cloud Services and Dynamics 365 features can level up your projects in a major way. We've compiled a few starting points for you to explore and expand your Mixed Reality knowledge.
A developers job is never done, especially when learning a new tool or SDK. The following sections can take you into areas beyond the beginner level material you've already completed, along with helpful resources if you get stuck. Note that these topics and resources are not in any sequential order, so feel free to jump around and explore!
If you want to test out your application on a HoloLens device while still in development, you can stream it directly from your PC by using either the Unreal editor or a packaged Windows executable file.
If you're looking to debug the application with Visual Studio, follow these instructions.
Developing for mixed reality comes with performance checkpoints that depend on the platform. A HoloLens 2 app must run at 60 frames per second for holograms to appear stable and responsive. Luckily, we have performance recommendations for achieving this in your Unreal applications.
| HoloLens 2 Feature | Earliest Supported Unreal Engine Version |
|---|---|
| ARM64 support | 4.23 |
| Streaming from a PC | 4.23 |
| Spatial mapping | 4.23 |
| Hand and joint tracking | 4.23 |
| Eye tracking | 4.23 |
| Voice input | 4.23 |
| Spatial anchors | 4.23 |
| Camera access | 4.23 |
| QR codes | 4.23 |
| Spatial audio | 4.23 |
| Spectator Screen support for streaming | 4.24 |
| Planar LSR over streaming | 4.24 |
| Sample apps (HoloLens2Example and Mission AR) | 4.24 |
| Mobile multi-View: Performance hits 60 fps | 4.25 |
| 3rd camera render | 4.25 |
| Streaming from a packaged desktop app | 4.25.1 |
| Azure Spatial Anchors for HoloLens 2 (beta) | 4.25 |
| OpenXR support (beta) | 4.25 |
| UX Tools support (0.8) | 4.25 |
| Developer docs & tutorials | 4.25 |
[!div class="nextstepaction"] Install the tools