This project showcases Augmented Reality by projecting a 3D Mesh Model onto a 2D reference surface. It is programmed solely in C++ and uses the metal-cpp library as Apple Silicon GPU interface. Also, OpenCV is being used for the usage of Computer Vision algorithms. As the IDE of choice, Visual Studio Code (VSCode) has been used.
This project works only on OSX devices with Apple Silicon GPUs. Also have a look at my other projects on Augmented Reality with Python and Augmented Reality on Android.
Here you can see the project in action. Also note the change of light exposure on change of position.
ar-3d-rendering-showcase.mov
In this section I'm going to elaborate some of the key features of this project.
In order to detect a reference image in the scene, Computer Vision algorithms for feature detection and feature matching are used. I'm using a SIFT detector for reliable matches and a Brute Force matcher with cross-check. After computing the matches, they can be used to obtain a homography estimation. Note that it only works for planar objects as reference images. From this homography we can then extract the rotation matrix as well as the translation vector which then are used to render the 3D object in the right position.
Basically we have three different coordinate systems which we need to take care of:
- Camera (width corresponds to number of columns (Y), height corresponds to number of rows(X))
- Reference Object (width corresponds to X, height corresponds to Y)
- 3D Mesh Model
The 3D Mesh Model will be projected onto the reference object. In order to render the 3D Mesh Model correctly, I had to export it with the up axis being the negative Z-Axis so that it aligns with the reference object's coordinate frame.
For all the position estimation calculations, it is assumed that the camera stands at the point (0;0;0) and the rendered 3D Mesh Object will be rendered into position relative to that.
As mentioned before, the Metal Shading Language is used to interact with Apple Silicon GPUs. The vertices of the 3D Mesh Model are then passed to the GPU which computes the position as well as the textures.
In order to compile updates to the model.metal, the shell script compile-metal-lib.sh in the project root can be used. All it does is to build the metal library which then can be loaded from the C++ context.
Just for fun there is an artificial light at a fixed position in the scene which illuminates the rendered object depending on its position and rotation.
The homography estimations work without prior camera calibration as provided in the course "Image Processing and Computer Vision 2" at the University of Applied Sciences in Eastern Switzerland (OST). The 3D Rendering utilizes the GPU for performance and is based on the great tutorial at https://metaltutorial.com.
First you need to install OpenCV which you can do with homebrew. Afterwards, you have to adapt the path to OpenCV, OpenCV_DIR, in src/CMakeLists.txt.
Then you can open the project in any IDE. I used VSCode with the "CMake Tools" extension and clang 15.0.0 compiler.
If you get the following error:
Assertion failed: (metalDefaultLibrary != nullptr), function createDefaultLibrary, file MetalEngine.cpp, line 39.
Make sure you run the project from the project root - this can be checked if the application prints the following in the beginning:
Current path is "<project-root>"
To register a new reference object, please refer to Register new reference object to unwarp the image. Then copy the resulting image to assets/private and set the filename of the attribute referenceImage in ARWebcam.hpp.