Where Am I?

Our project has two parts - the server for running the localization algorithm, with the code in the Server folder, and the mixed reality application for showing user position and overlay in UnityApplication. For clarity, we merge the READMEs for both projects below. They are also available in the respective folders.

Server - Image-Based Localization with HLoc

This project runs a server to predict image location using the HLoc algorithm.

---

How It Works

1. Input: The client sends an image to the server.
2. Processing: The server uses the HLoc algorithm to predict the location based on the input image.
3. Output: The server sends the location prediction back to the client.

---

Data

The following directory structure is necessary for the feature extraction and inference

.
├── datasets
│ ├── dataset_1 # Input images for localization
│ │ ├──  session_1
│ │ │ ├──  processed_data  
│ │ │ │ ├──  images    
│ │ │ │ │ ├── 1.jpg  
│ │ │ │ │ └── ...  
│ │ │ │ └──  depths  
│ │ │ │ │ ├──  1.png    
│ │ │ │ │ └──  ...  
│ │ │ ├──  images.txt  
│ │ │ ├──  depths.txt  
│ │ │ ├──  rigs.txt  
│ │ │ └──  global_trajectories.txt  
│ │ └──  ...  
│ └── ...  
├── outputs  
│ ├── dataset_1  
│ │ ├── Netvlad_features.h5  
│ │ └── SuperGlue_features.h5  
│ └── ...  

Preprocessing

For preprocessing you need to have for every image in your data a depth image and a known global position and rotation. Then do feature extraction using HLoc

---

Inference

To do inference we provide two files

• inference.py
• server.py

server.py runs a server that does real time prediction of received images and can be run with:

fastapi run server.py

while inference.py is useful for testing and does inference on a directory of images and visualizes.

---

Requirements

• Python 3.11
• HLoc library and its dependencies
• Fastapi

---

Where Am I? AR Application

This repository contains the AR application of the Where Am I? project for the Mixed Reality course at ETH. The project follows a commonly used unity project structure, but we provide a brief summary nonetheless.

We only mention the files created by us - rest is generated by the unity project automatically, and there is nothing to describe.

• Assets - all assets for the unity project, including scenes, materials, models and more
  • Assets/Materials - VFX shaders, responsible for the pulsing points, trajectory footsteps animations, map reveal and more
  • Assets/Models - 3D models used in the app. Includes the HG overlay, markers placed on the map and some utility models
  • Assets/Prefabs - Unity prefabs, = objects that can be added into the world with scripts and components added and with some preset properties
  • Assets/Rendering - The render texture for the minimap in the top left corner (only in the Windows demo)
  • Assets/Scenes - the scenes of our app. FirstScene.unity is the main one used in the app, DebugScene.unity has limited functionality, e.g. no networking and some missing features
  • Assets/Scripts - All C# scripts used in the app
  • Assets/Sprites - Textures used. Includes the footprint, map textures, the HG floor plan, and more
  • Remaining folders were autogenerated by unity or the added packages
• Utils - code we used for the preprocessing of the LaMAR dataset
  • Utils/LamarPointcloudsDownscaling - reading the PLY file format and selecting a uniformly sampled subset of the points
  • Utils/LamarReadTrajectories - parsing the trajectory files of LaMAR, converting them into a format ready to use in unity
  • Utils/NavigationGraph - reading LaMAR trajectory files, converting them into a navigation graph that can be used in the app
• Remaining code is generated automatically, we used the procedures shown in the tutorials to create an initial AR project and to obtain all required packages