Add lidar basemap to map expansion (nutonomy#500)

* Style and type changes

* Tests and patches for lanes

* Add new error case

* Tests on ego poses

* Render map and test new map

* Cleanup and fix empty connectivity

* Remove disconnected subtrees

* Script to fix maps

* Remove traffic light poses

* Hard-code maps

* Restore fix map

* Typos and format

* Add new bitmap class

* Render basemap

* Better bitmap support

* Bug

* Rename get_lane to get_arcline_path

* fixed bug in tutorial

* Addressed review comments

* Check earlier if map version is correct

* More error checks

* Improve documentation, bump map to v1.3

* Added warning to drop_disconnected_lanes

* Drop one lane

Co-authored-by: Holger Caesar <holger.caesar@motional.com>
Co-authored-by: Oscar Beijbom <oscar.beijbom@motional.com>

README.md

@@ -14,11 +14,13 @@ Welcome to the devkit of the [nuScenes](https://www.nuscenes.org/nuscenes) and [
   - [Prediction challenge](#prediction-challenge)
   - [CAN bus expansion](#can-bus-expansion)
   - [Map expansion](#map-expansion)
+  - [Map versions](#map-versions)
   - [Getting started with nuScenes](#getting-started-with-nuscenes)
 - [Known issues](#known-issues)
 - [Citation](#citation)
 
 ## Changelog
+- Nov. 23, 2020: Devkit v1.1.2: Release map-expansion v1.3 with lidar basemap.
 - Nov. 9, 2020: Devkit v1.1.1: Lidarseg evaluation code, NeurIPS challenge announcement.
 - Aug. 31, 2020: Devkit v1.1.0: nuImages v1.0 and nuScenes-lidarseg v1.0 code release.
 - Jul. 7, 2020: Devkit v1.0.9: Misc updates on map and prediction code.
@@ -125,10 +127,19 @@ To install this expansion, please follow these steps:
 In July 2019 we published a map expansion with 11 semantic layers (crosswalk, sidewalk, traffic lights, stop lines, lanes, etc.).
 To install this expansion, please follow these steps:
 - Download the expansion from the [Download page](https://www.nuscenes.org/download),
-- Extract the .json files to your nuScenes `maps` folder.
+- Extract the contents (folders `basemap`, `expansion` and `prediction`) to your nuScenes `maps` folder.
 - Get the latest version of the nuscenes-devkit.
 - If you already have a previous version of the devkit, update the pip requirements (see [details](https://github.com/nutonomy/nuscenes-devkit/blob/master/docs/installation.md)): `pip install -r setup/requirements.txt`
 - Get started with the [map expansion tutorial](https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/tutorials/map_expansion_tutorial.ipynb).
+For more information, see the [map versions](#map-versions) below.
+
+### Map versions
+Here we give a brief overview of the different map versions:
+- **v1.3**: Add BitMap class that supports new lidar basemap and legacy semantic prior map. Remove [one broken lane](https://github.com/nutonomy/nuscenes-devkit/issues/493).
+- **v1.2**: Expand devkit and maps to include arcline paths and lane connectivity for the prediction challenge.
+- **v1.1**: Resolved issues with ego poses being off the drivable surface.
+- **v1.0**: Initial map expansion release from July 2019. Supports 11 semantic layers.
+- **nuScenes v1.0**: Came with a bitmap for the semantic prior. All code is contained in nuscenes.py.
 
 ### Getting started with nuScenes
 Please follow these steps to make yourself familiar with the nuScenes dataset:
@@ -155,6 +166,7 @@ However, some minor issues remain:
 - For *singapore-hollandvillage* and *singapore-queenstown* the traffic light 3d poses are all 0 (except for tz).
 - For *boston-seaport*, the ego poses of 3 scenes (499, 515, 517) are slightly incorrect and 2 scenes (501, 502) are outside the annotated area. 
 - For *singapore-onenorth*, the ego poses of about 10 scenes were off the drivable surface. This has been **resolved in map v1.1**.
+- Some lanes are disconnected from the rest of the lanes. We chose to keep these as they still provide valuable information. 
 
 **Annotations**:
 - A small number of 3d bounding boxes is annotated despite the object being temporarily occluded. For this reason we make sure to **filter objects without lidar or radar points** in the nuScenes benchmarks. See [issue 366](https://github.com/nutonomy/nuscenes-devkit/issues/366).

python-sdk/nuscenes/eval/detection/data_classes.py

@@ -21,7 +21,7 @@ def __init__(self,
                  dist_th_tp: float,
                  min_recall: float,
                  min_precision: float,
-                 max_boxes_per_sample: float,
+                 max_boxes_per_sample: int,
                  mean_ap_weight: int):
 
         assert set(class_range.keys()) == set(DETECTION_NAMES), "Class count mismatch."

python-sdk/nuscenes/eval/prediction/baseline_model_inference.py

@@ -18,7 +18,7 @@ def main(version: str, data_root: str,
          split_name: str, output_dir: str, config_name: str = 'predict_2020_icra.json') -> None:
     """
     Performs inference for all of the baseline models defined in the physics model module.
-    :param version: nuScenes data set version.
+    :param version: nuScenes dataset version.
     :param data_root: Directory where the NuScenes data is stored.
     :param split_name: nuScenes data split name, e.g. train, val, mini_train, etc.
     :param output_dir: Directory where predictions should be stored.

python-sdk/nuscenes/eval/prediction/compute_metrics.py

@@ -44,7 +44,7 @@ def main(version: str, data_root: str, submission_path: str,
     """
     Computes metrics for a submission stored in submission_path with a given submission_name with the metrics
     specified by the config_name.
-    :param version: nuScenes data set version.
+    :param version: nuScenes dataset version.
     :param data_root: Directory storing NuScenes data.
     :param submission_path: Directory storing submission.
     :param config_name: Name of config file.

python-sdk/nuscenes/eval/prediction/metrics.py

@@ -207,7 +207,7 @@ def __init__(self, k_to_report: List[int], aggregators: List[Aggregator]):
         """
         Computes the minimum average displacement error over the top k predictions.
         :param k_to_report:  Will report the top k result for the k in this list.
-        :param aggregators: How to aggregate the results across the data set.
+        :param aggregators: How to aggregate the results across the dataset.
         """
         super().__init__()
         self.k_to_report = k_to_report
@@ -242,7 +242,7 @@ def __init__(self, k_to_report, aggregators: List[Aggregator]):
         """
         Computes the minimum final displacement error over the top k predictions.
         :param k_to_report:  Will report the top k result for the k in this list.
-        :param aggregators: How to aggregate the results across the data set.
+        :param aggregators: How to aggregate the results across the dataset.
         """
         super().__init__()
         self.k_to_report = k_to_report
@@ -279,7 +279,7 @@ def __init__(self, k_to_report: List[int], aggregators: List[Aggregator],
         If any point in the prediction is more than tolerance meters from the ground truth, it is a miss.
         This metric computes the fraction of predictions that are misses over the top k most likely predictions.
         :param k_to_report: Will report the top k result for the k in this list.
-        :param aggregators: How to aggregate the results across the data set.
+        :param aggregators: How to aggregate the results across the dataset.
         :param tolerance: Threshold to consider if a prediction is a hit or not.
         """
         self.k_to_report = k_to_report
@@ -318,7 +318,7 @@ def __init__(self, helper: PredictHelper, aggregators: List[Aggregator]):
         The OffRoadRate is defined as the fraction of trajectories that are not entirely contained
         in the drivable area of the map.
         :param helper: Instance of PredictHelper. Used to determine the map version for each prediction.
-        :param aggregators: How to aggregate the results across the data set.
+        :param aggregators: How to aggregate the results across the dataset.
         """
         self._aggregators = aggregators
         self.helper = helper

python-sdk/nuscenes/eval/prediction/splits.py

@@ -15,7 +15,7 @@ def get_prediction_challenge_split(split: str, dataroot: str = '/data/sets/nusce
     """
     Gets a list of {instance_token}_{sample_token} strings for each split.
     :param split: One of 'mini_train', 'mini_val', 'train', 'val'.
-    :param dataroot: Path to the nuScenes data set.
+    :param dataroot: Path to the nuScenes dataset.
     :return: List of tokens belonging to the split. Format {instance_token}_{sample_token}.
     """
     if split not in {'mini_train', 'mini_val', 'train', 'train_val', 'val'}:
@@ -26,7 +26,7 @@ def get_prediction_challenge_split(split: str, dataroot: str = '/data/sets/nusce
     else:
         split_name = split
 
-    path_to_file = os.path.join(dataroot, "maps", "prediction_scenes.json")
+    path_to_file = os.path.join(dataroot, "maps", "prediction", "prediction_scenes.json")
     prediction_scenes = json.load(open(path_to_file, "r"))
     scenes = create_splits_scenes()
     scenes_for_split = scenes[split_name]

python-sdk/nuscenes/map_expansion/bitmap.py

@@ -0,0 +1,75 @@
+import os
+from typing import Tuple, Any
+
+import numpy as np
+from PIL import Image
+import matplotlib.pyplot as plt
+
+Axis = Any
+
+
+class BitMap:
+
+    def __init__(self, dataroot: str, map_name: str, layer_name: str):
+        """
+        This class is used to render bitmap map layers. Currently these are:
+        - semantic_prior: The semantic prior (driveable surface and sidewalks) mask from nuScenes 1.0.
+        - basemap: The HD lidar basemap used for localization and as general context.
+
+        :param dataroot: Path of the nuScenes dataset.
+        :param map_name: Which map out of `singapore-onenorth`, `singepore-hollandvillage`, `singapore-queenstown` and
+            'boston-seaport'.
+        :param layer_name: The type of bitmap map, `semanitc_prior` or `basemap.
+        """
+        self.dataroot = dataroot
+        self.map_name = map_name
+        self.layer_name = layer_name
+
+        self.image = self.load_bitmap()
+
+    def load_bitmap(self) -> np.ndarray:
+        """
+        Load the specified bitmap.
+        """
+        # Load bitmap.
+        if self.layer_name == 'basemap':
+            map_path = os.path.join(self.dataroot, 'maps', 'basemap', self.map_name + '.png')
+        elif self.layer_name == 'semantic_prior':
+            map_hashes = {
+                'singapore-onenorth': '53992ee3023e5494b90c316c183be829',
+                'singapore-hollandvillage': '37819e65e09e5547b8a3ceaefba56bb2',
+                'singapore-queenstown': '93406b464a165eaba6d9de76ca09f5da',
+                'boston-seaport': '36092f0b03a857c6a3403e25b4b7aab3'
+            }
+            map_hash = map_hashes[self.map_name]
+            map_path = os.path.join(self.dataroot, 'maps', map_hash + '.png')
+        else:
+            raise Exception('Error: Invalid bitmap layer: %s' % self.layer_name)
+
+        # Convert to numpy.
+        if os.path.exists(map_path):
+            image = np.array(Image.open(map_path))
+        else:
+            raise Exception('Error: Cannot find %s %s! Please make sure that the map is correctly installed.'
+                            % (self.layer_name, map_path))
+
+        # Invert semantic prior colors.
+        if self.layer_name == 'semantic_prior':
+            image = image.max() - image
+
+        return image
+
+    def render(self, canvas_edge: Tuple[float, float], ax: Axis = None):
+        """
+        Render the bitmap.
+        Note: Regardless of the image dimensions, the image will be rendered to occupy the entire map.
+        :param canvas_edge: The dimension of the current map in meters (width, height).
+        :param ax: Optional axis to render to.
+        """
+        if ax is None:
+            ax = plt.subplot()
+        x, y = canvas_edge
+        if len(self.image.shape) == 2:
+            ax.imshow(self.image, extent=[0, x, 0, y], cmap='gray')
+        else:
+            ax.imshow(self.image, extent=[0, x, 0, y])