In nuScenes-lidarseg, we annotate every point in the lidar pointcloud with a semantic label. All the labels from nuScenes are carried over into nuScenes-lidarseg; in addition, more "stuff" (background) classes have been included. Thus, nuScenes-lidarseg contains both foreground classes (pedestrians, vehicles, cyclists, etc.) and background classes (driveable surface, nature, buildings, etc.).
- Label each point with a class.
- Use the camera images to facilitate, check and validate the labels.
- Each point belongs to only one class, i.e., one class per point.
- Extremities such as vehicle doors, car mirrors and human limbs should be assigned the same label as the object. Note that in contrast to the nuScenes 3d cuboids, the lidarseg labels include car mirrors and antennas.
- Minimum number of points
- An object can have as little as one point.
In such cases, that point should only be labeled if it is certain that the point belongs to a class
(with additional verification by looking at the corresponding camera frame).
Otherwise, the point should be labeled as
static.other.
- An object can have as little as one point.
In such cases, that point should only be labeled if it is certain that the point belongs to a class
(with additional verification by looking at the corresponding camera frame).
Otherwise, the point should be labeled as
- Other static object vs noise.
- Other static object: Points that belong to some physical object, but are not defined in our taxonomy.
- Noise: Points that do not correspond to physical objects or surfaces in the environment (e.g. noise, reflections, dust, fog, raindrops or smoke).
- Terrain vs other flat.
- Terrain: Grass, all kinds of horizontal vegetation, soil or sand. These areas are not meant to be driven on. This label includes a possibly delimiting curb. Single grass stalks do not need to be annotated and get the label of the region they are growing on.
- Short bushes / grass with heights of less than 20cm, should be labeled as terrain. Similarly, tall bushes / grass which are higher than 20cm should be labeled as vegetation.
- Other flat: Horizontal surfaces which cannot be classified as ground plane / sidewalk / terrain, e.g., water.
- Terrain vs sidewalk
- Terrain: See above.
- Sidewalk: A sidewalk is a walkway designed for pedestrians and / or cyclists. Sidewalks are always paved.
The following classes are in addition to the existing ones in nuScenes:
| Label ID | Label | Short Description |
|---|---|---|
| 0 | noise |
Any lidar return that does not correspond to a physical object, such as dust, vapor, noise, fog, raindrops, smoke and reflections. |
| 24 | flat.driveable_surface |
All paved or unpaved surfaces that a car can drive on with no concern of traffic rules. |
| 25 | flat.sidewalk |
Sidewalk, pedestrian walkways, bike paths, etc. Part of the ground designated for pedestrians or cyclists. Sidewalks do not have to be next to a road. |
| 26 | flat.terrain |
Natural horizontal surfaces such as ground level horizontal vegetation (< 20 cm tall), grass, rolling hills, soil, sand and gravel. |
| 27 | flat.other |
All other forms of horizontal ground-level structures that do not belong to any of driveable_surface, curb, sidewalk and terrain. Includes elevated parts of traffic islands, delimiters, rail tracks, stairs with at most 3 steps and larger bodies of water (lakes, rivers). |
| 28 | static.manmade |
Includes man-made structures but not limited to: buildings, walls, guard rails, fences, poles, drainages, hydrants, flags, banners, street signs, electric circuit boxes, traffic lights, parking meters and stairs with more than 3 steps. |
| 29 | static.vegetation |
Any vegetation in the frame that is higher than the ground, including bushes, plants, potted plants, trees, etc. Only tall grass (> 20cm) is part of this, ground level grass is part of flat.terrain. |
| 30 | static.other |
Points in the background that are not distinguishable. Or objects that do not match any of the above labels. |
| 31 | vehicle.ego |
The vehicle on which the cameras, radar and lidar are mounted, that is sometimes visible at the bottom of the image. |
Below are examples of the classes added in nuScenes-lidarseg. For simplicity, we only show lidar points which are relevant to the class being discussed.
Points on the ego vehicle generally arise due to self-occlusion, in which some lidar beams hit the ego vehicle.
When the pointcloud is projected into a chosen camera image, the devkit removes points which are less than
1m in front of the camera to prevent such points from cluttering the image. Thus, users will not see points
belonging to vehicle.ego projected onto the camera images when using the devkit. To give examples, of the
vehicle.ego class, the bird's eye view (BEV) is used instead:
