Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eval refactor step2 #53

Merged
merged 26 commits into from
Mar 14, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
edf0b94
reorganized repo to more allow for easier injection of tests. Moved d…
oscar-nutonomy Mar 6, 2019
b696a06
added README, minor renames and cleanup
oscar-nutonomy Mar 6, 2019
cd99b1f
Merge remote-tracking branch 'origin/release_v0.2' into eval_refactor
oscar-nutonomy Mar 6, 2019
548df59
removed v0.1 from the nuscenes asserts
oscar-nutonomy Mar 6, 2019
b8a41df
renamed main detection eval file
oscar-nutonomy Mar 6, 2019
e4b7120
fixed typo
oscar-nutonomy Mar 6, 2019
3d0d01a
Added verify and test instructions to main README
oscar-nutonomy Mar 6, 2019
8efede2
added more thorough regresssion type test. This seem to have triggere…
oscar-nutonomy Mar 6, 2019
6d76c3d
Update README.md
oscar-nutonomy Mar 7, 2019
3fe86dd
Minor cleanups, added more descriptive error messages
holger-motional Mar 7, 2019
d379353
Standardized dataroot to /data/sets/nuscenes
holger-motional Mar 7, 2019
873b30b
Catch case when obligatory attributes are missing, changed attribute_…
holger-motional Mar 7, 2019
e2e4d01
Merge branch 'release_v0.2' into eval_refactor_step2
holger-motional Mar 7, 2019
64d9a7d
fixed random seed. Passes unit-tests now
oscar-nutonomy Mar 8, 2019
f73fcfe
started re-factor
oscar-nutonomy Mar 8, 2019
9ddd59e
Added data-classes for the EvalBoxes. Cleaned up loading. Same result…
oscar-nutonomy Mar 8, 2019
481488a
updated all tests to use the EvalBox data class. All tests passing now.
oscar-nutonomy Mar 8, 2019
4cfe6c1
More cleanup. Still no changes to algorithm
oscar-nutonomy Mar 9, 2019
025c93b
changes to calculating distance from ego_vehicle (#57)
oscar-nutonomy Mar 9, 2019
e33051e
Typing and comments
holger-motional Mar 9, 2019
d21123b
Merge branch 'eval_refactor_step2' of github.com:nutonomy/nuscenes-de…
holger-motional Mar 9, 2019
273ff7a
Changed result format to accept at most one attribute (#60)
holger-motional Mar 11, 2019
ab1b69a
minor attr_acc cleanup
oscar-nutonomy Mar 11, 2019
e779114
reviewer comments
oscar-nutonomy Mar 11, 2019
fb9c4dc
merged conflice with v0.2 branch
oscar-nutonomy Mar 14, 2019
7a9a4d5
final conflict and reviewer comments
oscar-nutonomy Mar 14, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
changes to calculating distance from ego_vehicle (#57)
  • Loading branch information
oscar-nutonomy authored Mar 9, 2019
commit 025c93b9411792d220e471d552af4d30fdce4c3c
2 changes: 1 addition & 1 deletion python-sdk/nuscenes/eval/detection/data_classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ def __getitem__(self, item):
return self.boxes[item]

@property
def all(self):
def all(self) -> List[EvalBox]:
ab = []
for sample_token in self.sample_tokens:
ab.extend(self[sample_token])
Expand Down
38 changes: 9 additions & 29 deletions python-sdk/nuscenes/eval/detection/loaders.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
import json
from typing import List, Dict
import tqdm

import numpy as np
import tqdm

from nuscenes.eval.detection.data_classes import EvalBoxes, EvalBox
from nuscenes.eval.detection.utils import category_to_detection_name, boxes_to_sensor
from nuscenes.eval.detection.utils import category_to_detection_name
from nuscenes.utils.splits import create_splits_scenes


Expand Down Expand Up @@ -81,46 +81,26 @@ def load_gt(nusc, eval_split, cfg) -> EvalBoxes:


def add_center_dist(nusc, eval_boxes: EvalBoxes):
""" Appends the center distance from ego vehicle to each box. """
""" Adds the center distance from ego vehicle to each box. """

for sample_token in eval_boxes.sample_tokens:
sample_rec = nusc.get('sample', sample_token)
sd_record = nusc.get('sample_data', sample_rec['data']['LIDAR_TOP'])
cs_record = nusc.get('calibrated_sensor', sd_record['calibrated_sensor_token'])
pose_record = nusc.get('ego_pose', sd_record['ego_pose_token'])

eval_boxes.boxes[sample_token] = append_ego_dist_tmp(eval_boxes.boxes[sample_token], pose_record, cs_record)
for box in eval_boxes[sample_token]:
# Both boxes and ego pose are given in global coord system, so distance can be calculated directly.
diff = np.array(pose_record['translation'][:2]) - np.array(box.translation[:2])
box.ego_dist = np.sqrt(np.sum(diff ** 2))

return eval_boxes


def append_ego_dist_tmp(sample_boxes: List[EvalBox], pose_record: Dict, cs_record: Dict) -> List[EvalBox]:
"""
Removes all boxes that are not within the valid eval_range of the LIDAR.
:param sample_boxes: A list of sample_annotation OR sample_result entries.
:param pose_record: An ego_pose entry stored as a dict.
:param cs_record: A calibrated_sensor entry stored as a dict.
:param eval_range: Range in meters beyond which boxes are ignored.
:return: The filtered sample_boxes and their distances to the sensor.
"""
# TODO: remove this whole method. Once we move to the distance to ego vehicle frame, this becomes trivial.
# Moved boxes to lidar coordinate frame
sample_boxes_sensor = boxes_to_sensor(sample_boxes, pose_record, cs_record)

# Filter boxes outside the relevant area.
result = []
for box_sensor, box_global in zip(sample_boxes_sensor, sample_boxes):
box_global.ego_dist = np.sqrt(np.sum(box_sensor.center[:2] ** 2))
result.append(box_global) # Add the sample_box, not the box.

return result


def filter_eval_boxes(nusc, eval_boxes: EvalBoxes, max_dist: float):
""" Applies filtering to boxes. Distance, bike-racks and point per box. """

# TODO: add the other filtering here
for sample_token in eval_boxes.sample_tokens:
eval_boxes.boxes[sample_token] = [box for box in eval_boxes[sample_token] if box.ego_dist < max_dist]

return eval_boxes
return eval_boxes
3 changes: 2 additions & 1 deletion python-sdk/nuscenes/eval/detection/tests/test_main.py
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,8 @@ def test_delta(self):
metrics = nusc_eval.run()

# Score of 0.22082865720221012 was measured on the branch "release_v0.2" on March 7 2019.
self.assertAlmostEqual(metrics['weighted_sum'], 0.22082865720221012)
# After changing to measure center distance from the ego-vehicle this changed to 0.2199307290627096
self.assertAlmostEqual(metrics['weighted_sum'], 0.2199307290627096)


if __name__ == '__main__':
Expand Down