Eval refactor step2 #53

oscar-nutonomy · 2019-03-06T23:15:26Z

This pull-requests re-organizes the detection eval code. There are no changes to the algorithm or the output. Changes include:

All evaluation configs are captured in a configuration file.
Added dataclasses for EvalBoxes and EvalConfigs.
The main class is much leaner. Loading happens in loaders.py and the main algorithms are in algo.py.
Misc clean-ups through using the power of the data-classes.
Renamed main variables to gt_boxes and pred_boxes (from all_annotations' and all_results`).

There is a bunch of stuff remaining, but I think this is a fine chunk to review. Next steps include

Add some more methods on the data-classes.
Add data-classes for all_metrics and raw_metrics
Store raw_metrics to disk.
Implement rendering method that operations on raw_metrics data-class.

I commented out the CLI to main for now until we know what it will look like. I also commented out the in-line plotting in average_precision and calc_tp_metrics. As mentioned above, once we can store raw_metrics to disk we can build new renderers.

…etection eval code to eval/detection subfolder

…d an error in the evaluation code

…labels to be bools, typos

…s verified on test_delta()

…vkit into eval_refactor_step2

python-sdk/nuscenes/eval/detection/data_classes.py

holger-motional · 2019-03-09T03:00:37Z

python-sdk/nuscenes/eval/detection/loaders.py

+                    rotation=sample_annotation['rotation'],
+                    velocity=list(nusc.box_velocity(sample_annotation['token'])),
+                    detection_name=detection_name,
+                    detection_score=0,


Perhaps nan or None for GT boxes?

python-sdk/nuscenes/eval/detection/utils.py

* Replaced attribute_scores by a single attribute_name * Fixed attribute bug * Updated weighted sum score * Added a back-port for test fixture to make test reproducible.

varun-nutonomy

Looks good to me, left some very minor comments inline.

varun-nutonomy · 2019-03-11T17:21:13Z

README.md

@@ -56,6 +56,10 @@ Finally, set NUSCENES env. variable that points to your data folder
 export NUSCENES="/data/sets/nuscenes"
 ```

+### Verify install
+To verify your environment run `python -m unittest` in the `python-sdk` folder. 
+You can also run `assert_download.py` in the `nuscenes/scrips` folder.


Typo: scripts.

varun-nutonomy · 2019-03-11T17:21:42Z

python-sdk/nuscenes/eval/detection/algo.py

+    """
+
+    # Count the positives.
+    npos = len([1 for gt_box in gt_boxes.all if gt_box.detection_name == class_name])


Can be replaced with
npos = sum(gt_box.detection_name == class_name for gt_box in gt_boxes.all)

varun-nutonomy · 2019-03-11T17:22:37Z

python-sdk/nuscenes/eval/detection/algo.py

+    pred_confs = [box.detection_score for box in pred_boxes_list]
+
+    # Sort by confidence.
+    sortind = [i for (v, i) in sorted((v, i) for (i, v) in enumerate(pred_confs))][::-1]


Can be replaced with

# Sort by confidence in descending order. sortind = np.argsort(pred_confs)[::-1]

varun-nutonomy · 2019-03-13T17:18:55Z

python-sdk/nuscenes/eval/detection/algo.py

+    if len(raw_metrics) == 0:
+        return tp_metrics
+
+    for metric_name in {key: [] for key in cfg.metric_names}:


I don't understand why it is written like this? Why not just for metric_name in cfg.metric_names: ?

varun-nutonomy · 2019-03-13T17:35:31Z

python-sdk/nuscenes/utils/map_mask.py

@@ -58,7 +58,7 @@ def is_on_mask(self, x, y, dilation=0) -> np.array:
        Determine whether the given coordinates are on the (optionally dilated) map mask.
        :param x: Global x coordinates. Can be a scalar, list or a numpy array of x coordinates.
        :param y: Global y coordinates. Can be a scalar, list or a numpy array of x coordinates.
-        :param dilation: <float>. Optional dilation of map mask.
+        :param dilation: Optional dilation of map mask.


Missing typing for this method and the next one.

holger-motional

Looks good.

* Added setup.py for pip installation * Added speedup to query many points with single function call in map mask * Added missing methods for improved map mask * Changes for v0.2 * Updated tutorial path * Changed logic s.t. map table points to logs, not 1 to 1 * Fix indexing bug * Reset paths to default * Removed print statement from setup.py * Fixed a bug for large map files * Not excluding any logs from map rendering * Fixed bug with missing "self" keyword in create_splits_logs(nusc), added dataroot parameter * Address bug for classes without attributes * Fixed misspelled motorcycle * Added missing keyword self * Added checks for nan values * Removed teaser splits as they is not compatible with the new logs * Added (preliminary) splits for v0.2 * Workaround to allow evaluating a subset of the data * Splits hardcoded in create_splits_logs * Fix bugs if a results file has zero TPs * Simpler MapMask (#31) * deleted distance_mask, binary_mask. Created dilate_mask(). * cleanup docstring * Added missing Dict import (#35) * Typing fix (#36) * Added missing Dict import * Fixed typing to be compatible where future annotations package cannot be loaded * Updated code to new folder structure (#37) * Devkit now accepts versions 0.3, 0.4 and 1.0 * Fix TPs analog to commit 94544b5 (#41) * Fix yaw_diff calculation (#46) * Added unittest to catch previous error in yaw_diff * Updated dataset to v0.4, updated train/val/test splits * Added assertion * Cleaned up map API. * Evaluation Fixes (#51) * Correct recall edge case * Apply distance filtering to xy only * Return max error if recall is too low * Added dataset version as command line argument to nuscenes_eval.py * Function to get scenes in each split, script to check dataset completeness * Added command line arguments to launch scripts * Hotfix for call to create_splits_logs * Structure re-org towards more tests. (#52) - Moves files around to enable easier inject of tests. - Enables running unit-tests per usual python -m unittest - Requires env. variable $NUSCENES for test. - Added trivial unit-tests for NuScenes & NuScenesEval. Currently uses v0.2. We should switch to mini when data is ready. - Moved detection eval code to eval/detection to future-proof for next task. This PR is only the first step. The next PR will break out the detection eval code to enable testing. * Updated dataset splits to v1.0 * Updated version to 1.0 and database folder to /data/sets/nuscenes * Added explanation that visibility token may be empty * Style update (#59) * Changed format of data_classes.py and avoided default parameters in RadarPointCloud * Typing and line breaks in long method signatures * Reset dynprop_states to previous settings * Spacing and line breaks * Spacing * Break circular import dependencies * More flexible setup.py * Cleanup setup.py * Updated radar comments * fix setup to include root folder * Backporting bugfix from 873b30b until eval pr is merged * new pip package * Workaround to allow version string to include the split, e.g. v1.0-trainval * Mini splits and modified split logic (#63) * Added mini splits and updated overall logic, note: changed method signatures * Added an assertion * Docstrings * Fixed typo * Updated eval code to new splits function * Eval refactor step2 (#53) * reorganized repo to more allow for easier injection of tests. Moved detection eval code to eval/detection subfolder * Added verify and test instructions to main README * Added more thorough regression type test. This seem to have triggered an error in the evaluation code * Standardized dataroot to /data/sets/nuscenes * Fixed random seed. Passes unit-tests now * Added data-classes for the EvalBoxes. Cleaned up loading. Same results verified on test_delta() * Updated all tests to use the EvalBox data class. All tests passing now. * Changes to calculating distance from ego_vehicle (#57) * Changed result format to accept at most one attribute (#60) * Replaced attribute_scores by a single attribute_name * v0.2 CI preparation. (#65) * updated to 1.0-mini split for all tests. Skipped those that require DB * Misc Reorg (#68) * reorganize geometry code * add shortcuts to common classes * move quaternion yaw back * change default nuscenes to mini * Integrate mini_split into continuous integration (#71) * Enabled v1.0-mini to be used in CI for unit-tests. * Updated detection metrics code (#66) * Reorganized eval code to more allow for easier injection of tests. * Make nuScenes compatible to Python3.5/Python3.6 (#70) * Removed python 3.6 and 3.7 only features to make compatible with 3.5 * nuscenes-devkit pip package v0.3.4 * Added num_lidar_points and num_radar_points to schema * Added instructions to include lidar and radar points * update create_splits_logs to new database versions (#74) * Some unittests for algo.py and util.py (#76) * Added unittest for utils.py and algo.py * Repeat unit tests for Python 3.6, 3.7 + bug fix for false negative (#77) * Fixed false negative issue. Added support for 3.6 and 3.7. * Create a configuration factory (#78) * create a configuration factory * add unittests * fix the unittests * added token field to Box class (#79) * added token field to Box class * fixed line-break * Update README installation instructions. (#75) * Cleaned up README. Changed to support for python 3.6 and 3.7. * added change log message for release * * Moved continuous integration and Docker dependencies to 'setup' folder. (#80) * Updated readme files (#81) * Added setup/instrucions.md, updated Debris class definition, tuned main README * update setup.py to new repo structure * Update tutorial.ipynb * Simple Unittests for AP calculation (#82) * Clean up old unit tests and added tests for AP calculation * Revert setup changes to get CI working (#84) * Revert "* Moved continuous integration and Docker dependencies to 'setup' folder. (#80)" This reverts commit fa55433. * revert setup.py changes * Bikerack filtering (#83) * spellcheck * add points_in_box() method to check whether points are inside the box * test points_in_box() method * perform bike rack filtering * docstring for filter_eval_boxes * added unittests for filter_eval_boxes() * update comment * fix docstring * can't use Box type because of a circular import. So switched to 'Box' * can't use Box type because of a circular import. So switched to 'Box' * change threshold after bikerack filtering * get max_dist from eval_detection_configs * Update eval renderers (#85) * Updated renderers for detection results. * Fixed bugs in detection evaluation. * Added color options for render_egoposes_on_map (#86) * add color option for render maps * Summary AP plot update (#87) * updated the plots * better legend text alignment * updated legend positions and pretty names for attributes and TP metrics * Tex table (#91) * added reder tex table * cleaned up tex render method * minor changes * add credits * misc cleanup * Instruction updates and misc (#89) * Set default version for export egoposes, better outputs * Added v1.0 to legal datasets * Corrected visibility level * Set face color to black, better status messages * Updated face color * Set set to trainval in misc scripts * Bugfix to preserver black bg color * Made close_dist a parameter * Merged truck classes in the instructions * Removed train from instructions * Merged police officers into single class * Reset changes to tutorial * Updated path to requirements.txt * Removed v1.0 which was used for setup purposes * Cleanup * Fix visibility levels in instructions * Also print empty categories * Fixed bug * Reversed previous changes * Updated schema comments for map * Reverted previous changes * misc cleanup * next pip package * added unittests for TP metrics * Readme update (#93) * OVerhauled eval detection page * Overhauled main readme * Removed deprecated links * Rewording * Addressed review comments * fixed the visualize sample, need to test it * added comments to unit tests * [WIP] Jupyter notebook tutorial (#90) * Updated tutorial * Unittests for true positive error metrics calculation (#92) * added unittests for TP metrics * fixed visualize sample * added comments to unit tests * added option to save sample rendering to disk * added back showing plot if savepath is not given * Update README.md Clarifications * Update schema.md added an example for subcategories * update dates * cleanup imports, fix readme typos, cleaner scripts * cleanup evaluate * fix splits * update unittest to split logic * update test delta

oscar-nutonomy added 8 commits March 6, 2019 12:25

reorganized repo to more allow for easier injection of tests. Moved d…

edf0b94

…etection eval code to eval/detection subfolder

added README, minor renames and cleanup

b696a06

Merge remote-tracking branch 'origin/release_v0.2' into eval_refactor

cd99b1f

removed v0.1 from the nuscenes asserts

548df59

renamed main detection eval file

b8a41df

fixed typo

e4b7120

Added verify and test instructions to main README

3d0d01a

added more thorough regresssion type test. This seem to have triggere…

8efede2

…d an error in the evaluation code

oscar-nutonomy self-assigned this Mar 6, 2019

oscar-nutonomy requested a review from holger-motional March 6, 2019 23:15

oscar-nutonomy and others added 2 commits March 6, 2019 18:20

Update README.md

6d76c3d

Minor cleanups, added more descriptive error messages

3fe86dd

oscar-nutonomy changed the base branch from eval_refactor to release_v0.2 March 7, 2019 05:45

holger-motional added 2 commits March 7, 2019 16:24

Standardized dataroot to /data/sets/nuscenes

d379353

Catch case when obligatory attributes are missing, changed attribute_…

873b30b

…labels to be bools, typos

This comment has been minimized.

Sign in to view

holger-motional and others added 6 commits March 7, 2019 17:32

Merge branch 'release_v0.2' into eval_refactor_step2

e2e4d01

fixed random seed. Passes unit-tests now

64d9a7d

started re-factor

f73fcfe

Added data-classes for the EvalBoxes. Cleaned up loading. Same result…

9ddd59e

…s verified on test_delta()

updated all tests to use the EvalBox data class. All tests passing now.

481488a

More cleanup. Still no changes to algorithm

4cfe6c1

oscar-nutonomy requested review from Alex-nutonomy, sourabh-nutonomy and varun-nutonomy March 9, 2019 02:13

oscar-nutonomy changed the title ~~[WIP] Eval refactor step2~~ Eval refactor step2 Mar 9, 2019

oscar-nutonomy and others added 3 commits March 8, 2019 18:58

changes to calculating distance from ego_vehicle (#57)

025c93b

Typing and comments

e33051e

Merge branch 'eval_refactor_step2' of github.com:nutonomy/nuscenes-de…

d21123b

…vkit into eval_refactor_step2

holger-motional reviewed Mar 9, 2019

View reviewed changes

holger-motional and others added 3 commits March 11, 2019 10:37

Changed result format to accept at most one attribute (#60)

273ff7a

* Replaced attribute_scores by a single attribute_name * Fixed attribute bug * Updated weighted sum score * Added a back-port for test fixture to make test reproducible.

minor attr_acc cleanup

ab1b69a

reviewer comments

e779114

varun-nutonomy reviewed Mar 13, 2019

View reviewed changes

holger-motional approved these changes Mar 14, 2019

View reviewed changes

oscar-nutonomy added 2 commits March 13, 2019 19:19

merged conflice with v0.2 branch

fb9c4dc

final conflict and reviewer comments

7a9a4d5

oscar-nutonomy merged commit 54edf61 into release_v0.2 Mar 14, 2019

oscar-nutonomy deleted the eval_refactor_step2 branch March 14, 2019 02:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eval refactor step2 #53

Eval refactor step2 #53

oscar-nutonomy commented Mar 6, 2019 •

edited by holger-motional

Loading

This comment has been minimized.

holger-motional Mar 9, 2019

varun-nutonomy left a comment

varun-nutonomy Mar 11, 2019

varun-nutonomy Mar 11, 2019

varun-nutonomy Mar 11, 2019

varun-nutonomy Mar 13, 2019

varun-nutonomy Mar 13, 2019

holger-motional left a comment

Eval refactor step2 #53

Eval refactor step2 #53

Conversation

oscar-nutonomy commented Mar 6, 2019 • edited by holger-motional Loading

This comment has been minimized.

holger-motional Mar 9, 2019

Choose a reason for hiding this comment

varun-nutonomy left a comment

Choose a reason for hiding this comment

varun-nutonomy Mar 11, 2019

Choose a reason for hiding this comment

varun-nutonomy Mar 11, 2019

Choose a reason for hiding this comment

varun-nutonomy Mar 11, 2019

Choose a reason for hiding this comment

varun-nutonomy Mar 13, 2019

Choose a reason for hiding this comment

varun-nutonomy Mar 13, 2019

Choose a reason for hiding this comment

holger-motional left a comment

Choose a reason for hiding this comment

oscar-nutonomy commented Mar 6, 2019 •

edited by holger-motional

Loading