merging upstream here

README.md

@@ -7,25 +7,20 @@ We extend the original [SORT](https://github.com/abewley/sort) algorithm to
 integrate appearance information based on a deep appearance descriptor.
 See the [arXiv preprint](https://arxiv.org/abs/1703.07402) for more information.
 
-## Dependencies
-
-The code is compatible with Python 2.7 and 3. The following dependencies are
-needed to run the tracker:
-
-* NumPy
-* sklean
-* OpenCV
-
-Additionally, feature generation requires TensorFlow (>= 1.0).
-
 ## Installation
 
-First, clone the repository:
+First, clone the repository and install dependencies:
 ```
 git clone https://github.com/nwojke/deep_sort.git
+cd deep_sort
+
+# The following command installs all the dependencies required to run the
+# tracker and regenerate detections. If you only need to run the tracker with
+# existing detections, you can use pip install -r requirements.txt instead.
+pip install -r requirements-gpu.txt
 ```
 Then, download pre-generated detections and the CNN checkpoint file from
-[here](https://owncloud.uni-koblenz.de/owncloud/s/f9JB0Jr7f3zzqs8).
+[here](https://drive.google.com/open?id=18fKzfqnqhqW3s9zwsCbnVJ5XF2JFeqMp).
 
 *NOTE:* The candidate object locations of our pre-generated detections are
 taken from the following paper:
@@ -46,7 +41,7 @@ We assume resources have been extracted to the repository root directory and
 the MOT16 benchmark data is in `./MOT16`:
 ```
 python deep_sort_app.py \
-    --sequence_dir=./MOT16/test/MOT16-06
+    --sequence_dir=./MOT16/test/MOT16-06 \
     --detection_file=./resources/detections/MOT16_POI_test/MOT16-06.npy \
     --min_confidence=0.3 \
     --nn_budget=100 \
@@ -65,19 +60,33 @@ The following example generates these features from standard MOT challenge
 detections. Again, we assume resources have been extracted to the repository
 root directory and MOT16 data is in `./MOT16`:
 ```
-python generate_detections.py
-    --model=resources/networks/mars-small128.ckpt \
+python tools/generate_detections.py \
+    --model=resources/networks/mars-small128.pb \
     --mot_dir=./MOT16/train \
     --output_dir=./resources/detections/MOT16_train
 ```
-For each sequence of the MOT16 dataset, the output is stored as separate binary
-file in NumPy native format. Each file contains an array of shape `Nx138`,
-where N is the number of detections in the corresponding MOT sequence.
-The first 10 columns of this array contain the raw MOT detection copied over
-from the input file. The remaining 128 columns store the appearance descriptor.
-The files generated by this command can be used as input for the
+The model has been generated with TensorFlow 1.5. If you run into
+incompatibility, re-export the frozen inference graph to obtain a new
+`mars-small128.pb` that is compatible with your version:
+```
+python tools/freeze_model.py
+```
+The ``generate_detections.py`` stores for each sequence of the MOT16 dataset
+a separate binary file in NumPy native format. Each file contains an array of
+shape `Nx138`, where N is the number of detections in the corresponding MOT
+sequence. The first 10 columns of this array contain the raw MOT detection
+copied over from the input file. The remaining 128 columns store the appearance
+descriptor. The files generated by this command can be used as input for the
 `deep_sort_app.py`.
 
+**NOTE**: If ``python tools/generate_detections.py`` raises a TensorFlow error,
+try passing an absolute path to the ``--model`` argument. This might help in
+some cases.
+
+## Training the model
+
+To train the deep association metric model we used a novel [cosine metric learning](https://github.com/nwojke/cosine_metric_learning) approach which is provided as a separate repository.
+
 ## Highlevel overview of source files
 
 In the top-level directory are executable scripts to execute, evaluate, and
@@ -99,7 +108,8 @@ In package `deep_sort` is the main tracking code:
 
 The `deep_sort_app.py` expects detections in a custom format, stored in .npy
 files. These can be computed from MOTChallenge detections using
-`generate_detections.py`. We also provide [pre-generated detections](https://owncloud.uni-koblenz.de/owncloud/s/f9JB0Jr7f3zzqs8).
+`generate_detections.py`. We also provide
+[pre-generated detections](https://drive.google.com/open?id=1VVqtL0klSUvLnmBKS89il1EKC3IxUBVK).
 
 ## Citing DeepSORT
 
@@ -110,14 +120,17 @@ If you find this repo useful in your research, please consider citing the follow
       author={Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich},
       booktitle={2017 IEEE International Conference on Image Processing (ICIP)},
       year={2017},
-      pages={3645--3649}
+      pages={3645--3649},
+      organization={IEEE},
+      doi={10.1109/ICIP.2017.8296962}
     }
 
-    @inproceedings{Bewley2016_sort,
-      author={Bewley, Alex and Ge, Zongyuan and Ott, Lionel and Ramos, Fabio and Upcroft, Ben},
-      booktitle={2016 IEEE International Conference on Image Processing (ICIP)},
-      title={Simple online and realtime tracking},
-      year={2016},
-      pages={3464-3468},
-      doi={10.1109/ICIP.2016.7533003}
+    @inproceedings{Wojke2018deep,
+      title={Deep Cosine Metric Learning for Person Re-identification},
+      author={Wojke, Nicolai and Bewley, Alex},
+      booktitle={2018 IEEE Winter Conference on Applications of Computer Vision (WACV)},
+      year={2018},
+      pages={748--756},
+      organization={IEEE},
+      doi={10.1109/WACV.2018.00087}
     }

application_util/image_viewer.py

@@ -252,7 +252,7 @@ def colored_points(self, points, colors=None, skip_index_check=False):
         if colors is None:
             colors = np.repeat(
                 self._color, len(points)).reshape(3, len(points)).T
-        indices = (points + .5).astype(np.int)
+        indices = (points + .5).astype(np.int64)
         self.image[indices[:, 1], indices[:, 0], :] = colors
 
     def enable_videowriter(self, output_filename, fourcc_string="MJPG",

application_util/preprocessing.py

@@ -37,7 +37,7 @@ def non_max_suppression(boxes, max_bbox_overlap, scores=None):
     if len(boxes) == 0:
         return []
 
-    boxes = boxes.astype(np.float)
+    boxes = boxes.astype(np.float64)
     pick = []
 
     x1 = boxes[:, 0]

application_util/visualization.py

@@ -113,7 +113,7 @@ def draw_groundtruth(self, track_ids, boxes):
         self.viewer.thickness = 2
         for track_id, box in zip(track_ids, boxes):
             self.viewer.color = create_unique_color_uchar(track_id)
-            self.viewer.rectangle(*box.astype(np.int), label=str(track_id))
+            self.viewer.rectangle(*box.astype(np.int64), label=str(track_id))
 
     def draw_detections(self, detections):
         self.viewer.thickness = 2
@@ -128,7 +128,7 @@ def draw_trackers(self, tracks):
                 continue
             self.viewer.color = create_unique_color_uchar(track.track_id)
             self.viewer.rectangle(
-                *track.to_tlwh().astype(np.int), label=str(track.track_id))
+                *track.to_tlwh().astype(np.int64), label=str(track.track_id))
             # self.viewer.gaussian(track.mean[:2], track.covariance[:2, :2],
             #                      label="%d" % track.track_id)
 #

deep_sort/detection.py

@@ -27,7 +27,7 @@ class Detection(object):
     """
 
     def __init__(self, tlwh, confidence, feature):
-        self.tlwh = np.asarray(tlwh, dtype=np.float)
+        self.tlwh = np.asarray(tlwh, dtype=np.float64)
         self.confidence = float(confidence)
         self.feature = np.asarray(feature, dtype=np.float32)
 

deep_sort/linear_assignment.py

@@ -1,7 +1,7 @@
 # vim: expandtab:ts=4:sw=4
 from __future__ import absolute_import
 import numpy as np
-from sklearn.utils.linear_assignment_ import linear_assignment
+from scipy.optimize import linear_sum_assignment
 from . import kalman_filter
 
 
@@ -55,7 +55,7 @@ def min_cost_matching(
     cost_matrix = distance_metric(
         tracks, detections, track_indices, detection_indices)
     cost_matrix[cost_matrix > max_distance] = max_distance + 1e-5
-    indices = linear_assignment(cost_matrix)
+    indices = np.asarray(linear_sum_assignment(cost_matrix)).T
 
     matches, unmatched_tracks, unmatched_detections = [], [], []
     for col, detection_idx in enumerate(detection_indices):

deep_sort_app.py

@@ -114,7 +114,7 @@ def create_detections(detection_mat, frame_idx, min_height=0):
         Returns detection responses at given frame index.
 
     """
-    frame_indices = detection_mat[:, 0].astype(np.int)
+    frame_indices = detection_mat[:, 0].astype(np.int64)
     mask = frame_indices == frame_idx
 
     detection_list = []
@@ -144,7 +144,7 @@ def run(sequence_dir, detection_file, output_file, min_confidence,
         Detection confidence threshold. Disregard all detections that have
         a confidence lower than this value.
     nms_max_overlap: float
-        Maximum detection overlap (non-maxima suppression threshold).
+        Maximum detection overlap (non-maximum suppression threshold).
     min_detection_height : int
         Detection height threshold. Disregard all detections that have
         a height lower than this value.
@@ -171,7 +171,7 @@ def frame_callback(vis, frame_idx):
             seq_info["detections"], frame_idx, min_detection_height)
         detections = [d for d in detections if d.confidence >= min_confidence]
 
-        # Run non-maxima suppression.
+        # Run non-maximum suppression.
         boxes = np.array([d.tlwh for d in detections])
         scores = np.array([d.confidence for d in detections])
         indices = preprocessing.non_max_suppression(
@@ -212,6 +212,12 @@ def frame_callback(vis, frame_idx):
             row[0], row[1], row[2], row[3], row[4], row[5]),file=f)
 
 
+def bool_string(input_string):
+    if input_string not in {"True","False"}:
+        raise ValueError("Please Enter a valid Ture/False choice")
+    else:
+        return (input_string == "True")
+
 def parse_args():
     """ Parse command line arguments.
     """
@@ -235,7 +241,7 @@ def parse_args():
         "box height. Detections with height smaller than this value are "
         "disregarded", default=0, type=int)
     parser.add_argument(
-        "--nms_max_overlap",  help="Non-maxima suppression threshold: Maximum "
+        "--nms_max_overlap",  help="Non-maximum suppression threshold: Maximum "
         "detection overlap.", default=1.0, type=float)
     parser.add_argument(
         "--max_cosine_distance", help="Gating threshold for cosine distance "
@@ -245,7 +251,7 @@ def parse_args():
         "gallery. If None, no budget is enforced.", type=int, default=None)
     parser.add_argument(
         "--display", help="Show intermediate tracking results",
-        default=True, type=bool)
+        default=True, type=bool_string)
     return parser.parse_args()
 
 

evaluate_motchallenge.py

@@ -19,14 +19,15 @@ def parse_args():
         "be created if it does not exist.", default="results")
     parser.add_argument(
         "--min_confidence", help="Detection confidence threshold. Disregard "
-        "all detections that have a confidence lower than this value.",
-        default=0.0, type=float)
+        "all detections that have a confidence lower than this value. Set to "
+        "0.3 to reproduce results in the paper.",
+        default=0.3, type=float)
     parser.add_argument(
         "--min_detection_height", help="Threshold on the detection bounding "
         "box height. Detections with height smaller than this value are "
         "disregarded", default=0, type=int)
     parser.add_argument(
-        "--nms_max_overlap",  help="Non-maxima suppression threshold: Maximum "
+        "--nms_max_overlap",  help="Non-maximum suppression threshold: Maximum "
         "detection overlap.", default=1.0, type=float)
     parser.add_argument(
         "--max_cosine_distance", help="Gating threshold for cosine distance "