Added PAT metric #659

mohan1914 · 2021-09-14T11:56:05Z

Add PAT metric for panoptic tracking evaluation.

lubing-motional · 2021-09-15T05:56:41Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

@@ -336,6 +382,65 @@ def get_lstq(self) -> Tuple[np.ndarray, np.ndarray]:
        lstq = np.sqrt(s_assoc * s_cls)
        return lstq, s_assoc

+    def get_pat(self) -> Tuple[np.ndarray, np.ndarray]:


return Tuple should have 3 arrays.

lubing-motional · 2021-09-15T05:57:49Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+        pq_all = sq_all * rq_all
+
+        # then do the REAL mean (no ignored classes)
+        SQ = sq_all[self.include].mean()


use low-case variables to be consistent

remove unused SQ, RQ.

lubing-motional · 2021-09-15T09:28:44Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+        #accumulate class-agnostic predictions
+        unique_pred_, counts_pred_ = np.unique(x_inst_row[1][x_inst_row[1] > 0], return_counts=True) 
+        for p_id in unique_pred_[counts_pred_ > self.min_points]:
+                if p_id not in self.instance_preds[scene]:


indentation is incorrect

lubing-motional · 2021-09-15T09:55:20Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    unique_pr_id, counts_pr_id = np.unique(pr_ids, return_counts=True)
+
+                    track_length = len(pr_ids)
+                    unique_pr_id, counts_pr_id = unique_pr_id[unique_pr_id!=1], counts_pr_id[unique_pr_id!=1] 


could you add more inline comments for lines like this, why need to remove pr_id == 1, I feel it is hard to decode all the logic by just reading the code.

lubing-motional · 2021-09-15T10:01:45Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+            matched_gt_[[id2idx_gt_[g_id] for g_id in gt_labels_[tp_indexes_agnostic]]] = True
+
+            for idx, value in enumerate(tp_indexes_agnostic):
+                if value == True:


use if value:

lubing-motional · 2021-09-15T10:02:10Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                        self.instance_gts[scene][cl][g_label].append(p_label)
+
+            for g_label in unique_gt_:
+                if matched_gt_[id2idx_gt_[g_label]] == False: 


use if not matched_gt_[id2idx_gt_[g_label]]

lubing-motional · 2021-09-15T10:05:31Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+            for g_label in unique_gt_:
+                if matched_gt_[id2idx_gt_[g_label]] == False: 
+                   if g_label not in self.instance_gts[scene][cl]:
+                        self.instance_gts[scene][cl][g_label] = [1,]


could you double check the indentations, here has 5 spaces, i see many places have wrong indentations.

holger-motional · 2021-09-15T10:51:09Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+            pred_areas_ = np.array([counts_pred_[id2idx_pred_[p_id]] for p_id in pred_labels_])
+            intersections_ = counts_combo_
+            unions_ = gt_areas_ + pred_areas_ - intersections_
+            ious_agnostic = intersections_.astype(np.float32) / unions_.astype(np.float32)


Could unions be 0?

No. gt_areas_ can never be 0.

holger-motional · 2021-09-15T10:51:43Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

@@ -186,6 +224,14 @@ def add_batch_panoptic(self,
                x_inst_row[0] = x_inst_row[0][gt_not_in_excl_mask]
                y_inst_row[0] = y_inst_row[0][gt_not_in_excl_mask]

+        #accumulate class-agnostic predictions


Format to: # Accumulate

holger-motional · 2021-09-15T10:52:13Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    unique_pr_id, counts_pr_id = np.unique(pr_ids, return_counts=True)
+
+                    track_length = len(pr_ids)
+                    unique_pr_id, counts_pr_id = unique_pr_id[unique_pr_id!=1], counts_pr_id[unique_pr_id!=1] 


spacing around !=

holger-motional · 2021-09-15T10:52:32Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                            fp_pr_id.append(0)
+
+                    fp_pr_id = np.array(fp_pr_id)
+                    gt_id_aq = (np.sum(counts_pr_id**2/np.double(track_length + fp_pr_id)))/np.double(track_length)


Let's add more spacing around operators

whyekit-motional · 2021-09-16T00:50:11Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+            unique_pred_, counts_pred_ = np.unique(x_inst_row[x_inst_row > 0], return_counts=True)
+            id2idx_pred_ = {inst_id: idx for idx, inst_id in enumerate(unique_pred_)}
+            gt_labels_ = unique_combo_ // self.offset
+            pred_labels_ = unique_combo_ % self.offset


Perhaps some comments on what these two lines are doing?:

gt_labels_ = unique_combo_ // self.offset pred_labels_ = unique_combo_ % self.offset

Added the comment. Let me know if it is intuitive.

whyekit-motional · 2021-09-16T00:52:11Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

@@ -160,6 +196,8 @@ def add_batch_panoptic(self,
            self.gts[scene] = [{} for _ in range(self.n_classes)]
            self.intersects[scene] = [{} for _ in range(self.n_classes)]
            self.intersects_ovr[scene] = [{} for _ in range(self.n_classes)]
+            self.instance_preds[scene] = {}
+            self.instance_gts[scene] = [{} for _ in range(self.n_classes)]
        # Make sure instance IDs are non-zeros. Otherwise, they will be ignored. Note in nuScenes-panoptic,


nuScenes-panoptic --> Panoptic nuScenes?

Corrected it

lubing-motional · 2021-09-17T05:25:14Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    track_length = len(pr_ids)
+                    # void/stuff have instance value 1 due to the +1 in ln205 as well as unmatched gt is denoted by 1 
+                    # Thus we remove 1 from the prediction id list  
+                    unique_pr_id, counts_pr_id = unique_pr_id[unique_pr_id != 1], counts_pr_id[unique_pr_id != 1] 


tks for the inline comment, sigh, this looks like a hack on top of another hack..

lubing-motional · 2021-09-17T05:26:10Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    #     preds[uid]: TPA + FPA (class-agnostic)
+                    #     counts_pr_id[idx]: TPA (class-agnostic)
+                    # If prediction id is not in preds it means it has number of points < self.min_points.
+                    # Similar to PQ computation we consider pred with number of points < self.min_points with IoU overlap greater than 0.5


change line if it exceeds 120 line width

lubing-motional · 2021-09-17T05:26:39Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                        # Total possible id switches
+                        total_ids = track_length - 1
+                        # Gt tracks with no corresponding prediction match are assigned 1.
+                        # We consider an id switch occurs if previous predicted id and the current one don't match for the given gt track


exceed 120 line width

lubing-motional · 2021-09-17T05:30:19Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    unique_pr_id, counts_pr_id = unique_pr_id[unique_pr_id != 1], counts_pr_id[unique_pr_id != 1] 
+                    fp_pr_id = []
+
+                    # Computes the total false positve for each prediction id:


typo: positive

lubing-motional · 2021-09-17T05:37:08Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+        pq = pq_all[self.include].mean()
+
+        accumulate_tq = 0.0
+        accumlate_norm = 0


typo: accumulate_norm

lubing-motional · 2021-09-17T05:46:56Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

+                    # Accumulate TQ over all the possible unique gt instances
+                    accumulate_tq += np.sqrt(gt_id_aq * gt_id_is)
+                    # Count the total number of unique gt instances
+                    accumlate_norm +=1 


minor: space after +=, similarly for line 461.

lubing-motional · 2021-09-17T05:50:30Z

python-sdk/nuscenes/eval/panoptic/panoptic_track_evaluator.py

@@ -129,6 +133,48 @@ def get_panoptic_track_stats(self,
            unique_combo_, counts_combo_ = np.unique(offset_combo_, return_counts=True)
            self.update_dict_stat(cl_intersects, unique_combo_, counts_combo_)

+            # Computation for PAT score


could you help to correct the return typing to Tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, Dict[int, int], Dict[int, int], np.ndarray] in lines 83-84. tks.

lubing-motional · 2021-09-17T05:57:39Z

python-sdk/nuscenes/eval/panoptic/evaluate.py

        mean_ptq, class_all_ptq, mean_sptq, class_all_sptq = self.evaluator['tracking'].get_ptq()
        mean_iou, class_all_iou = self.evaluator['tracking'].getSemIoU()
        lstq, s_assoc = self.evaluator['tracking'].get_lstq()
        mean_motsa, mean_s_motsa, mean_motsp = self.evaluator['tracking'].get_motsa()

-        results = self.wrap_result_mopt(mean_ptq=mean_ptq,


update the function doc-string part at line 203-210 to add the added PAT, PQ, TQ fields

lubing-motional

Took a second pass, clearer that previous round, even though it's still hard for me to follow the logic for all the details due to the implementation complexity, but think it's hard to improve that without massive changes. So I'd proceed to approve it due to the time being.

lubing-motional · 2021-09-17T06:24:35Z

@mohan1914 Pls promote the version to 1.1.8 for following 2 places. you can refer to PR
(1) https://github.com/nutonomy/nuscenes-devkit/blob/master/setup/setup.py#L42
(2) https://github.com/nutonomy/nuscenes-devkit/blob/master/README.md, Add following line in the ChangeLog
"Sep 17, 2021: Devkit v1.1.8: Add PAT metric to Panoptic nuScenes."

added PAT metric

103ec34

whyekit-motional requested review from lubing-motional, holger-motional and whyekit-motional September 15, 2021 05:29

lubing-motional reviewed Sep 15, 2021

View reviewed changes

holger-motional approved these changes Sep 15, 2021

View reviewed changes

holger-motional changed the title ~~added PAT metric~~ Added PAT metric Sep 15, 2021

whyekit-motional reviewed Sep 16, 2021

View reviewed changes

formating fix and add comments

2ce197c

lubing-motional reviewed Sep 17, 2021

View reviewed changes

lubing-motional approved these changes Sep 17, 2021

View reviewed changes

mohan1914 added 3 commits September 17, 2021 13:58

formatting fixes v2

ad87505

add 1.1.8 to changelog

4f88875

add Panoptic nuScenes to changelog

c402fa7

lubing-motional merged commit 5058621 into nutonomy:master Sep 18, 2021

Added PAT metric #659

Added PAT metric #659

Conversation

mohan1914 commented Sep 14, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lubing-motional left a comment

Choose a reason for hiding this comment

lubing-motional commented Sep 17, 2021 • edited Loading

lubing-motional commented Sep 17, 2021 •

edited

Loading