Visualize bounding boxes as Napari shapes layer (#590)

* Basic implementation of bboxes visualization as shapes

* Basic implementation of bboxes visualization as shapes

* Added test coverage and polish

* Removed redundant set_text method for shapes style

* Finalized docs

* Revised based on review

* Apply suggestions from code review

Committed all review changes not subject to deeper revision

Co-authored-by: sfmig <33267254+sfmig@users.noreply.github.com>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Added test coverage for bboxes set_text_by

* Small edits to user guide

* Use ds.ds_type == "bboxes" to identify bbox datasets

* Small edits

* Refactor and split bloated test. Clarify test on bboxes style. Parametrise test over bboxes and poses data.

* Fix docstring rendering

---------

Co-authored-by: sfmig <33267254+sfmig@users.noreply.github.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: 3 people

docs/source/_static/napari_bboxes_layer.png

@@ -5,7 +5,7 @@ The `movement` graphical user interface (GUI), powered by our custom plugin for
 [napari](napari:), makes it easy to view and explore `movement`
 motion tracks. Currently, you can use it to
 visualise 2D [movement datasets](target-poses-and-bboxes-dataset)
-as points and tracks overlaid on video frames.
+as points, tracks, and rectangular bounding boxes (if defined) overlaid on video frames.
 
 :::{warning}
 The GUI is still in early stages of development but we are working on ironing
@@ -138,17 +138,21 @@ an expanded `Load tracked data` menu. To load tracked data in napari:
 
 The data will be loaded into the viewer as a
 [points layer](napari:howtos/layers/points.html) and as a [tracks layer](napari:howtos/layers/tracks.html).
+If the input file is a bounding boxes dataset, an additional napari [shapes layer](napari:howtos/layers/shapes.html) is loaded.
 By default, the data is added at the top of the layer list and the points layer is selected.
 
+For a poses dataset, you will see a view similar to this:
+
 (target-widget-screenshot)=
 
 ![napari widget with poses dataset loaded](../_static/napari_plugin_data_tracks.png)
 
+And for a bounding boxes dataset, you will see a view more like the one below:
+
+![napari widget with shapes loaded](../_static/napari_bboxes_layer.png)
 
-You will see a view similar to the one above. Notice the three
-layers on the left-hand side list: the
-image layer, that holds the background information (i.e., the loaded video or image), the points layer and the tracks layer.
-You can toggle the visibility of each of these layers by clicking on the eye icon.
+
+Note the additional bounding boxes layer that is loaded for bounding boxes datasets. For both poses and bounding boxes datasets, you can toggle the visibility of any of these layers by clicking on the eye icon.
 
 
 ### The points layer
@@ -205,8 +209,7 @@ Remember that the current frame is determined by the position of the frame slide
 The trajectory made up of all positions of a keypoint on all frames before the current frame is called _tail_.
 Similarly, its trajectory on all frames after the current frame is called _head_.
 
-Both tail and head tracks are represented as lines connecting the keypoints
-of the same individual across frames. The colour of the tracks follows
+Both tail and head tracks are represented as lines connecting a single keypoint across frames. The colour of the tracks follows
 the colour of the markers, and the length of the tracks can be adjusted in the
 [tracks layer](napari:howtos/layers/tracks.html) controls panel, with the `tail length` and `head length` sliders.
 
@@ -243,3 +246,27 @@ working on a workaround, stay tuned!
 - Also note that currently the `show ID` checkbox in the [tracks layer](napari:howtos/layers/tracks.html) controls panel refers to
 an internal napari track ID, rather than the individual or the keypoint ID. This is a known issue and we are working on a fix or workaround.
 :::
+
+### The boxes layer
+
+The boxes layer is loaded for bounding boxes datasets only. It shows the bounding boxes for the current frame, as rectangles color-coded by individual.
+
+The name of the individual is shown in the lower left of the bounding box by default. This can be toggled by selecting the [shapes layer](napari:howtos/layers/shapes.html) in the layer list and clicking the `display text` checkbox in the layer controls panel.
+
+As with tracks and points, you can use the frame slider at the bottom of the viewer to move through the frames of the dataset. This will update the bounding boxes and the rest of the loaded data in sync.
+
+:::{admonition} Changing bounding box size, colour and shape
+:class: tip
+
+You can change edge and face colour as well as the positions of a bounding box's corner vertices using the
+[shapes layer](napari:howtos/layers/shapes.html) controls panel.
+
+You can use the following keyboard shortcuts to toggle the bounding boxes selection:
+- To select all the bounding boxes in the current frame, enable the select shapes tool (`5` or `S`) and press `A`.
+- To deselect, click off of the bounding boxes.
+- Individual vertices can be selected instead of the entire rectangle using the select vertex tool: `4` or `D`.
+
+You can find all the [keyboard shortcuts](napari:guides/preferences.html#shortcuts) in the top menu of the
+`napari` window, under `Preferences > Shortcuts`.
+
+:::

docs/source/user_guide/gui.md

@@ -4,8 +4,6 @@
 import pandas as pd
 import xarray as xr
 
-# get logger
-
 
 def _construct_properties_dataframe(ds: xr.Dataset) -> pd.DataFrame:
     """Construct a properties DataFrame from a ``movement`` dataset."""
@@ -23,9 +21,25 @@ def _construct_properties_dataframe(ds: xr.Dataset) -> pd.DataFrame:
     return pd.DataFrame(data).reindex(columns=desired_order)
 
 
-def ds_to_napari_tracks(
+def _construct_track_and_time_cols(
+    ds: xr.Dataset,
+) -> tuple[np.ndarray, np.ndarray]:
+    """Compute napari track_id and time columns from a ``movement`` dataset."""
+    n_frames = ds.sizes["time"]
+    n_individuals = ds.sizes["individuals"]
+    n_keypoints = ds.sizes.get("keypoints", 1)
+    n_tracks = n_individuals * n_keypoints
+
+    # Each keypoint of each individual is a separate track
+    track_id_col = np.repeat(np.arange(n_tracks), n_frames).reshape(-1, 1)
+    time_col = np.tile(np.arange(n_frames), (n_tracks)).reshape(-1, 1)
+
+    return track_id_col, time_col
+
+
+def ds_to_napari_layers(
     ds: xr.Dataset,
-) -> tuple[np.ndarray, pd.DataFrame]:
+) -> tuple[np.ndarray, np.ndarray | None, pd.DataFrame]:
     """Convert ``movement`` dataset to napari Tracks array and properties.
 
     Parameters
@@ -36,12 +50,20 @@ def ds_to_napari_tracks(
 
     Returns
     -------
-    data : np.ndarray
-        napari Tracks array with shape (N, 4),
+    points_as_napari : np.ndarray
+        position data as a napari Tracks array with shape (N, 4),
         where N is n_keypoints * n_individuals * n_frames
         and the 4 columns are (track_id, frame_idx, y, x).
+    bboxes_as_napari : np.ndarray | None
+        bounding box data as a napari Shapes array with shape (N, 4, 4),
+        where N is n_individuals * n_frames and each (4, 4) entry is
+        a matrix of 4 rows (1 per corner vertex, starting from upper left
+        and progressing in counterclockwise order) with the columns
+        (track_id, frame, y, x). Returns None when the input dataset doesn't
+        have a "shape" variable.
     properties : pd.DataFrame
-        DataFrame with properties (individual, keypoint, time, confidence).
+        DataFrame with properties (individual, keypoint, time, confidence)
+        for use with napari layers.
 
     Notes
     -----
@@ -55,12 +77,9 @@ def ds_to_napari_tracks(
     .. [2] https://napari.org/stable/howtos/layers/points.html
 
     """
-    n_frames = ds.sizes["time"]
-    n_individuals = ds.sizes["individuals"]
-    n_keypoints = ds.sizes.get("keypoints", 1)
-    n_tracks = n_individuals * n_keypoints
+    # Construct the track_ID and time columns for the napari Tracks array
+    track_id_col, time_col = _construct_track_and_time_cols(ds)
 
-    # Construct the napari Tracks array
     # Reorder axes to (individuals, keypoints, frames, xy)
     axes_reordering: tuple[int, ...] = (2, 0, 1)
     if "keypoints" in ds.coords:
@@ -70,10 +89,50 @@ def ds_to_napari_tracks(
         axes_reordering,  # to: individuals, keypoints, frames, xy
     ).reshape(-1, 2)[:, [1, 0]]  # swap x and y columns
 
-    # Each keypoint of each individual is a separate track
-    track_id_col = np.repeat(np.arange(n_tracks), n_frames).reshape(-1, 1)
-    time_col = np.tile(np.arange(n_frames), (n_tracks)).reshape(-1, 1)
-    data = np.hstack((track_id_col, time_col, yx_cols))
+    points_as_napari = np.hstack((track_id_col, time_col, yx_cols))
+    bboxes_as_napari = None
+
+    # Construct the napari Shapes array if the input dataset is a
+    # bounding boxes one
+    if ds.ds_type == "bboxes":
+        # Compute bbox corners
+        xmin_ymin = ds.position - (ds.shape / 2)
+        xmax_ymax = ds.position + (ds.shape / 2)
+
+        # initialise xmax, ymin corner as xmin, ymin
+        xmax_ymin = xmin_ymin.copy()
+        # overwrite its x coordinate to xmax
+        xmax_ymin.loc[{"space": "x"}] = xmax_ymax.loc[{"space": "x"}]
+
+        # initialise xmin, ymin corner as xmin, ymin
+        xmin_ymax = xmin_ymin.copy()
+        # overwrite its y coordinate to ymax
+        xmin_ymax.loc[{"space": "y"}] = xmax_ymax.loc[{"space": "y"}]
+
+        # Add track_id and time columns to each corner array
+        corner_arrays_with_track_id_and_time = [
+            np.c_[
+                track_id_col,
+                time_col,
+                np.transpose(corner.values, axes_reordering).reshape(-1, 2),
+            ]
+            for corner in [xmin_ymin, xmin_ymax, xmax_ymax, xmax_ymin]
+        ]
+
+        # Concatenate corner arrays along columns
+        corners_array = np.concatenate(
+            corner_arrays_with_track_id_and_time, axis=1
+        )
+
+        # Reshape to napari expected format
+        # goes through corners counterclockwise from xmin_ymin
+        # in image coordinates
+        corners_array = corners_array.reshape(
+            -1, 4, 4
+        )  # last dimension: track_id, time, x, y
+        bboxes_as_napari = corners_array[
+            :, :, [0, 1, 3, 2]
+        ]  # swap x and y columns
 
     # Construct the properties DataFrame
     # Stack individuals, time and keypoints (if present) dimensions
@@ -85,4 +144,4 @@ def ds_to_napari_tracks(
 
     properties = _construct_properties_dataframe(ds_)
 
-    return data, properties
+    return points_as_napari, bboxes_as_napari, properties

movement/napari/convert.py

@@ -121,6 +121,75 @@ def set_color_by(self, property: str, cmap: str | None = None) -> None:
             self.colormap = cmap
 
 
+@dataclass
+class BoxesStyle(LayerStyle):
+    """Style properties for a napari Shapes layer."""
+
+    edge_width: int = 3
+    opacity: float = 1.0
+    shape_type: str = "rectangle"
+    face_color: str = "#FFFFFF00"  # transparent face
+    edge_colormap: str = DEFAULT_COLORMAP
+    text: dict = field(
+        default_factory=lambda: {
+            "visible": True,  # default visible text for bboxes
+            "anchor": "lower_left",
+            "translation": 5,  # pixels
+        }
+    )
+
+    def set_color_by(
+        self,
+        property: str,
+        properties_df: pd.DataFrame,
+        cmap: str | None = None,
+    ) -> None:
+        """Color boxes and text by chosen column in the properties DataFrame.
+
+        Parameters
+        ----------
+        property : str
+            The column name in the properties DataFrame to color shape edges
+            and associated text by.
+        properties_df : pd.DataFrame
+            The properties DataFrame containing the data for generating the
+            colormap.
+        cmap : str, optional
+            The name of the colormap to use, otherwise use the edge_colormap.
+
+        Notes
+        -----
+        The input property is expected to be a column in the properties
+        dataframe and it is used to define the color of the text. A factorized
+        version of the property ("<property>_factorized") is used to define the
+        edges color, and is also expected to be present in the properties
+        dataframe.
+
+        """
+        # Compute color cycle based on property
+        if cmap is None:
+            cmap = self.edge_colormap
+        n_colors = len(properties_df[property].unique())
+        color_cycle = _sample_colormap(n_colors, cmap)
+
+        # Set color for edges and text
+        self.edge_color = property + "_factorized"
+        self.edge_color_cycle = color_cycle
+        self.text["color"] = {"feature": property}
+        self.text["color"].update({"colormap": color_cycle})
+
+    def set_text_by(self, property: str) -> None:
+        """Set the text property for the boxes layer.
+
+        Parameters
+        ----------
+        property : str
+            The column name in the properties DataFrame to use for text.
+
+        """
+        self.text["string"] = property
+
+
 def _sample_colormap(n: int, cmap_name: str) -> list[tuple]:
     """Sample n equally-spaced colors from a napari colormap.