Potential bug on drawing segmentation mask with holes

[XiaoSong9905]

Instructions To Reproduce the 🐛 Bug:

1. model has predicted segmentation masks with holes. (white area are predicted mask)

2. When Visualizer.draw_instance_predictions() create GenericMask and call Visualizer.overlay_instances

3. Visualizer.overlay_instances draws mask with "polygon" formate, which considers the holes inside the prediction (which have it own polygon) as part of the prediction.

4. This would induce result like this, where the holes are also considered as predicted area.

5. This is mainly because when the class::GenericMask return the polygon, it does not consider the hole information

and this have to do with cv2.findContours(mask.astype("uint8"), cv2.RETR_CCOMP, cv2.CHAIN_APPROX_NONE) return holes inside the mask but there's not subsequrnce operation to filter out those polygon

[alexriedel1]

Maybe using the method RETR_EXTERNAL for findContours can solve this, as it only finds outer contours?
cv2.findContours(input, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

see: https://docs.opencv.org/trunk/d3/dc0/group__imgproc__shape.html#ga819779b9857cc2f8601e6526a3a5bc71

[hiroto01]

Hi, is there update regarding this?
@ppwwyyxx ? @XiaoSong9905

[MjdMahasneh]

@ppwwyyxx any updates on this! I have actually been trying to find a work around over a month now! even RLEs do not account for holes, could you kindly advice? I am using detectron2 as the backend for my industrial research, it would be massively helpful for me and others.

@XiaoSong9905 @alexriedel1 @hiroto01 any advice on this, please?

Looking forward.

[cm7S]

@MjdMahasneh RLEs do account for holes, it's just that the visualizer converts the masks to polygons (which do not account for holes)

To fix this you need to split the outer contour polygon and insert the polygon of the hole inside it, like done here using cv2.findcontours:
#4196 (comment)