How to get corresponding feature regions of final detections from feature map of backbone?

[igygi]

Hi,

For every output detection [x1, y1, x2, y2], I would like to extract its corresponding region in the feature map output of the backbone of Faster-RCNN. Similarly, I want to extract the corresponding region in the feature map for the target (groundtruth) bounding boxes.

Can you point me to how this should be done?

Thank you.

[akashprakas]

I am not sure the if the below is right,but you can look into it

import torchvision
from torchvision.models.detection import FasterRCNN
from torchvision.models.detection import roi_heads

This is from the forward method of FasterRCNN

features = self.backbone(images.tensors)
if isinstance(features, torch.Tensor):
    features = OrderedDict([('0', features)])
proposals, proposal_losses = self.rpn(images, features, targets)
detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)

This is from the forward method of roi_heads

        if self.training:
            proposals, matched_idxs, labels, regression_targets = self.select_training_samples(proposals, targets)
        else:
            labels = None
            regression_targets = None
            matched_idxs = None

        box_features = self.box_roi_pool(features, proposals, image_shapes)
        box_features = self.box_head(box_features)
        class_logits, box_regression = self.box_predictor(box_features)

1. If you look at the forward method of FasterRCNN you can see that the features are first passed through the rpn and proposals from the rpn to the roi_heads.
2. Inside the forward of roi_heads box_features is the roi pooled or aligned features, the forward of roi_heads only returns results and losses, if you want the features you might need to store box_features from box_roi_pool and return it .

[igygi]

Will try this out! But, if I want to get the corresponding features for the target boxes, does it make sense to pass the target bounding boxes instead of the proposal bounding boxes to box_roi_pool?