Change default value of eps in FrozenBatchNorm to match BatchNorm

[juyunsang]

❓ Questions and Help

Hello
Loss is nan error occurs when I learn fast rcnn with resnext101 backbone
My code is as follows

backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

error message

Epoch: [0]  [   0/7208]  eta: 1:27:42  lr: 0.000040  loss: 40613806080.0000 (40613806080.0000)  loss_box_reg: 7979147264.0000 (7979147264.0000)  loss_classifier: 11993160704.0000 (11993160704.0000)  loss_objectness: 9486380032.0000 (9486380032.0000)  loss_rpn_box_reg: 11155118080.0000 (11155118080.0000)  time: 0.7301  data: 0.4106  max mem: 1241
Loss is nan, stopping training

When i change the backbone to resnet50 and resnet152, no error occrus.

Please note that this issue tracker is not a help form and this issue will be closed.

We have a set of listed resources available on the website. Our primary means of support is our discussion forum:

• Discussion Forum

[pmeier]

Hi @juyunsang

as our template states:

Please note that this issue tracker is not a help form and this issue will be closed. [...] Our primary means of support is our discussion forum.

---

Without knowing your data its hard to tell what is going wrong. I'm assuming your data is not corrupt since the other models are working. Thus, this might be a hyper-parameter problem. The loss in the first step seems fairly large (~ 40e9). In a first step I would reduce the learning rate and see if this already solves the problem.

[fmassa]

@juyunsang note that we do not provide pre-trained weights for detection models with the resnext101 backbone, which might explain the issue you are facing. You might be finetuning a detection model with ResNet50 pre-trained on COCO, while training it from scratch with ResNeXt101

[juyunsang]

@fmassa
Thank you for reply
I don't understand your mention
You might be finetuning a detection model with ResNet50 pre-trained on COCO, while training it from scratch with ResNeXt101
Are you saying that I should use the code below?

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

Can you show me the code for how I can use the resnext101 backbone?

[fmassa]

@juyunsang My understanding was that you were using

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

for ResNet50, and

backbone = resnet_fpn_backbone('resnext101_32x8d', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

for ResNeXt101. Is it the case?

[juyunsang]

@fmassa
Yes!
It works well with the code below.

backbone = resnet_fpn_backbone('resnet50', pretrained=True)
model = FasterRCNN(backbone, num_classes)
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

However, changing the parameter of resnet_fpn_backbone function to resnext101_32x8d results in a Loss is Nan error.

[fmassa]

Thanks for confirming.

My first bet would be that it's an issue with the FPN, because we forgot to run the weight initialization, as discussed in #2326

Could you check if implementing that fix could solve the issue for you?

[juyunsang]

@fmassa
Thank you for reply
I change code self.children to self.modules
but error still occued..

vision/torchvision/ops/feature_pyramid_network.py

Lines 59 to 63 in c2e8a00

	# initialize parameters now to avoid modifying the initialization of top_blocks
	for m in self.children():
	if isinstance(m, nn.Conv2d):
	nn.init.kaiming_uniform_(m.weight, a=1)
	nn.init.constant_(m.bias, 0)

[fmassa]

Ok, I think I know what's going on. The resnext implementation might have weights which are zero in the batch norm, and we might need to set eps in

vision/torchvision/ops/misc.py

Line 54 in 497744b

eps: float = 0.,

to be 1e-5 to match the default values in PyTorch.

This can be done by changing the norm_layer in

vision/torchvision/models/detection/backbone_utils.py

Line 44 in 497744b

def resnet_fpn_backbone(backbone_name, pretrained, norm_layer=misc_nn_ops.FrozenBatchNorm2d, trainable_layers=3):

to be a lambda with the eps set out to be 1e-5.

Can you try this out and report back?

The same thing happened to @szagoruyko with WideResNets. Changing the default value of eps in FrozenBatchNorm should be taken into account, but it will change the results for pre-trained models, and this should be checked to see how much it affects performance

[dkadish]

I was having a somewhat similar issue and this seems to fix it.
I was trying to load the weights from a pre-trained torchvision.models.resnet50 model into a torchvision.models.detection.backbone_utils.resnet_fpn_backbone for use in a FasterRCNN model. Renaming the state dict keys (dict([('.'.join(['body'] + k.split('.')[1:]),v) for k,v in checkpoint["state_dict"].items()])) and loading it with strict=False worked, but I was getting Loss is nan, stopping training messages during training.

Using functools.partial as below created a model that got through training.

from functools import partial
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
from torchvision.ops.misc import FrozenBatchNorm2d

FBN = partial(FrozenBatchNorm2d, eps=1E-5)
backbone = resnet_fpn_backbone('resnet50', pretrained=True, norm_layer=FBN, trainable_layers=3).cuda()

[fmassa]

I think it might be time to think about changing the default in FrozenBatchNorm2d to more closely align with what BatchNorm in PyTorch does, we should just check that this change doesn't affect the performance of the currently trained models in any way.