Fix QuantizableMobileNetV3 weight loading issue

[datumbox]

Fixes #4959

cc @pmeier, @NicolasHug This is a bug which wasn't caught due to our current testing strategy. Anything that depends on loading the real weights of the models is currently not covered.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit 9d1a609 (more details on the Dr. CI page):

---

💚 💚 Looks good so far! There are no failures yet. 💚 💚

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[prabhat00155]

Thanks @datumbox! Should we add a unit test to cover this?

[datumbox]

@prabhat00155 Yes, we should. Unfortunately we can't do it yet. To test it we would need to download the weights and try to load them on the model. Our current practice is to avoid downloading the weights in all tests to avoid flakiness. I think we should review this an redesign our model testing approach. This is probably a big effort and we should plan for it.

[pmeier]

IIUC, we want to test loading the state dict, right? If yes, the solution is probably not that hard to achieve. Can't we have a separate test that loads the model with pretrained=False and then does something like

from collections import OrderedDict

state_dict = OrderedDict([(name, torch.randn_like(weight)) for name, weight in model.state_dict()])
model.load_state_dict(state_dict)
torch.testing.assert_close(model.state_dict(), state_dict)

[datumbox]

@pmeier It's a creative proposal. :) Unfortunately I don't think this tests all scenarios because it assumes that the current keys retrieved from mode.state_dict() are the same as in the stored weights. This can be an incorrect assumption, especially when large scale refactoring is involved (see #4487). Moreover, there are situations where depending on the parameters passed to the builder methods, the behaviour changes. An example of this is the current bug fix. I don't believe you can cover this with the proposed trick. What we need to do is download the weights and ensure they still load properly.

[pmeier]

In that case, there is probably little we can do besides #4772 or downloading the weights in the background while doing something not I/O bound such as compiling torchvision.