Amend the weights only if quantize=True. (#4966)

[datumbox]

Cherrypick from #4966

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit acad1d5 (more details on the Dr. CI page):

---

• 1/1 failures introduced in this PR

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

cmake_windows_cpu (1/1)

Step: "set -ex
source packaging/windows/internal/vc_install_helper.sh
packaging/build_cmake.sh
" (full log | diagnosis details | 🔁 rerun)

warnings.warn(f'Error checking compiler version for {compiler}: {error}')

Installed c:\users\circleci\project
Processing dependencies for torchvision==0.11.0.dev20211208
Searching for numpy
Reading https://pypi.org/simple/numpy/
Downloading https://files.pythonhosted.org/packages/7a/7f/5384d4c85a2349bd89ff1c0253bff77c7b617e48af201f2d823fc619189a/numpy-1.22.0rc1.zip#sha256=bc991b3f8ea7c0f6703df2bc23c098cfe6f1a3a5e8a3a901eb6a5619275d53ff
Best match: numpy 1.22.0rc1
Processing numpy-1.22.0rc1.zip
Writing C:\Users\circleci\AppData\Local\Temp\easy_install-vtr0_bfp\numpy-1.22.0rc1\setup.cfg
Running numpy-1.22.0rc1\setup.py -q bdist_egg --dist-dir C:\Users\circleci\AppData\Local\Temp\easy_install-vtr0_bfp\numpy-1.22.0rc1\egg-dist-tmp-di7kavq4
C:\tools\miniconda3\lib\site-packages\torch\utils\cpp_extension.py:316: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Traceback (most recent call last):
  File "C:\tools\miniconda3\lib\site-packages\setuptools\sandbox.py", line 154, in save_modules
    yield saved
  File "C:\tools\miniconda3\lib\site-packages\setuptools\sandbox.py", line 195, in setup_context
    yield
  File "C:\tools\miniconda3\lib\site-packages\setuptools\sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "C:\tools\miniconda3\lib\site-packages\setuptools\sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "C:\Users\circleci\AppData\Local\Temp\easy_install-vtr0_bfp\numpy-1.22.0rc1\setup.py", line 34, in <module>

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.