MLPerf ResNet inference workflow failing on Windows

[gfursin]

Hi @anandhu-eng and @arjunsuresh,

I just tried the CM-MLPerf workflow for ResNet benchmark with the latest CM4MLOps mlperf-inference branch on my Windows machine and it fails when downloading a model.

It's weird because we have a GitHub test for this benchmark on Windows...

Do you mind to check if it works on your side, please? If not, I am curious why our tests didn't capture that.

Here is the command I used on Windows 11 (copy/paste from our GitHub test):

cm pull repo mlcommons@cm4mlops --branch=mlperf-inference

cm rm cache -f

cm run script --tags=run-mlperf,inference,_submission,_short --submitter="cTuning" --hw_name=default --model=resnet50 --implementation=python --backend=onnxruntime --device=cpu --scenario=Offline --test_query_count=500 --target_qps=1 -v --quiet 

and here is the error:

Downloading from https://zenodo.org/record/4735647/files/resnet50_v1.onnx

File resnet50_v1.onnx already present, original checksum and computed checksum matches! Skipping Download..

Traceback (most recent call last):
  File "C:\!Progs\Python310\Scripts\cm-script.py", line 33, in <module>
    sys.exit(load_entry_point('cmind==2.3.8.1', 'console_scripts', 'cm')())
  File "C:\!Progs\Python310\lib\site-packages\cmind\cli.py", line 37, in run
    r = cm.access(argv, out='con')
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1477, in _run
    r = customize_code.preprocess(ii)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\script\run-mlperf-inference-app\customize.py", line 231, in preprocess
    r = cm.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 761, in access
    return cm.access(i)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1367, in _run
    r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1537, in _run
    r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta,  env, state, const, const_state, add_deps_recursive,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 2882, in _call_run_deps
    r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 3033, in _run_deps
    r = self.cmind.access(ii)
  File "C:\!Progs\Python310\lib\site-packages\cmind\core.py", line 605, in access
    r = action_addr(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 212, in run
    r = self._run(i)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\automation\script\module.py", line 1477, in _run
    r = customize_code.preprocess(ii)
  File "D:\Work1\CM\repos\mlcommons@cm4mlops\script\download-file\customize.py", line 210, in preprocess
    env['CM_DOWNLOAD_CMD'] =  env['CM_DOWNLOAD_CMD'].replace('&', '^&').replace('|', '^|').replace('(', '^(').replace(')', '^)')
KeyError: 'CM_DOWNLOAD_CMD'
                                                                                                                

Thanks!

[arjunsuresh]

Hi @gfursin this PR should fix it. I think it went fine in github action because the cache was clean.

[gfursin]

Thank you @arjunsuresh, but I now got another error at the same place with the same command (after cleaning cache):

CM error: Downloaded path D:\Work1\CM\repos\local\cache\0ad2afc45a05440d\resnet50_v1.onnx does not exist. Probably CM_DOWNLOAD_FILENAME is not set and CM_DOWNLOAD_URL given is not pointing to a file!

[arjunsuresh]

oh. Looks like the script is broken on Windows. @anandhu-eng can you please check?

We actually don't have the github test for R50 on Windows - I believe the workflow is having other issues too on Windows.

[gfursin]

Oh, I didn't see that it was excluded from the test ... It will be nice to fix it and add to the tests since I fixed and used it this spring and it worked ...

[anandhu-eng]

Hi @arjunsuresh @gfursin , the issue occurs because md5sum is not available on Windows for verifying checksums, and it was not properly caught in our code. We could use certutil on Windows for the same purpose. Let me fix it.

[gfursin]

Actually, we do have md5sum for Windows. I added it to the script get-sys-utils-cm a while ago (https://zenodo.org/record/6501550/files/cm-artifact-os-windows-32.zip).

That's where I keep the min set of aux tools for Windows (including wget).

I remember I made our download scripts compatible with Windows and md5sum was working last year using this dependency ... @anandhu-eng - I suggest to use it. I will try to update it with the latest wget.exe today too ...

[gfursin]

@anandhu-eng . In fact, we use get-sys-utils-min on Windows instead of get-sys-utils-cm . It has md5sum that I use on Windows.

[arjunsuresh]

Hi @gfursin I think the problem started when we added md5sum check for the downloads via cmutil. Since this is done in preprocess function, md5sum check is done via a subprocess call.