Tests for MLPerf inference submission generation

[anandhu-eng]

Hi @arjunsuresh , these are the identified tests which could help us cover various possibilities for bugs in submission generation through CM for non-CM based benchmarking method. Please feel free to add more, delete or modify them:

• Case-1: model_maping.json in SUT folder

• Case-2: model_mapping.json in individual folder

• Case-3: model_mapping.json not present but model name is matching with the official one in submission checker

• Case-4: model_mapping.json is not present but model name is mapped to official model name in submission checker. Example: resnet50 to resnet

• Case-5: Case-1 to Case-4 is not satisfied. The gh action will be successfull if the submission generation fails.

• Case-6: Case-2 but model_mapping.json is not present in any of the folders. The gh action will be successful if the submission generation fails.

• Case-7: sut_info.json is not completely filled but the SUT folder name is in required format(hardware_name-implementation-device-framework-run_config)

[arjunsuresh]

Nice @anandhu-eng . Cases 1-6 are for open division and so case 7 is the most important one. We use cm-sut-info.json and sut-info.json?

[anandhu-eng]

I believe we are only using cm-sut-info.json in our code. Let me confirm it and update the code otherwise

[arjunsuresh]

@anandhu-eng We now have cm test script --tags=... working and can specify test inputs like this. The tests are automatically triggered when the corresponding script meta file is changed.

[arjunsuresh]

Hi @anandhu-eng are these completed now?

[anandhu-eng]

Hi @arjunsuresh , case 3 and case 4 are the same. Also do we need case 6 to be in github actions? Rest all are added to inference repo.

[arjunsuresh]

yes, I think case-6 is useful to have.