Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add CPUExecutionProvider to the ONNX E2E #1398

Merged
merged 34 commits into from
Jan 13, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
863e7a2
Add CPU EP
kshpv Nov 30, 2022
c408986
Fix bug
kshpv Nov 30, 2022
5e2f944
Update report creation
kshpv Dec 1, 2022
f042429
Merge remote-tracking branch 'remote/develop' into onnx_e2e_cpu
kshpv Dec 1, 2022
7421d95
Update configs
kshpv Dec 2, 2022
0cd4a50
Update reference; Update generate_report
kshpv Dec 2, 2022
35d8deb
Add ov_ep_only parameter
kshpv Dec 4, 2022
fabbd1f
Fix bug
kshpv Dec 5, 2022
a2e09a0
Fix bug
kshpv Dec 5, 2022
bef3420
Fix pylint
kshpv Dec 6, 2022
2c7bc74
Add meta to report
kshpv Dec 7, 2022
a597067
Update html generation based on yattag
kshpv Dec 7, 2022
4fc835b
Update generating table
kshpv Dec 8, 2022
86e073b
Merge cells
kshpv Dec 8, 2022
2fea9bc
Fix typo
kshpv Dec 8, 2022
87e7ffd
Fix merging cells of the Provider names
kshpv Dec 9, 2022
35c9e47
Divide is_ov_ep_only to disable-ov-ep and enable-cpu-ep
kshpv Dec 9, 2022
f3fcfc0
Fix pylint
kshpv Dec 9, 2022
977fb2b
Fix typo
kshpv Dec 9, 2022
2dd6e83
Make getting row colors before generating final df
kshpv Dec 11, 2022
425b616
Update config to not override masks
kshpv Dec 13, 2022
5c7ebc6
Add commands for all providers
kshpv Dec 13, 2022
0822c79
Update Makefile
kshpv Dec 13, 2022
3c2b458
Add target_tags to quanitze only on ones
kshpv Dec 13, 2022
639ce68
Update Makefile
kshpv Dec 14, 2022
a0d45da
Add extra line
kshpv Dec 14, 2022
dd0ac14
Add --model_names option
kshpv Dec 20, 2022
759827a
Merge remote-tracking branch 'remote/develop' into onnx_e2e_cpu
kshpv Dec 20, 2022
a7f21c5
Update Makefile
kshpv Dec 20, 2022
12a94fb
logger
kshpv Dec 20, 2022
d0624c3
Merge remote-tracking branch 'remote/develop' into onnx_e2e_cpu
kshpv Jan 11, 2023
e9a3d97
Update yattag version
kshpv Jan 11, 2023
4ef409f
Remove e2e target in makefile
kshpv Jan 12, 2023
e4e2033
Update doc
kshpv Jan 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Update doc
  • Loading branch information
kshpv committed Jan 12, 2023
commit e4e2033dc2519a00a7c593693232efdb5b589779
15 changes: 8 additions & 7 deletions tests/onnx/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,15 @@ We provide two types of tests.
1. `--model-dir`: The directory path which includes ONNX Model ZOO models (.onnx files). See [#prepare-models](benchmarking/README.md#prepare-models) for details.
2. `--data-dir`: The directory path which includes datasets (ImageNet2012, COCO, Cityscapes, and VOC) [#prepare-models](benchmarking/README.md#prepare-models).
3. `--output-dir`: The directory path where the test results will be saved.
4. (Optional) `--ckpt-dir`: Directory path to save quantized models.
5. (Optional) `--anno-dir`: Directory path for dataset annotations. Please refer to [OpenVINO accuracy checker](https://github.com/openvinotoolkit/open_model_zoo/tree/master/tools/accuracy_checker).
6. (Optional) `--eval-size`: The number of samples for evaluation.
7. (Optional) `--ptq-size`: The number of samples for calibrating quantization parameters.
8. (Optional) `--enable-ov-ep`: If the parameter is set then the accuracy validation of the quantized models
4. (Optional) `--model-names`: String containing model names to test. Model name is the prefix of the name of AccuracyChecker config before the '.' symbol. Please, provide the model names using ' ' as a separator.
5. (Optional) `--ckpt-dir`: Directory path to save quantized models.
6. (Optional) `--anno-dir`: Directory path for dataset annotations. Please refer to [OpenVINO accuracy checker](https://github.com/openvinotoolkit/open_model_zoo/tree/master/tools/accuracy_checker).
7. (Optional) `--eval-size`: The number of samples for evaluation.
8. (Optional) `--ptq-size`: The number of samples for calibrating quantization parameters.
9. (Optional) `--enable-ov-ep`: If the parameter is set then the accuracy validation of the quantized models
will be enabled for OpenVINOExecutionProvider.
9. (Optional) `--disable-cpu-ep`: If the parameter is set then the accuracy validation of the quantized models
will be disabled for CPUExecutionProvider.
10. (Optional) `--disable-cpu-ep`: If the parameter is set then the accuracy validation of the quantized models
will be disabled for CPUExecutionProvider.

If you want to test the reference (not quantized) model accuracy - try the following command.

Expand Down
2 changes: 1 addition & 1 deletion tests/onnx/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ def pytest_addoption(parser):
)
parser.addoption(
"--model-names", type=str, default=None,
help="[e2e-test-onnx-model-zoo] String containing model names. "
help="[e2e-test-onnx-model-zoo] String containing model names to test. "
"Please, provide the model names using ' ' as a separator."
)
parser.addoption(
Expand Down