Repository navigation
Select model package variants in EP priority order - #33508
Open
maxim-davgalev wants to merge 2 commits into
Open
maxim-davgalev wants to merge 2 commits into
maxim-davgalev wants to merge 2 commits into
Conversation
Capture EPs in priority order instead of considering only the first. For the first EP with matching variants, select the variant with the highest compatibility score. Compatibility scores are used only to choose between variants for the same EP and do not override EP priority: once an EP with a matching variant is found, variants for later EPs are not considered, even if they have higher scores. This allows packages without a variant for the first EP to use a matching variant for a later EP, including the ORT CPU EP, instead of failing variant selection.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Cross-platform provider reconstruction needs human validation, and mixed-device factory replacement lacks regression coverage.
1 open issue
What changed in this PR
Updates ONNX Runtime model packages to select variants by execution provider (EP) priority, then compatibility score within the first matching EP.
Changes:
- Captures ordered EPs and recreates the selected provider for sessions.
- Adds selection, fallback, and error-path tests.
- Documents selection and session-creation rules.
| File | Description |
|---|---|
| onnxruntime/test/autoep/test_model_package.cc | Adds selection and session regressions. |
| onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc | Enables simulated compatibility errors. |
| onnxruntime/core/session/utils.h | Updates EP capture interface. |
| onnxruntime/core/session/utils.cc | Captures all providers and checks selected EP registration. |
| onnxruntime/core/session/provider_policy_context.h | Exposes per-provider selections and policy telemetry. |
| onnxruntime/core/session/provider_policy_context.cc | Preserves selected devices per provider. |
| onnxruntime/core/session/model_package/README.md | Documents priority and session behavior. |
| onnxruntime/core/session/model_package/model_package_variant_selector.h | Updates selection contract. |
| onnxruntime/core/session/model_package/model_package_variant_selector.cc | Implements priority-ordered selection. |
| onnxruntime/core/session/model_package/model_package_options.h | Stores selection policy. |
| onnxruntime/core/session/model_package/model_package_options.cc | Captures ordered factories and devices. |
| onnxruntime/core/session/model_package/model_package_context.h | Tracks the selected EP. |
| onnxruntime/core/session/model_package/model_package_context.cc | Filters devices and configures selected providers. |
| onnxruntime/core/session/model_package_api.cc | Applies selected-provider options and telemetry. |
| include/onnxruntime/core/session/onnxruntime_c_api.h | Clarifies API documentation without changing signatures. |
Review effort: Balanced
Add a test with one CPU device and two GPU devices to verify that CreateSession creates the selected EP only for devices matching the variant's device type. The test covers both the default CreateSession path and the path where the caller provides session options without EP factories.
Contributor
Author
|
Hi Tianlei Wu (@tianleiwu), would you be the right person to review this PR, or could you suggest an appropriate reviewer? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Model package variant selection previously considered only the first EP captured from the session options.
A package without a variant for that EP failed with "No suitable model variant found", even when a later EP had
a matching variant. For example, with a CPU plugin EP registered, PREFER_CPU can select [plugin EP, ORT CPU EP]. A package containing only an ORT CPU EP variant was therefore rejected.
This PR considers EPs in priority order: the order determined by a policy or the order in which EPs were added to
the session options. For the first EP with matching variants, it selects the variant with the highest compatibility score. Compatibility scores are used only to choose between variants for the same EP and do not override EP priority: once an EP with a matching variant is found, variants for later EPs are not considered, even if they have higher scores. If several matching variants for that EP have the highest score, the first one listed in the manifest is selected.
The public C API documentation and model package README describe these rules.
Tests