Skip to content

Select model package variants in EP priority order #33508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
maxim-davgalev wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Select model package variants in EP priority order #33508

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a8fbff4
Select model package variants in EP priority order
maxim-davgalev Sep 30, 2026
9e74382
Test that model package sessions use the variant's device type
maxim-davgalev Oct 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 37 additions & 1 deletion include/onnxruntime/core/session/onnxruntime_c_api.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8933,7 +8933,11 @@ struct OrtInteropApi {
struct OrtModelPackageApi {
/** \brief Create model package options from session options.
*
* Captures the execution provider configuration used to select a component variant.
* Records the EP configuration used to choose a variant for a model package component and create sessions.
* EP priority comes from the selection policy or the order in which EPs were added.
* For explicitly added EPs, their provider factories are kept for session creation.
* ORT also stores pointers to the selected OrtEpDevice instances for plugin EPs added with
* OrtApi::SessionOptionsAppendExecutionProvider_V2 and for EPs selected by a policy.
*
* \param[in] env The ORT environment.
* \param[in] session_options Session options containing the execution provider configuration.
Expand Down Expand Up @@ -9100,6 +9104,17 @@ struct OrtModelPackageApi {
_Outptr_ const char** out_path);

/** \brief Select a component and resolve its best matching variant.
*
* EPs are considered in priority order, as determined by the selection policy or the order in which they were added
* to the session options. For the first EP with matching variants, the variant with the highest compatibility score
* is selected.
* If an EP has no variants or all its variants are incompatible, selection continues to the next EP.
* If ORT cannot check a variant's compatibility and no other variant matches that EP, selection returns an error
* instead of considering EPs with lower priority. The error identifies the EP.
* This can happen when the variant requires a device but no devices were captured for the EP, or when the EP's
* compatibility callback returns an error.
* Without captured devices, the built-in CPU EP can still match a variant that requires a CPU, but no other device
* type.
*
* The returned component context is independent of `context` and must be released with
* ReleaseModelPackageComponentContext().
Expand Down Expand Up @@ -9164,6 +9179,27 @@ struct OrtModelPackageApi {
* provider options are not applied; package path-valued session options are still added when the caller did not
* provide them.
*
* If the effective options contain no EP factories, ONNX Runtime adds only the EP targeted by the selected variant.
* It uses the captured devices that match the variant's declared device type, or all captured devices if no type is
* declared. The EP is added as in OrtApi::SessionOptionsAppendExecutionProvider_V2. The usual CPU EP fallback still
* applies. ORT does not run the selection policy during session creation.
* Device defaults are used only for EP options that have not already been set.
* Custom operator domains provided by the EP are registered before loading the model.
* If no OrtEpDevice instances were captured for the EP, ORT uses the provider factory saved from the session options
* used to create the model package options. The built-in CPU EP does not require a saved factory.
* On the default path, non-empty package provider options cause an error if no devices were captured.
*
* If the caller supplies EP factories, ORT uses that list.
* If the selected EP is not the built-in CPU EP, at least one factory in the list must create it.
*
* The CPU registration APIs behave differently here:
* - Adding the built-in ORT CPU EP as "CPU" through OrtApi::SessionOptionsAppendExecutionProvider does not add
* a provider factory to the options.
* - OrtSessionOptionsAppendExecutionProvider_CPU adds a CPU EP factory.
*
* ORT automatically adds the built-in CPU EP if it has not already been registered.
* If any other selected EP cannot be created or registered, session creation fails.
*
* \param[in] env The ORT environment.
* \param[in] context The selected component context.
* \param[in] session_options Optional caller-provided session options.
Expand Down
120 changes: 86 additions & 34 deletions onnxruntime/core/session/model_package/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ adds three things on top**:

1. The `executor_info["ort"]` payload schema (this is ORT's slot in the
variant body).
2. The variant selection algorithm, which queries each execution provider
factory and picks the highest-scoring variant.
2. The variant selection algorithm considers execution providers in priority
order. For the first EP with matching variants, it selects the variant
with the highest compatibility score.
3. The stable `OrtModelPackageApi` C API table that wraps the library and
exposes session creation through `OrtApi::GetModelPackageApi`.

Expand All @@ -25,7 +26,7 @@ itself never links against ORT.

| File | Responsibility |
| ------------------------------------- | -------------- |
| `model_package_context.h/.cc` | Translates the `model_package` library's C info tree into ORT-internal C++ structs (`ModelPackageInfo`, `ComponentInfo`, `VariantInfo`, `VariantModelInfo`). Parses the `executor_info["ort"]` payload. Owns `ModelPackageContext` (package-level) and `ModelPackageComponentContext` (per-component, with selected variant and captured provider factory). |
| `model_package_context.h/.cc` | Translates the `model_package` library's C info tree into ORT-internal C++ structs (`ModelPackageInfo`, `ComponentInfo`, `VariantInfo`, `VariantModelInfo`). Parses the `executor_info["ort"]` payload. Owns `ModelPackageContext` (package-level) and `ModelPackageComponentContext` (per-component, with the selected variant, the captured EPs, and their provider factories saved for session creation). |
| `model_package_options.h/.cc` | `ModelPackageOptions` snapshots EP intent (factories, devices, EP-name list) from an `OrtSessionOptions` when `OrtModelPackageApi::CreateModelPackageOptionsFromSessionOptions` is called. Drives variant selection and provider construction. |
| `model_package_variant_selector.h/.cc`| `VariantSelector::SelectVariant` picks the best variant from a component given the EP list. Uses `OrtEpFactory::ValidateCompiledModelCompatibilityInfo`. |

Expand Down Expand Up @@ -78,40 +79,56 @@ variant can carry per-consumer payloads side by side.

## Variant selection

`ModelPackageOptions(env, session_options)` captures the **EP intent**: the
first execution provider selected from the session options or EP policy, plus
its associated `OrtEpDevice` / `OrtHardwareDevice` metadata. Explicit provider
factories are retained so legacy providers can also be recreated after the
original options are released.

`VariantSelector::SelectVariant(component, ep_infos, &selected)` then walks
the component's variants and picks the best match:

1. Use only the **first** EP. A policy may rank several EPs; callers that
need a specific EP should put it first. Selection does not fall through
to other EPs when that EP has no matching variant.
`ModelPackageOptions(env, session_options)` records the EPs in priority
order and saves their device information and metadata. When a policy
selects the EPs, ORT records the selected devices and the factory used
to create each EP. For explicitly added EPs, device information comes
from the provider's `GetEpDevices()`, which currently returns devices
only for plugin EPs. ORT saves the provider factories from the session
options so it can create legacy EPs again after those options are
released.

`VariantSelector::SelectVariant` selects the best matching variant for
the component and identifies its target EP:

1. Consider the captured EPs in priority order, as determined by the
selection policy or the order in which they were added to the session
options. Select the first EP with a matching variant.
2. For each variant, require `variant.ep == ep_info.ep_name`.
3. If `variant.device` is set (`"cpu"` / `"gpu"` / `"npu"`), require it to
match at least one of the EP's `OrtHardwareDevice` entries. The built-in
CPU EP also matches `"device": "cpu"` without device metadata. Other EPs
without hardware metadata can only match variants with no device constraint;
a default memory-device type is not sufficient to infer the target hardware.
match at least one of the EP's `OrtHardwareDevice` entries.
The built-in CPU EP matches `"device": "cpu"` even without captured
devices, but does not match other device types.
If captured devices exist but none has the required type, the variant
does not match.
If no devices were captured for an EP other than the built-in CPU EP,
ORT cannot check the requirement. If no other variant matches that EP,
selection returns an error instead of considering EPs with lower priority.
A default memory device type is not enough to determine the target
hardware.
4. If both pass, call `OrtEpFactory::ValidateCompiledModelCompatibilityInfo`
with `variant.compatibility_string`. The EP returns an
`OrtCompiledModelCompatibility` enum which maps to a score:

| Enum | Score |
| -------------------------------------------- | ----- |
| `EP_SUPPORTED_OPTIMAL` | 100 |
| `EP_SUPPORTED_PREFER_RECOMPILATION` | 50 |
| `EP_NOT_APPLICABLE` (or EP too old / no ABI) | 0 |
| `EP_UNSUPPORTED` | rejected |
| Enum | Score |
| ------------------------------------------------------------------ | ----- |
| `EP_SUPPORTED_OPTIMAL` | 100 |
| `EP_SUPPORTED_PREFER_RECOMPILATION` | 50 |
| `EP_NOT_APPLICABLE` (or EP too old / no ABI / no factory captured) | 0 |
| `EP_UNSUPPORTED` | rejected |

`EP_UNSUPPORTED` means the check succeeded but the variant is not
compatible. If the callback itself returns an error, compatibility
remains unknown. If no other variant matches that EP, selection returns
an error instead of considering EPs with lower priority.

5. Pick the highest-scoring matching variant. Manifest declaration order
breaks ties.
5. Select the matching variant with the highest compatibility score for
that EP. If several variants have the same score, select the first one
in the manifest.

If no variant matches, `SelectComponent` fails with "No suitable model
variant found for the configured execution providers."
variant found for the configured execution providers." If selection stops at
an EP as described in steps 3 and 4, the error also names that EP.

ORT does **not** parse `compatibility_string`. The EP owns the format and
may encode multiple sub-targets (SoC ids, ISA flags, etc.) into the single
Expand Down Expand Up @@ -147,11 +164,46 @@ EP was selected. Two paths:
selection / EP discovery; its non-EP session configuration is not
implicitly inherited.

When no explicit provider factories are supplied, either path uses the captured
EP, fills missing device-default options, and registers its custom-op domains.
The selection policy is not re-run. When explicit factories are supplied, the
caller must keep them compatible with the selected variant. Failure to recreate
the selected EP is an error, not an implicit switch to CPU.
When the effective session options contain no EP factories, ORT adds
only the EP targeted by the selected variant, as in
`SessionOptionsAppendExecutionProvider_V2`. ORT uses the devices captured
for that EP. If the variant specifies a device type, ORT uses only
devices of that type. Otherwise, it uses all captured devices for that EP.

Device defaults are used only for EP options that have not already been
set, including options from the variant's `session_options`. Custom
operator domains provided by the EP are registered before loading the
model. On the default path, the variant's `provider_options` are passed
to the EP as in `SessionOptionsAppendExecutionProvider_V2`. For example,
CoreML rejects an `MLComputeUnits` value that enables an accelerator
outside the device selection.

If no `OrtEpDevice` instances were captured for the EP, ORT uses the
provider factory saved from the session options used to create the
model package options. The built-in CPU EP does not require a saved
factory. On the default path, non-empty package `provider_options`
cause an error if no devices were captured.

No other EP is added, apart from the usual ORT CPU EP fallback. This
prevents an EP without a matching variant from taking nodes of a model
built for another EP. `CreateSession` uses the EP chosen for the
variant, even if the session options specify a selection policy.

If the caller supplies EP factories, ORT uses that list. The caller must
ensure compatibility with the selected variant. If the selected EP is not
the built-in CPU EP, at least one factory in the list must create it.
The CPU registration APIs behave differently:

- Adding the built-in ORT CPU EP as "CPU" through
`SessionOptionsAppendExecutionProvider` does not add a provider factory
to the options.
- `OrtSessionOptionsAppendExecutionProvider_CPU` adds a CPU EP factory.

ORT automatically adds the built-in CPU EP if it has not already
been registered. If any other selected EP cannot be created or
registered, session creation fails. After successful registration,
unsupported nodes may still use the usual ORT CPU EP fallback unless
it is disabled.

Each session creates its own EP instances through the normal ORT factory
registration path. A component context can create multiple sessions sequentially;
Expand Down Expand Up @@ -205,7 +257,7 @@ const OrtModelPackageApi* package_api = ort->GetModelPackageApi();

OrtSessionOptions* so = nullptr;
ort->CreateSessionOptions(&so);
ort->SessionOptionsAppendExecutionProvider(so, "CUDAExecutionProvider", nullptr, nullptr, 0);
ort->SessionOptionsSetEpSelectionPolicy(so, OrtExecutionProviderDevicePolicy_PREFER_GPU);

OrtModelPackageOptions* mp_opts = nullptr;
package_api->CreateModelPackageOptionsFromSessionOptions(env, so, &mp_opts);
Expand Down
70 changes: 50 additions & 20 deletions onnxruntime/core/session/model_package/model_package_context.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
#include "core/framework/error_code_helper.h"
#include "core/graph/constants.h"
#include "core/providers/providers.h"
#include "core/session/abi_devices.h"
#include "core/session/model_package/model_package_context.h"
#include "core/session/model_package/model_package_options.h"
#include "core/session/model_package/model_package_variant_selector.h"
Expand Down Expand Up @@ -94,7 +95,7 @@ ModelPackageComponentContext::ModelPackageComponentContext(const std::string& co
owned_ep_infos_(options.EpInfos()),
execution_devices_(options.ExecutionDevices()),
devices_selected_(options.DevicesSelected()),
from_policy_(options.FromPolicy()) {
selection_policy_(options.SelectionPolicy()) {
// Point the span at our owned copy.
ep_infos_ = gsl::span<const VariantSelectionEpInfo>(owned_ep_infos_);
}
Expand All @@ -112,27 +113,29 @@ Status ModelPackageComponentContext::ResolveVariant() {
"ModelPackageComponentContext::ResolveVariant requires non-empty ep_infos "
"(from ModelPackageOptions or explicit span).");

return ResolveVariantImpl(ep_infos_);
return ResolveVariantImpl();
}

Status ModelPackageComponentContext::ResolveVariantImpl(gsl::span<const VariantSelectionEpInfo> ep_infos) {
std::optional<VariantInfo> selected_variant;
Status ModelPackageComponentContext::ResolveVariantImpl() {
std::optional<VariantSelection> selection;
VariantSelector selector;
ORT_RETURN_IF_ERROR(selector.SelectVariant(*this, ep_infos, selected_variant));
ORT_RETURN_IF_ERROR(selector.SelectVariant(*this, ep_infos_, selection));

ORT_RETURN_IF(!selected_variant.has_value(),
ORT_RETURN_IF(!selection.has_value(),
"No suitable model variant found for the configured execution providers.");

ORT_RETURN_IF(selected_variant->component_name != component_model_name_,
const VariantInfo& selected_variant = selection->variant;
ORT_RETURN_IF(selected_variant.component_name != component_model_name_,
"Selected variant's component model name does not match context's component model name.");

component_model_info_.selected_variant_index.reset();
selected_ep_index_.reset();

size_t matched_variants = 0;
for (size_t i = 0; i < component_model_info_.variants.size(); ++i) {
if (component_model_info_.variants[i].variant_name == selected_variant->variant_name) {
if (component_model_info_.variants[i].variant_name == selected_variant.variant_name) {
component_model_info_.selected_variant_index = i;
component_model_info_.variants[i] = *selected_variant;
component_model_info_.variants[i] = selected_variant;
++matched_variants;
}
}
Expand All @@ -142,9 +145,32 @@ Status ModelPackageComponentContext::ResolveVariantImpl(gsl::span<const VariantS
ORT_RETURN_IF(matched_variants > 1,
"Selected variant matched multiple variants; selection is ambiguous.");

selected_ep_index_ = selection->ep_index;
return Status::OK();
}

const VariantSelectionEpInfo* ModelPackageComponentContext::SelectedEpInfo() const {
return selected_ep_index_.has_value() ? &ep_infos_[*selected_ep_index_] : nullptr;
}

InlinedVector<const OrtEpDevice*> ModelPackageComponentContext::SelectedEpDevices() const {
InlinedVector<const OrtEpDevice*> ep_devices;
if (!selected_ep_index_.has_value()) {
return ep_devices;
}

// selected_ep_index_ is set only after a variant has been selected.
const std::optional<std::string>& device =
component_model_info_.variants[*component_model_info_.selected_variant_index].ep_compatibility.device;
for (const OrtEpDevice* ep_device : ep_infos_[*selected_ep_index_].ep_devices) {
if (!device.has_value() || MatchesVariantDevice(ep_device->device, *device)) {
ep_devices.push_back(ep_device);
}
}

return ep_devices;
}

Status ModelPackageComponentContext::GetSelectedVariantFolderPath(const std::filesystem::path*& out_folder_path) const {
out_folder_path = nullptr;

Expand Down Expand Up @@ -313,21 +339,25 @@ Status ModelPackageComponentContext::ConfigureSessionOptions(
return Status::OK();
}

ORT_RETURN_IF(ep_infos_.empty(), "No execution provider was selected for the model package.");
const auto& ep_info = ep_infos_[0];
const VariantSelectionEpInfo* ep_info = SelectedEpInfo();
ORT_RETURN_IF(ep_info == nullptr, "No execution provider was selected for the model package.");
const InlinedVector<const OrtEpDevice*> ep_devices = SelectedEpDevices();
ProviderPolicyContext policy_context;
ORT_RETURN_IF_ERROR(policy_context.AddEpDefaultOptionsToSession(options.value, ep_info.ep_devices));
ORT_RETURN_IF_ERROR(AddEpCustomDomainsToSessionOptions(ep_info.ep_devices, options));

if (ep_info.provider_factory) {
options.provider_factories.push_back(ep_info.provider_factory);
} else if (!ep_info.ep_devices.empty()) {
ORT_RETURN_IF_ERROR(policy_context.AddEpDefaultOptionsToSession(
options.value, std::vector<const OrtEpDevice*>(ep_devices.begin(), ep_devices.end())));
ORT_RETURN_IF_ERROR(AddEpCustomDomainsToSessionOptions(ep_devices, options));

// The saved provider factory is configured for the full device list.
// Use it only if the selected variant does not exclude any device from that list.
if (ep_info->provider_factory && ep_devices.size() == ep_info->ep_devices.size()) {
options.provider_factories.push_back(ep_info->provider_factory);
} else if (!ep_devices.empty()) {
std::unique_ptr<IExecutionProviderFactory> provider_factory;
ORT_RETURN_IF_ERROR(CreateIExecutionProviderFactoryForEpDevices(env, ep_info.ep_devices, provider_factory));
ORT_RETURN_IF_ERROR(CreateIExecutionProviderFactoryForEpDevices(env, ep_devices, provider_factory));
Comment thread
maxim-davgalev marked this conversation as resolved.
options.provider_factories.push_back(std::move(provider_factory));
} else {
ORT_RETURN_IF(ep_info.ep_name != kCpuExecutionProvider,
"Cannot recreate the selected execution provider: ", ep_info.ep_name);
ORT_RETURN_IF(ep_info->ep_name != kCpuExecutionProvider,
"Cannot recreate the selected execution provider: ", ep_info->ep_name);
}

// Variant selection has already resolved the policy. Do not select a different EP at session creation.
Expand Down
Loading