[OpenVINO] Enabling OpenVINO backend in test harness by suryasidd · Pull Request #17730 · pytorch/executorch

suryasidd · 2026-02-26T04:39:41Z

Summary

This PR enables OpenVINO test flows into the test harness to enable operator tests and models tests on OpenVINO backend.
Also updates OpenVINO version to 2026.0
Added Qwen2.5 support

* Fixed skip patterns logic

Fix openvino int8 tests

pytorch-bot · 2026-02-26T04:39:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17730

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 17 New Failures, 6 Cancelled Jobs, 1 Pending

As of commit 8d722b4 with merge base 331ac4a ():

NEW FAILURES - The following jobs have failed:

pull / test-samsung-models-linux / linux-job (gh)
RuntimeError: Command docker exec -t 970c3b9360a9a0dd98beaa0715f45af30c80604e997e756242389259596bd9f5 /exec failed with exit code 1
pull / test-samsung-quantmodels-linux / linux-job (gh)
RuntimeError: Command docker exec -t 0f711dc9f9707f1bd3164a75ac3d62df380d2651dd6f93a10341c3dec38bd880 /exec failed with exit code 1
pull / unittest-editable / macos / macos-job (gh)
backends/xnnpack/test/ops/test_conv2d.py::TestConv2d::test_fp16_conv2d
Test OpenVINO Backend / test-openvino / test-backend-linux (openvino, models) / linux-job (gh)
RuntimeError: Command docker exec -t 84e866d73d9a44fd995f1ac91762635e38a67932be4f7e17130480988cb2567e /exec failed with exit code 1
Test OpenVINO Backend / test-openvino / test-backend-linux (openvino, operators) / linux-job (gh)
RuntimeError: Command docker exec -t c3f8ab12f9ce16759a744a1a6a5b275b2890a8015abec9d6f2303c4d0ccb40d9 /exec failed with exit code 1
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a16w, models) / linux-job (gh)
RuntimeError: Command docker exec -t 66d5cc0d8ea265325037fed718238561b1db45cdcdd76b9158a963a8e59a73b5 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a16w, operators) / linux-job (gh)
RuntimeError: Command docker exec -t da4dbe66dfe747856481b1792d7dab77964cde76896304999de77221e005560b /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w_block, models) / linux-job (gh)
RuntimeError: Command docker exec -t 2dde0aade779c584c788cb518d49b6e1dc2eeafa305190327ffd250041738418 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w_block, operators) / linux-job (gh)
RuntimeError: Command docker exec -t c34c0fc825caf80a4192be30d002a30354ca6f2035c5419890728596b9c96ebf /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w, models) / linux-job (gh)
RuntimeError: Command docker exec -t e668479f315c7cdc2221c8e53b5cb66a4853a000afce8de00670a3d5d07c1244 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w, operators) / linux-job (gh)
RuntimeError: Command docker exec -t 8615198617bce47de1cf760319195a4fa3f9ce4c87dea38118df25aba4bb997b /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a8w, models) / linux-job (gh)
RuntimeError: Command docker exec -t 54499e277857ab30bc7be8a903c2b8725f1d2c6e47b8d1114c6cc76a9594f7da /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_16a8w, operators) / linux-job (gh)
RuntimeError: Command docker exec -t ea0766a8bdad23a2c9d1c6c8b083a2eb65bf06e24e143a6355ab806a4cf1ec34 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_8a8w, models) / linux-job (gh)
RuntimeError: Command docker exec -t 8b2c1b04fe59e1769c73cad27367ad62c26f58af2bd6b725a12bb3ef422d1dcf /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn_8a8w, operators) / linux-job (gh)
RuntimeError: Command docker exec -t dc2d98d694c0b2411fb3c7b4474a732616cb86a6a4f34970fb9e2ee3ed06f76c /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
RuntimeError: Command docker exec -t 6bd7a9960561ec3469ee1179c02f254c9a791e489690d4c3bed96a221d13d2d1 /exec failed with exit code 2
Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
RuntimeError: Command docker exec -t 7c7af496829eb4e97f2b6e82d1bc0c1804161ca2a9a4d24510ac1396e188b673 /exec failed with exit code 2

CANCELLED JOBS - The following jobs were cancelled. Please retry:

Test ARM Backend / test-arm / test-backend-linux (arm_tosa_fp, models) / linux-job (gh)
##[error]The operation was canceled.
Test ARM Backend / test-arm / test-backend-linux (arm_vgf_fp, models) / linux-job (gh)
##[error]The operation was canceled.
Test CoreML Backend / test-coreml / test-backend-macos (coreml_static_int8, operators) / macos-job (gh)
##[error]The operation was canceled.
Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan_static_int8_per_channel, operators) / linux-job (gh)
##[error]The operation was canceled.
Test Vulkan Backend / test-vulkan / test-backend-linux (vulkan, operators) / linux-job (gh)
##[error]The operation was canceled.
trunk / test-models-macos-coreml (mobilebert) / macos-job (gh)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-02-26T04:40:22Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Copilot

Pull request overview

This PR integrates the OpenVINO backend into the ExecuTorch backend test harness so operator/model suites can be exercised through new OpenVINO test flows and a dedicated CI workflow, while also updating the CI OpenVINO package version and adding an OpenVINO-specific preprocessing correctness pass.

Changes:

Add OpenVINO test flows (including an INT8/PTQ flow) and register them in the test suite flow registry.
Add OpenVINO tester plumbing (custom stages) and enable OpenVINO setup/build in CI test scripts + a scheduled workflow.
Add an OpenVINO preprocess pass to decompose div(rounding_mode="floor") into div + floor to match PyTorch semantics; extend quantize stage to plumb fold_quantize.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
backends/test/suite/flows/openvino.py	Defines OpenVINO and OpenVINO INT8 test flows (with skip patterns).
backends/test/suite/flow.py	Registers OpenVINO flows in `all_flows()` (but currently disables QNN registration).
backends/test/suite/conftest.py	Skips tests early when a flow’s `skip_patterns` match the test name.
backends/test/harness/stages/quantize.py	Adds `fold_quantize` option and forwards it into `convert_pt2e`.
backends/openvino/test/tester/tester.py	Introduces `OpenVINOTester` and OpenVINO-specific stage implementations.
backends/openvino/test/tester/init.py	Exports new OpenVINO tester symbols.
backends/openvino/scripts/openvino_build.sh	Changes install invocation used by OpenVINO build script.
backends/openvino/runtime/OpenvinoBackend.cpp	Adds scalar type mapping for `Double` → `f64`.
backends/openvino/preprocess.py	Runs the new floor-divide decomposition pass during preprocess.
backends/openvino/partitioner.py	Minor formatting change.
backends/openvino/_passes/decompose_floor_divide_pass.py	Adds `DecomposeFloorDividePass` implementation.
backends/openvino/_passes/init.py	Exposes the new pass from the package.
.github/workflows/test-backend-openvino.yml	Adds a scheduled/on-demand OpenVINO backend CI workflow.
.github/workflows/_test_backend.yml	Updates backend description to include OpenVINO.
.ci/scripts/test_backend.sh	Adds OpenVINO environment setup + build flag for OpenVINO flows.
.ci/scripts/setup-openvino.sh	Updates OpenVINO version/build used in CI setup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-26T04:46:07Z

backends/test/suite/flow.py

+    # try:
+    #     from executorch.backends.test.suite.flows.qualcomm import (
+    #         QNN_16A16W_TEST_FLOW,
+    #         QNN_16A4W_BLOCK_TEST_FLOW,
+    #         QNN_16A4W_TEST_FLOW,
+    #         QNN_16A8W_TEST_FLOW,
+    #         QNN_8A8W_TEST_FLOW,
+    #         QNN_TEST_FLOW,
+    #     )
+
+    #     flows += [
+    #         QNN_TEST_FLOW,
+    #         QNN_16A16W_TEST_FLOW,
+    #         QNN_16A8W_TEST_FLOW,
+    #         QNN_16A4W_TEST_FLOW,
+    #         QNN_16A4W_BLOCK_TEST_FLOW,
+    #         QNN_8A8W_TEST_FLOW,
+    #     ]
+    # except Exception as e:
+    #     logger.info(f"Skipping QNN flow registration: {e}")


The QNN flow registration has been commented out here, which removes all Qualcomm flows from all_flows() and will stop the test suite from generating flow_qnn* parameters/markers. If the intent is to enable OpenVINO in addition to existing backends, please restore the QNN try: block and add OpenVINO as a separate block (or keep both), rather than leaving QNN disabled in commented code.

Suggested change

# try:

# from executorch.backends.test.suite.flows.qualcomm import (

# QNN_16A16W_TEST_FLOW,

# QNN_16A4W_BLOCK_TEST_FLOW,

# QNN_16A4W_TEST_FLOW,

# QNN_16A8W_TEST_FLOW,

# QNN_8A8W_TEST_FLOW,

# QNN_TEST_FLOW,

# )

# flows += [

# QNN_TEST_FLOW,

# QNN_16A16W_TEST_FLOW,

# QNN_16A8W_TEST_FLOW,

# QNN_16A4W_TEST_FLOW,

# QNN_16A4W_BLOCK_TEST_FLOW,

# QNN_8A8W_TEST_FLOW,

# ]

# except Exception as e:

# logger.info(f"Skipping QNN flow registration: {e}")

try:

from executorch.backends.test.suite.flows.qualcomm import (

QNN_16A16W_TEST_FLOW,

QNN_16A4W_BLOCK_TEST_FLOW,

QNN_16A4W_TEST_FLOW,

QNN_16A8W_TEST_FLOW,

QNN_8A8W_TEST_FLOW,

QNN_TEST_FLOW,

)

flows += [

QNN_TEST_FLOW,

QNN_16A16W_TEST_FLOW,

QNN_16A8W_TEST_FLOW,

QNN_16A4W_TEST_FLOW,

QNN_16A4W_BLOCK_TEST_FLOW,

QNN_8A8W_TEST_FLOW,

]

except Exception as e:

logger.info(f"Skipping QNN flow registration: {e}")

backends/openvino/preprocess.py

Copilot · 2026-02-26T04:46:08Z

backends/openvino/scripts/openvino_build.sh


    # Build the package
-    ./install_executorch.sh --use-pt-pinned-commit
+    ./install_executorch.sh --minimal


Switching from --use-pt-pinned-commit to --minimal changes the PyTorch selection behavior (defaulting back to nightly) and can make OpenVINO builds/tests less reproducible. If the goal is to reduce dependencies, consider keeping the pinned PyTorch option while also using minimal installs (e.g., pass both flags or make it configurable).

backends/openvino/_passes/decompose_floor_divide_pass.py

Copilot

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-26T06:47:31Z

backends/openvino/_passes/decompose_floor_divide_pass.py

+            opset = _get_opset(node.target)
+            a, b = node.args[0], node.args[1]
+
+            a_dtype = _node_dtype(a)
+            is_integer = a_dtype is not None and not a_dtype.is_floating_point
+
+            with graph.inserting_before(node):
+                if is_integer:
+                    a_f = graph.call_function(
+                        opset["to_copy"], (a,), {"dtype": torch.float32}
+                    )
+                    b_f = graph.call_function(
+                        opset["to_copy"], (b,), {"dtype": torch.float32}
+                    )
+                    div_node = graph.call_function(opset["div"], (a_f, b_f))
+                    floored = graph.call_function(opset["floor"], (div_node,))
+                    result = graph.call_function(
+                        opset["to_copy"], (floored,), {"dtype": a_dtype}
+                    )


DecomposeFloorDividePass decides the integer-vs-float rewrite based only on a's dtype and then casts the result back to a_dtype. This is incorrect for mixed-dtype cases where div would promote to a floating output (e.g. int / float) and also risks wrong semantics when dtype metadata is missing. Consider using the node's output dtype (node.meta['val'].dtype) or torch.result_type(a_val, b_val) to choose the rewrite, and only cast back when the output dtype is integral.

Copilot · 2026-02-26T06:47:32Z

backends/openvino/_passes/decompose_floor_divide_pass.py

+        graph.eliminate_dead_code()
+        graph_module.recompile()
+        graph_module = super().call(graph_module).graph_module
+        return PassResult(graph_module, True)


This pass always returns PassResult(..., True) even if it didn't rewrite any nodes. That can cause unnecessary downstream work and makes pass pipelines harder to reason about. Track whether any div.Tensor_mode(..., rounding_mode='floor') nodes were replaced and return changed=False when none were found.

Copilot · 2026-02-26T06:47:32Z

backends/openvino/preprocess.py

+        for pass_cls in [DimOrderOpsRevertPass, DecomposeFloorDividePass]:
+            result = pass_cls()(edge_program.graph_module)
+            if result and result.graph_module:
+                edge_program._graph_module = result.graph_module


DecomposeFloorDividePass is now part of the OpenVINO preprocess pipeline, but there are no OpenVINO tests asserting correct floor-division semantics (especially for negative values where trunc vs floor differs). Please add a targeted unit/op test under backends/openvino/tests/ops that exercises torch.div(..., rounding_mode='floor') / torch.floor_divide on negative integer inputs and verifies OpenVINO output matches PyTorch.

Suggested change

edge_program._graph_module = result.graph_module

edge_program.graph_module = result.graph_module

Copilot · 2026-02-26T06:47:32Z

backends/test/suite/flow.py

    try:
-        from executorch.backends.test.suite.flows.qualcomm import (
-            QNN_16A16W_TEST_FLOW,
-            QNN_16A4W_BLOCK_TEST_FLOW,
-            QNN_16A4W_TEST_FLOW,
-            QNN_16A8W_TEST_FLOW,
-            QNN_8A8W_TEST_FLOW,
-            QNN_TEST_FLOW,
+        from executorch.backends.test.suite.flows.openvino import (
+            OPENVINO_INT8_TEST_FLOW,
+            OPENVINO_TEST_FLOW,
        )

        flows += [
-            QNN_TEST_FLOW,
-            QNN_16A16W_TEST_FLOW,
-            QNN_16A8W_TEST_FLOW,
-            QNN_16A4W_TEST_FLOW,
-            QNN_16A4W_BLOCK_TEST_FLOW,
-            QNN_8A8W_TEST_FLOW,
+            OPENVINO_TEST_FLOW,
+            OPENVINO_INT8_TEST_FLOW,
        ]
    except Exception as e:
-        logger.info(f"Skipping QNN flow registration: {e}")
+        logger.info(f"Skipping OpenVINO flow registration: {e}")
+
+    # try:
+    #     from executorch.backends.test.suite.flows.qualcomm import (
+    #         QNN_16A16W_TEST_FLOW,
+    #         QNN_16A4W_BLOCK_TEST_FLOW,
+    #         QNN_16A4W_TEST_FLOW,
+    #         QNN_16A8W_TEST_FLOW,
+    #         QNN_8A8W_TEST_FLOW,
+    #         QNN_TEST_FLOW,
+    #     )
+
+    #     flows += [
+    #         QNN_TEST_FLOW,
+    #         QNN_16A16W_TEST_FLOW,
+    #         QNN_16A8W_TEST_FLOW,
+    #         QNN_16A4W_TEST_FLOW,
+    #         QNN_16A4W_BLOCK_TEST_FLOW,
+    #         QNN_8A8W_TEST_FLOW,
+    #     ]
+    # except Exception as e:
+    #     logger.info(f"Skipping QNN flow registration: {e}")


all_flows() no longer registers the Qualcomm/QNN flows (the entire QNN block is commented out), which will break QNN backend CI (e.g. test-backend-qnn.yml runs -m flow_qnn...) and removes QNN from the harness. Please restore the QNN flow registration and add the OpenVINO flows as an additional try-block rather than replacing QNN.

Copilot

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-26T07:48:06Z

.github/workflows/test-backend-openvino.yml

+  pull_request:
+    paths:
+      - .github/workflows/test-backend-openvino.yml
+      - .github/workflows/_test_backend.yml


pull_request.paths only watches workflow files, so changes to the OpenVINO backend/test harness (e.g. backends/openvino/**, .ci/scripts/**, backends/test/**) won’t trigger this workflow on PRs. This makes it easy for OpenVINO regressions to merge without any PR signal. Consider expanding paths similarly to other backend workflows (e.g. xnnpack) to include the relevant backend + harness directories.

Suggested change

- .github/workflows/_test_backend.yml

- .github/workflows/_test_backend.yml

- backends/openvino/**

- .ci/scripts/**

- backends/test/**

Copilot · 2026-02-26T07:48:06Z

.github/workflows/test-backend-openvino.yml

+    uses: ./.github/workflows/_test_backend.yml
+    with:
+      backend: openvino
+      flows: '["openvino"]'


This workflow only runs the openvino flow, but the PR also registers an openvino_int8 flow in the test harness. If INT8 is intended to be supported/kept green, consider adding it to the flows list here so it gets CI coverage (or document why it’s intentionally excluded).

Suggested change

flows: '["openvino"]'

flows: '["openvino", "openvino_int8"]'

Copilot · 2026-02-26T07:48:06Z

examples/openvino/aot_optimize_and_infer.py

    # Run the main function with parsed arguments
-    # Disable nncf patching as export of the patched model is not supported.
-    with nncf.torch.disable_patching():
-        main(
-            args.suite,
-            args.model,
-            args.input_shape,
-            args.export,
-            args.model_file_name,
-            args.quantize,
-            args.validate,
-            args.dataset,
-            args.device,
-            args.batch_size,
-            args.infer,
-            args.num_iter,
-            args.warmup_iter,
-            args.input_tensor_path,
-            args.output_tensor_path,
-        )
+    main(
+        args.suite,
+        args.model,
+        args.input_shape,
+        args.export,


The removal of nncf.torch.disable_patching() means this script now runs export/lowering without explicitly disabling NNCF’s Torch patching. Other OpenVINO-related examples in this repo still guard export with disable_patching() due to known side effects. If that concern still applies, consider reintroducing the context manager (or lazily importing nncf/quantize_model only when --quantize is used) to avoid export failures when NNCF is present.

suryasidd and others added 13 commits December 23, 2025 08:07

Added openvino to test harness

5ede955

Modified the tester functions in init

6921a1c

Fix openvino int8 tests

dfe13b4

Merge branch 'main' into openvino_prod

2f09cce

Added floor divide decomposition

a27a5e2

* Fixed skip patterns logic

Merge branch 'main' into openvino_prod

dbd9d2d

Merge branch 'main' into openvino_prod

e282f84

Merge branch 'main' into openvino_prod

e29ffca

Merge pull request #2 from daniil-lyakhov/openvino_prod

6eb2233

Fix openvino int8 tests

Cleanup debug statements

f7a600d

Merge branch 'main' into openvino_prod

1f9673b

Added openvino backend flow to ci

f30ab4c

Fixed linter issues

6a860ba

Copilot AI review requested due to automatic review settings February 26, 2026 04:39

suryasidd requested a review from cccclai as a code owner February 26, 2026 04:39

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Feb 26, 2026

Copilot started reviewing on behalf of suryasidd February 26, 2026 04:40 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

Added Qwen2.5 support for OpenVINO

6c8b7be

This comment was marked as duplicate.

Sign in to view

Added check for floor divide pass

a8f43cb

Copilot AI review requested due to automatic review settings February 26, 2026 06:37

Copilot started reviewing on behalf of suryasidd February 26, 2026 06:37 View session

Merge branch 'main' into openvino_prod

9df844b

Copilot AI reviewed Feb 26, 2026

View reviewed changes

suryasidd added 2 commits February 25, 2026 23:27

Removed nncf disable patching

31e7310

Fixed linter issues

8d722b4

Copilot AI review requested due to automatic review settings February 26, 2026 07:39

Copilot started reviewing on behalf of suryasidd February 26, 2026 07:39 View session

Copilot AI reviewed Feb 26, 2026

View reviewed changes

	edge_program._graph_module = result.graph_module
	edge_program.graph_module = result.graph_module

-      - .github/workflows/_test_backend.yml
+      - .github/workflows/_test_backend.yml
+      - backends/openvino/**
+      - .ci/scripts/**
+      - backends/test/**

Conversation

suryasidd commented Feb 26, 2026

Summary

Uh oh!

pytorch-bot bot commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17730

❌ 17 New Failures, 6 Cancelled Jobs, 1 Pending

Uh oh!

github-actions bot commented Feb 26, 2026

This PR needs a release notes: label

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as duplicate.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pytorch-bot bot commented Feb 26, 2026 •

edited

Loading

This PR needs a `release notes:` label