build(python): move the Linux NPU worker to 3.14 — OpenVINO now ships cp314 wheels

Part of #13743 · Wave 3 — depends on #13744, #13745, #13746. Do not bump an interpreter on a deploy path that is already broken.

Move the Linux native NPU worker from Python 3.11 to 3.14.

The blocker has expired

#10877 pinned this to 3.11 because OpenVINO shipped no 3.14 wheels. Measured against PyPI on 2026-08-08:

Package	Latest	Python tags	3.14?
`openvino`	2026.3.0	cp310, cp311, cp312, cp313, cp314	✅
`torch`	2.13.0	cp310, cp311, cp312, cp313, cp314	✅
`numpy`	2.5.1	cp312, cp313, cp314	✅
`openvino-dev`	2024.6.0	`py3-none-any` (universal)	✅

autobot-npu-worker/requirements.txt needs openvino>=2026.2.1 + torch + numpy — all satisfied by cp314-carrying releases — and does not include onnxruntime-openvino, the one package still missing cp314.

Scope

roles/npu-worker/tasks/main.yml (venv_python: python3.11, the python3.11 -m venv step, and the apt python3.11* packages), plus the 3.11 references in deploy-native-services.yml, deploy-full.yml, configure-python-provision-permissions.yml and provision-local-python.yml.

Not in scope

The Windows NPU worker (autobot-npu-worker/resources/windows-npu-worker/) pins onnxruntime-openvino, which has no cp314 (1.24.1 ships cp311–cp313). It stays on its current interpreter. Do not bundle it in.

Acceptance criteria

The Linux NPU worker venv is built with 3.14 and its full requirements set installs.
The NPU worker starts and serves inference on 3.14 — verified on a host, not by a successful pip install. OpenVINO failures are native-extension failures that appear at import or first inference, so a green install proves nothing.
ensure_venv_version.yml is applied so an existing 3.11 venv is replaced rather than reused.
Verified by deliberate failure: the guard rejects a venv left at the old version.

Tracked under epic #13743 (Python 3.14 consistency). Audit with the measurements: docs/audit/python_314_consistency.md, landing in PR #13750.

build(python): move the Linux NPU worker to 3.14 (#13747)

Implemented and merged — deliberately left OPEN.

PR #13800 squash-merged into Dev_new_gui. Verified in the base branch:

$ git show origin/Dev_new_gui:.../roles/npu-worker/tasks/main.yml | grep python3.1
- name: NPU worker | Install target Python interpreter (python3.14)
    venv_python: python3.14
    cmd: python3.14 -m venv {{ npu_install_dir }}/venv

$ git show origin/Dev_new_gui:.../playbooks/deploy-native-services.yml | grep -c python3.11
0

Why this stays open

AC 2 asks for the worker to start and serve inference on 3.14, verified on a host — not by a successful pip install. That distinction is the whole point here: OpenVINO failures are native-extension failures that surface at import or first inference, so every check I can run from here proves the dependency set solves, not that the NPU runs.

What is proven:

openvino 2026.3.0, torch 2.13.0, numpy 2.5.1 all publish cp314 (PyPI JSON API, 2026-08-08)
The worker's full requirements.txt resolves on 3.14 with no conflicts and no sdist fallback
ansible-playbook --syntax-check passes on all four changed playbooks and on provision-fleet-roles.yml, which applies the role

What is not proven, and needs a host:

The 3.14 venv builds on an NPU box and OpenVINO imports
The worker serves an inference request
ensure_venv_version actually removed the pre-existing 3.11 venv rather than leaving 3.11-built binaries in a 3.14 venv — this is the failure mode to watch, and it reports success at deploy time either way

To close

Deploy to an NPU host (via the builtin code-sync path) and attach the interpreter version plus a served inference. AC 4 also asks for a deliberate failure: confirm the guard rejects a venv left at the old version.

Do not treat this as unblocking #13748

The black py314 bump needs the worker to be running 3.14, not configured for it. Until a host is actually redeployed, that reformat writes PEP 758 syntax into shared code the still-3.11 worker imports — a SyntaxError at import, in production.

PR #13800 (merged to Dev_new_gui) references this issue with a close keyword.

build(python): move the Linux NPU worker to 3.14 (#13747)

If this issue is fully resolved, close it manually. If work remains, no action is needed.

Reopened — the state was wrong, not the work.

This was closed completed at 18:11 on 2026-08-09 with no host evidence anywhere on the issue: no deploy record, no interpreter version read off an NPU box, no served inference. The bot comment immediately above it says the quiet part — "If this issue is fully resolved, close it manually" — and the close keyword in PR #13800's body is what surfaced it. The code is merged and correct; what is missing is proof it runs.

AC 2 is explicit that a successful pip install does not count, and that is not pedantry: OpenVINO failures are native-extension failures that appear at import or first inference. Everything verifiable from a dev box — the full requirements graph resolving on 3.14, ansible-playbook --syntax-check on every changed playbook — shows the configuration is right. None of it shows the NPU works.

Closing on merge evidence is the exact pattern that got #12513 reopened twice.

Outstanding, needs a host:

3.14 venv builds on an NPU box and import openvino succeeds
The worker serves an inference request
ensure_venv_version actually removed the pre-existing 3.11 venv — the dangerous case is 3.11-built binaries left inside a 3.14 venv, which reports success at deploy time and fails later at import
AC 4's deliberate failure: the guard rejects a venv left at the old version

If you closed this knowingly and accept the risk, close it again and I will not reopen — it is your call, and I would rather say so once than keep flipping it. Recording the reasoning so the next reader knows which of the two happened.

test(ansible): verify ensure_venv_version rejects a stale venv by del…

fix(python): correct stale python3.11 NPU-worker declarations to 3.14 (…

Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current origin/main; leaving open.

3 of 4 ACs MET:

python3.14 venv build: main.yml:167,171.
ensure_venv_version wiring: main.yml:161-167.
Deliberate-failure guard test: test_ensure_venv_version.yml.

4th AC (starts and serves inference on 3.14, host-verified) NOT MET — the delivering PR (#13800) says so itself: "I have no NPU host to verify against, so the issue stays open pending that deploy evidence." That's a live-host verification I can't gather myself without a write probe against production, which is out of scope here.

Host evidence, gathered read-only on the deployment node — AC1 met, AC2 not met here

This issue gates pyproject.toml's black target-version bump (see the comment above that line, and #13748). I went looking for the host evidence it requires. Part of it is now in hand; the part that matters most is not, and this host cannot supply it.

AC1 — "the venv is built with 3.14 and its full requirements set installs" — MET

venv interpreter : Python 3.14.6   (/usr/bin/python3.14, pyvenv.cfg version = 3.14.6)
venv lib dir     : python3.14
pip check        : No broken requirements found.
openvino 2026.4.0 · torch 2.14.0 · numpy 2.5.3 · onnxruntime 1.28.0

The service unit runs that interpreter directly, and it is active with NRestarts=0 since 20:56 local.

AC2 — "starts and serves inference on 3.14, verified on a host" — PARTIALLY met, and this host cannot finish it

What is verified: the native-extension import path, which is the failure mode this AC was written about.

import openvino -> 2026.4.0-22959-99c81491cc3-releases/2026/4
ov.Core().available_devices -> ['CPU']

So OpenVINO's native extension loads under 3.14 and enumerates devices. A cp314 wheel that could not load would have failed right there, which is the "appears at import" half of the AC.

What is NOT verified, and why this node cannot:

available_devices is ['CPU'] — no NPU.
/dev/accel is absent.
lspci shows no NPU-class device.
The unit has produced no journal entries at all, and zero inference-related lines today. So it is running but has not served an inference here.

This is a CPU-only node. "Serves inference on real NPU hardware" cannot be demonstrated on it at any level of effort, so I am not ticking AC2 and not claiming the gate is cleared.

What this does and does not unblock

It does not unblock the black target-version bump. That comment is explicit — "Bump only once that host evidence lands on #13747, never on merge evidence alone (#13748)" — and the evidence it names is NPU-hardware inference, which is exactly the half still missing. Bumping on an import check would be substituting a weaker measurement for the one the decision asked for.

It does narrow what remains: the cp314 wheel set, the venv build and the native import are no longer open questions. What is left is one run on a node that has an NPU.

Suggested AC refinement

Name the node class the verification must run on. "A host" was satisfiable by a CPU-only node, which is how I got this far and still could not finish — the criterion should say NPU-equipped so the gap is visible before someone spends the effort.
Record what the inference evidence looks like: a log line, a device string, an infer_request completing. Otherwise "serves inference" is ticked on a service that merely stayed up, which is what this node would have let me claim.

Method note: everything above is read-only observation of the running deployment — systemctl show, pyvenv.cfg, pip check, and an openvino import inside the worker's own venv. No codebase code was run, nothing was installed, and nothing under /opt/autobot was modified.

Correction to my previous comment — the machine HAS an NPU; this environment cannot reach it

My previous comment said lspci showed no NPU-class device and implied the node has no NPU. That is wrong. The machine has an Intel AI Boost NPU. What is true is narrower and more useful.

The deployment runs under WSL2, and WSL2 does not pass the NPU through to the Linux guest:

kernel            : 6.18.x-microsoft-standard-WSL2
/dev/dxg          : present   (Microsoft GPU paravirtualization)
/dev/accel        : absent
/sys/class/accel  : absent
intel_vpu / ivpu  : not loaded
lspci             : 4 devices — one virtio controller, three Microsoft "3D controller" shims

So lspci was reporting the virtualized PCI topology, not the physical machine. My reading of it as "no NPU exists" was a measurement about the guest presented as a fact about the hardware — the thing this repo's own discipline docs warn about, and I did it on a public issue.

OpenVINO in the worker venv agrees:

available_devices                     -> ['CPU']
Core().get_property('NPU', 'FULL_DEVICE_NAME') -> RuntimeError from plugin.cpp:112

What this changes about AC2

AC2 is still not met, but for a different and more specific reason than I gave:

Not "this node has no NPU". It has one.
Actually: the Linux NPU worker runs inside WSL2, where the NPU is not exposed, so OpenVINO falls back to CPU. On this deployment the NPU worker cannot use the NPU at all — not during verification, and not in normal operation.

That is worth more than the 3.14 question it came from. If the production deployment is this WSL2 node, then the NPU worker is doing CPU inference today and the device it is named for is unreachable. I have not checked whether anything reports that, or whether a CPU fallback is logged — those are separate questions I am not answering here.

What still holds from the previous comment

AC1 is met and that evidence is unaffected: venv is Python 3.14.6, pip check clean, openvino 2026.4.0 / torch 2.14.0 / numpy 2.5.3 / onnxruntime 1.28.0, service active with no restarts. The native extension loads under 3.14 — a broken cp314 wheel would have failed at import, and it did not.

Revised suggestion for the AC

State that verification must run where the NPU is exposed to the OS running the worker, not merely present in the machine. A WSL2 guest on an NPU-equipped laptop satisfies "a host with an NPU" in plain English and still cannot do it.

Consolidated and verified — replaces the two comments above

I corrected myself twice here. This is the settled version, with each claim traced to what produced it.

The hardware


Intel AI Boost NPU	present on the machine
NVIDIA RTX 4070 Laptop (8 GiB, driver 617.14)	present and visible inside WSL2
deployment environment	WSL2 guest

What the guest can actually reach

/dev/dxg          present   -> NVIDIA passthrough works
nvidia-smi        RTX 4070 Laptop GPU, 617.14, 8188 MiB
torch 2.14.0+cu130  cuda available: True, device RTX 4070, capability (8, 9)

/dev/accel        absent
/sys/class/accel  absent
intel_vpu/ivpu    not loaded
OpenVINO Core().available_devices -> ['CPU']
Core().get_property('NPU','FULL_DEVICE_NAME') -> RuntimeError (plugin.cpp:112)

So WSL2 passes the NVIDIA GPU through and does not pass the Intel NPU through.

What the worker therefore does, read from the code and not inferred

autobot-infrastructure/autobot-npu-worker/docker/npu_model_manager.py selects a device purely through OpenVINO:

:88   npu_devices = [d for d in available_devices if "NPU" in d]
:93   logger.warning("No NPU devices detected - falling back to CPU")
:105  if self.core and "GPU" in self.core.available_devices:
:107  return "CPU"

available_devices is ['CPU'] on this node, so line 93 fires and the worker runs on CPU.

The RTX 4070 does not rescue this. OpenVINO's "GPU" device is an Intel iGPU; NVIDIA is not an OpenVINO device, so a CUDA-capable GPU sitting right there is invisible to this code path. The worker has torch with working CUDA in the same venv and never consults it.

Bearing on this issue

AC1 (venv on 3.14, full requirements install) — met, evidence in my first comment and unaffected by the corrections.

AC2 (serves inference on 3.14, verified on a host) — cannot be completed on this node, because the NPU is not exposed to the OS running the worker. Not for lack of hardware, which is what I wrongly said first.

So the black target-version gate in pyproject.toml stays closed. I am not bumping it.

Two things worth separating out, which I am not filing from inside this issue

The deployment's NPU worker is doing CPU inference and nothing says so outside a logger.warning. If this WSL2 node is production, the device the service is named for is unreachable in normal operation, not merely during verification.
A CUDA GPU is present, usable from the worker's own venv, and unused by it. Whether the OpenVINO-only device selection should fall back to CPU while a 4070 is idle is a design question, not a bug report, and it belongs to whoever owns the worker's routing.

Method, and where I went wrong

Read-only throughout: systemctl show, pyvenv.cfg, pip check, nvidia-smi, and library imports inside the worker's own venv. Nothing installed, nothing under the deployment modified, no codebase code run.

My first error was reading lspci — which in WSL2 enumerates four virtual devices — as a statement about the physical machine. My second was saying "CPU inference today" before checking whether another runtime in the same venv had a device; it turned out to be true, but I had not earned it at the time. Both were measurements about the guest presented as facts about the host.

Host evidence — read-only observation of the deployed worker

The wave-2 blockers (#13744, #13745, #13746) are all closed, so this is unblocked. Measured against the running deployment and against merged main.

Criterion 1 — venv built with 3.14, requirements install

Met. Every deployed service venv on the node reports the same interpreter:

autobot-npu-worker     Python 3.14.6
autobot-backend        Python 3.14.6
autobot-ai-stack       Python 3.14.6
autobot-slm-agent      Python 3.14.6
autobot-browser-worker Python 3.14.6

The dependency set from this issue's own table, as installed in the NPU worker venv:

Package	Predicted here	Installed
`openvino`	cp314 available	2026.4.0
`torch`	cp314 available	2.14.0+cu130
`numpy`	cp314 available	2.5.3
`onnxruntime-openvino`	no cp314, Windows only	not installed — correct

onnxruntime (plain) 1.28.0 and onnx 1.22.0 are present; neither is onnxruntime-openvino, so the Windows blocker is untouched and still out of scope as written.

Criterion 2 — starts and serves inference on 3.14

Partly met, and I am not ticking it.

The service is up, and the native extension genuinely loads — which is the part a green pip install cannot prove:

systemctl is-active autobot-npu-worker   ->  active
python                                    ->  3.14.6 CPython
import openvino                           ->  2026.4.0-22959
ov.Core().available_devices               ->  ['CPU']

So OpenVINO imports and initialises on 3.14 — no native-extension failure, which was the whole risk this criterion was written to catch.

But available_devices is ['CPU']. No NPU device is enumerated, so "serves inference" is demonstrated for CPU only and not for the accelerator this worker exists for. This node virtualises PCI and does not pass the Intel NPU through, so this is a property of where it was measured, not evidence that the NPU path is broken on 3.14. Confirming the NPU path needs a host with the device actually exposed — the same limitation recorded in #17988.

Criterion 3 — `ensure_venv_version` is applied

Met on merged main. roles/npu-worker/tasks/main.yml:

:165 — tasks_from: ensure_venv_version
:168 — venv_python: python3.14
:172 — python3.14 -m venv {{ npu_install_dir }}/venv

The playbook carries 12 python3. references, all 3.14, and zero sub-3.14 (counted with a control to confirm the matcher sees anything at all).

Criterion 4 — verified by deliberate failure

Not met, and the guard does not currently do what this criterion assumes. ensure_venv_version removes a wrong-version venv only when the existing interpreter still runs (rc == 0). When the venv exists but its interpreter is gone — the normal state after an interpreter upgrade — the guard no-ops, creates: reads a dangling symlink as absent so the rebuild runs anyway, and python3.14 -m venv without --clear carries the old site-packages forward.

Each step is measured in #17990, filed separately because it affects every role that calls the shared guard, not just this one.

That issue should land before this one is ticked: criterion 4 asks the guard to reject a venv left at the old version, and in the broken-interpreter case it does not.

Summary: 2 of 4 criteria met on evidence, 1 partly met and limited by the measuring node, 1 blocked on #17990. The interpreter move itself appears already done — the remaining work is making the replacement guard honest, not bumping a version.

Navigation Menu

Repository navigation

build(python): move the Linux NPU worker to 3.14 — OpenVINO now ships cp314 wheels #13747

The blocker has expired

Scope

Not in scope

Acceptance criteria

Why this stays open

To close

Do not treat this as unblocking #13748

1 remaining item

Host evidence, gathered read-only on the deployment node — AC1 met, AC2 not met here

AC1 — "the venv is built with 3.14 and its full requirements set installs" — MET

AC2 — "starts and serves inference on 3.14, verified on a host" — PARTIALLY met, and this host cannot finish it

What this does and does not unblock

Suggested AC refinement

Correction to my previous comment — the machine HAS an NPU; this environment cannot reach it

What this changes about AC2

What still holds from the previous comment

Revised suggestion for the AC

Consolidated and verified — replaces the two comments above

The hardware

What the guest can actually reach

What the worker therefore does, read from the code and not inferred

Bearing on this issue

Two things worth separating out, which I am not filing from inside this issue

Method, and where I went wrong

Host evidence — read-only observation of the deployed worker

Criterion 1 — venv built with 3.14, requirements install

Criterion 2 — starts and serves inference on 3.14

Criterion 3 — `ensure_venv_version` is applied

Criterion 4 — verified by deliberate failure

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

build(python): move the Linux NPU worker to 3.14 — OpenVINO now ships cp314 wheels #13747

Description

The blocker has expired

Scope

Not in scope

Acceptance criteria

Activity

mrveiss commented on Aug 8, 2026

mrveiss commented on Aug 9, 2026

Why this stays open

To close

Do not treat this as unblocking #13748

github-actions commented on Aug 9, 2026

mrveiss commented on Aug 9, 2026

1 remaining item

mrveiss commented on Sep 12, 2026

mrveiss commented on Oct 4, 2026

Host evidence, gathered read-only on the deployment node — AC1 met, AC2 not met here

AC1 — "the venv is built with 3.14 and its full requirements set installs" — MET

AC2 — "starts and serves inference on 3.14, verified on a host" — PARTIALLY met, and this host cannot finish it

What this does and does not unblock

Suggested AC refinement

mrveiss commented on Oct 4, 2026

Correction to my previous comment — the machine HAS an NPU; this environment cannot reach it

What this changes about AC2

What still holds from the previous comment

Revised suggestion for the AC

mrveiss commented on Oct 4, 2026

Consolidated and verified — replaces the two comments above

The hardware

What the guest can actually reach

What the worker therefore does, read from the code and not inferred

Bearing on this issue

Two things worth separating out, which I am not filing from inside this issue

Method, and where I went wrong

mrveiss commented on Oct 5, 2026

Host evidence — read-only observation of the deployed worker

Criterion 1 — venv built with 3.14, requirements install

Criterion 2 — starts and serves inference on 3.14

Criterion 3 — ensure_venv_version is applied

Criterion 4 — verified by deliberate failure

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Criterion 3 — `ensure_venv_version` is applied