Skip to content

build(python): move the Linux NPU worker to 3.14 (#13747) - #13800

Merged
mrveiss merged 2 commits into
Dev_new_guifrom
issue-13747
Aug 9, 2026
Merged

mrveiss merged 2 commits into
Dev_new_guifrom
issue-13747

Conversation

@mrveiss

@mrveiss mrveiss commented Aug 9, 2026

Copy link
Copy Markdown
Owner

Thinking Path

#10877 pinned the NPU worker to Python 3.11 because OpenVINO shipped no 3.14 wheels. That single pin is load-bearing beyond the worker: it is also why black's target-version stays at py312 (#13748), since PEP 758 except A, B: would be a hard SyntaxError on shared code the 3.11 worker imports.

I re-measured the premise rather than trusting the recorded reason. Against the PyPI JSON API on 2026-08-08:

Package Latest Python tags 3.14?
openvino 2026.3.0 cp310, cp311, cp312, cp313, cp314 ✅
torch 2.13.0 cp310, cp311, cp312, cp313, cp314 ✅
numpy 2.5.1 cp312, cp313, cp314 ✅
onnxruntime-openvino 1.24.1 cp311, cp312, cp313 ❌ no cp314

"openvino has a cp314 wheel" is weaker evidence than it looks, so I resolved the worker's actual requirements on 3.14 — the full graph, not one package:

$ venv/bin/python -m pip install --dry-run --ignore-installed -r autobot-npu-worker/requirements.txt
Would install ... numpy-2.5.1 openvino-2026.3.0 openvino-telemetry-2025.2.0 ... torch-2.13.0 triton-3.7.1 ...

No conflicts, no fallback to an sdist. That is the whole set the worker installs.

What Changed

Interpreter moved 3.11 → 3.14 in the two places that build the venv:

  • roles/npu-worker/tasks/main.yml — the canonical path (interpreter install, ensure_venv_version, -m venv)
  • playbooks/deploy-native-services.yml — the documented operator path (apt packages, ensure_venv_version, -m venv, virtualenv_python)

Plus the places that described the old split: the Phase 1.5 NPU play in deploy-full.yml, and the header comments in configure-python-provision-permissions.yml and provision-local-python.yml.

The upgrade-in-place hazard, and why it is already handled

The real risk is not a fresh install — it is a box that already has a 3.11 venv. Recreating it as 3.14 over existing site-packages leaves 3.11-built OpenVINO native extensions in place; that fails at import or first inference, long after the deploy reports success.

Both paths already call ensure_venv_version before creating the venv, which detects a wrong-version venv and removes it. That task is what makes this change safe in place, so I documented it as load-bearing rather than leaving it looking incidental.

Deliberately not in scope

The Windows NPU worker (autobot-npu-worker/resources/windows-npu-worker/) pins onnxruntime-openvino, which has no cp314. It is untouched — confirmed no Windows file appears in this diff.

I also kept the separate Phase 1.5 NPU play rather than folding it into the backend:aiml one. npu is a distinct host group that pattern does not reach, and the python314 role is idempotent, so a co-located host just skips the second install.

Verification

$ ansible-playbook --syntax-check -i inventory/hosts.yml <each changed playbook>
deploy-native-services.yml                    SYNTAX OK
deploy-full.yml                               SYNTAX OK
provision-local-python.yml                    SYNTAX OK
configure-python-provision-permissions.yml    SYNTAX OK

$ ansible-playbook --syntax-check -i inventory/hosts.yml playbooks/provision-fleet-roles.yml
SYNTAX OK          # this is the playbook that applies roles/npu-worker, so the role graph parses

No python3.11 pin remains for the Linux worker anywhere in ansible/. The one surviving mention is a parameter example in ensure_venv_version.yml's docstring, which documents the argument's form and is not a pin.

This PR does not close #13747

Its acceptance criteria require the worker to start and serve inference on 3.14, verified on a host — explicitly not by a successful pip install. OpenVINO failures are native-extension failures that surface at import or first inference, so a green resolution proves the dependency set solves, not that the NPU runs.

I have no NPU host to verify against, so the issue stays open pending that deploy evidence. Merging this is what makes the next NPU deploy use 3.14; confirming it worked is a separate, host-side step.

#13748 (black → py314) remains blocked and must not be taken as unblocked by this merge. It is unblocked only once the worker is running 3.14 — reversing that order writes PEP 758 syntax into shared code a still-3.11 worker imports, which is a SyntaxError at import, in production.

Model Used

Claude Opus 5 (1M context)

Part of #13743

mrveiss added 2 commits August 9, 2026 10:20
#10877 pinned this worker to Python 3.11 because OpenVINO shipped no 3.14
wheels, and that pin is what keeps black's target-version at py312 (#13748) —
PEP 758 syntax would be a SyntaxError on shared code the worker imports.

The premise has expired. Measured against the PyPI JSON API on 2026-08-08:
openvino 2026.3.0, torch 2.13.0 and numpy 2.5.1 all publish cp314. Verified
further by resolving the worker's actual requirements on 3.14 — pip solves the
full graph with no conflicts, landing openvino-2026.3.0, torch-2.13.0 and
numpy-2.5.1.

Changed the interpreter in roles/npu-worker (the canonical path) and in
deploy-native-services.yml, plus the phase in deploy-full.yml and the two
provisioning playbooks whose comments named the old split.

What makes this safe on an already-deployed box is ensure_venv_version, which
both paths already call: it detects the existing 3.11 venv as wrong-version and
removes it, so 3.11-built OpenVINO binaries are never carried into a 3.14 venv.
That failure would be a native ImportError at worker startup, well after the
deploy reported success.

The Windows NPU worker is deliberately untouched: it pins onnxruntime-openvino,
which still has no cp314 (1.24.1 ships cp311-cp313).

Verified: ansible-playbook --syntax-check passes on all four changed playbooks
and on provision-fleet-roles.yml, which applies roles/npu-worker, so the role
graph parses. No python3.11 pin remains for the Linux worker anywhere.

Does NOT close #13747: its acceptance criteria require the worker to start and
serve inference on 3.14 verified on a host, and OpenVINO failures are native-
extension failures that appear at import or first inference, not at pip install.
That evidence has to come from a deploy.
@mrveiss

mrveiss commented Aug 9, 2026

Copy link
Copy Markdown
Owner Author

Review — pre-merge pass

Reviewed against origin/Dev_new_gui at 8db4b879. Self-review recorded as a comment (GitHub blocks self-approval).

What this actually changes

Five files, all Ansible. Two of them build a venv and are the real change; three carry comments that named the old 3.11/3.14 split.

File Change Load-bearing?
roles/npu-worker/tasks/main.yml interpreter install, ensure_venv_version, -m venv → 3.14 yes — canonical path
playbooks/deploy-native-services.yml apt packages, ensure_venv_version, -m venv, virtualenv_python → 3.14 yes — documented operator path
playbooks/deploy-full.yml Phase 1.5 NPU play → 3.14 yes, name + version var
configure-python-provision-permissions.yml, provision-local-python.yml header comments no

The risk I looked hardest at

Not the fresh install — the upgrade in place. A box with an existing 3.11 venv, recreated as 3.14 over the same site-packages, keeps 3.11-built OpenVINO native extensions. That fails at import or first inference, well after the deploy reports success, which is the worst shape of failure available here.

Both changed paths already call ensure_venv_version before creating the venv, and that task removes a wrong-version venv. So the hazard is handled by code that already existed — I documented it as load-bearing rather than leaving it looking incidental, because a future edit that drops it would silently reintroduce this.

Checks

Risk Check Result
Wheels exist but the graph does not solve pip install --dry-run of the worker's real requirements.txt on 3.14 Solves: openvino-2026.3.0, torch-2.13.0, numpy-2.5.1, no conflicts, no sdist fallback
A 3.11 pin left behind Grepped python3.11 across ansible/ Only a parameter example in ensure_venv_version.yml's docstring — documents the argument form, not a pin
Windows worker dragged along git status for any Windows path Untouched. It pins onnxruntime-openvino, which has no cp314 (1.24.1 = cp311–cp313)
Broken playbook syntax ansible-playbook --syntax-check on all four changed playbooks All OK
Broken role graph --syntax-check on provision-fleet-roles.yml, which applies roles/npu-worker OK

Judgement call

I kept the separate Phase 1.5 NPU play instead of folding it into the backend:aiml one now that both install 3.14. npu is a distinct host group the backend:aiml pattern does not reach, and the python314 role is idempotent so a co-located host just skips the second install. Consolidating would have been a behaviour change dressed as cleanup.

What this PR does not do

It does not close #13747. The AC requires the worker to start and serve inference on 3.14 verified on a host; a clean dependency resolution is not that evidence, because OpenVINO fails at import or first inference rather than at install. No NPU host is reachable from here.

It does not unblock #13748. That needs the worker running 3.14, not a merged config change — and the Windows-worker question is still open.

Verdict

No blocking findings.

@mrveiss
mrveiss merged commit 317723e into Dev_new_gui Aug 9, 2026
35 checks passed
@mrveiss
mrveiss deleted the issue-13747 branch August 9, 2026 07:40
mrveiss added a commit that referenced this pull request Aug 25, 2026
…#13747)

roles/npu-worker moved to python3.14 in #13800, but five other places
still declared the pre-#13747 pin:

- code_sync.py's _COMPONENT_PYTHON_TARGET (self-described as 'the
  single authoritative mapping'), and the test asserting its value.
  Currently unreachable for the npu-worker key (_WORKER_COMPONENTS
  routes worker components around the only two readers of this dict),
  but wrong data under a claim of authority is worse than no claim.
- inventory/production.yml's python_version var for the npu group,
  with a comment restating the expired 'no 3.14 wheels' premise.
- check_venv_producers.py's docstring, describing the #13746 hazard
  as still-live with two different interpreters.
- pyproject.toml's black target-version comment and scripts/format.sh,
  both citing 'the 3.11 NPU worker' as the reason py314 stays unset.
  Corrected to the real, current blocker: #13747 needs host evidence
  the worker is running 3.14, not merely configured for it.

None of these change target-version itself — #13748 stays gated on
#13747's host verification, per the epic's explicit ordering.

_COMPONENT_PYTHON_TARGET and its test file are on the file-size
ratchet ceiling (6065 / 2098 lines); edits kept net-zero to stay
compliant, verified with scripts/check_python_file_size.py.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant