Skip to content

Create a proper Python project for the Python scripts #470

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jun 29, 2025

Conversation

multiplemonomials
Copy link
Collaborator

@multiplemonomials multiplemonomials commented Jun 21, 2025

Summary of changes

This PR creates a proper Python project (with a pyproject.toml) for all of the Python scripts used by Mbed. Previously, these just lived in a folder with a requirements.txt. Not only was this janky, but it also meant that executing the scripts was a huge pain, because you had to do

$ cd mbed-os/tools/python
$ python -m mbed_tools.cli.main <command>

in order to execute things.

With this PR, we now have a proper Python project, so it can be installed into the virtualenv and then you can just do

$ mbed-tools <command>

I ended up using Hatchling for this, rather than something more complex like Poetry or uv, as there are no native modules or other complex things in the packages, and it seemed like the simplest thing that would work.

The other big change I made is that now the pyocd and cysecuretools packages are installed lazily, instead of being included in the default set of dependencies. Both of these packages had pretty chonky sets of dependencies, and both were not needed for basic use cases of Mbed CE. Removing them should make stuff like CI builds run a bit faster and use less dependencies.

Pyocd is now installed the first time the PYOCD upload method is used, and cysecuretools is installed only when the Cypress TFM post-build script is used (i.e. for the CYTFM_064B0S2_4343W target only).

Impact of changes

  • Tools like memap and mbed-tools can now be run as commands (after activating the venv) rather than having to cd into the tools/python folder
  • Improvements to venv creation to be a bit more robust
  • Improved check of the python interpreter when Mbed venv is disabled, it now verifies that the mbed ce tools package is actually installed into the interpreter instead of just checking a few dependencies
  • pyocd and cysecuretools (and all their many dependencies) are now installed lazily when needed

Migration actions required

If you are using the default venv mode, no migration actions are needed. If you disabled venv mode via MBED_CREATE_PYTHON_VENV=FALSE, then the following command:

python -m pip install mbed-os/tools/requirements.txt

has changed to:

python -m pip install -e mbed-os/tools

Additionally, if not using the venv, you will need to install pyocd and/or cysecuretools manually if you were depending on these packages, as they are no longer included in the base Mbed requirements.

Documentation

Once this is merged I will update wiki pages to use the new shortened commands for things


Pull request type

[] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[] Feature update (New feature / Functionality change / New API)
[X] Major update (Breaking change E.g. Return code change / API behaviour change)

Test results

[] No Tests required for this change (E.g docs only update)
[X] Covered by existing mbed-os tests (Greentea or Unittest)
[] Tests / results supplied as part of this PR

# Special situation: if we have a cached interpreter location in the venv dir, but Python could be found,
# this means that the venv was deleted or symlinks to a missing python install location.
# So, use the system python and recreate it.
if("${Python3_EXECUTABLE}" MATCHES "${MBED_VENV_LOCATION}" AND NOT Python3_FOUND)
Copy link
Collaborator Author

@multiplemonomials multiplemonomials Jun 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have noticed two scenarios here that seemed to cause problems over the last year or so:

  1. The python venv is corrupted in some way (e.g. by removal of the system Python that it is pointing to). This causes find_package(Python3) to fail, with an error like "Python interpreter could not be run"
  2. The python venv is deleted, which will cause find_package(Python3) to silently return the system python instead of the venv.

I attempt to handle both those cases here. I have confirmed that #2 is fixed now, but I have not found a reliable way to reproduce situation #1 yet, so I sort of threw everything at it in the hope that we can make FindPython temporarily revert back to the system python.

@multiplemonomials multiplemonomials self-assigned this Jun 22, 2025
@multiplemonomials multiplemonomials changed the title [draft] Create a proper Python project for the Python scripts Create a proper Python project for the Python scripts Jun 22, 2025
Base automatically changed from dev/ambiq-apollo3 to master June 26, 2025 04:31
Copy link

@VictorWTang VictorWTang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couple really minor comments, but the Python reorganization looks good to me!

@multiplemonomials multiplemonomials force-pushed the dev/improve-python-structure branch from 6a2de1d to 05f37a0 Compare June 29, 2025 04:30
Co-authored-by: VictorWTang <33789988+VictorWTang@users.noreply.github.com>
@multiplemonomials multiplemonomials merged commit f598b77 into master Jun 29, 2025
52 checks passed
@multiplemonomials multiplemonomials deleted the dev/improve-python-structure branch June 29, 2025 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants