Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

python 3.13 transition #37197

Open
smoser opened this issue Dec 16, 2024 · 6 comments
Open

python 3.13 transition #37197

smoser opened this issue Dec 16, 2024 · 6 comments
Assignees
@smoser @dannf

Comments

@smoser
Copy link
Member

smoser commented Dec 16, 2024 •
edited
Loading

Last week we bumped the 'provider priority' value for the python 3.13 subpackage of packages named 'py3-'.
That is to say that before last week, if you did apk add py3-lxml you'd end up with the python-3.12 version, a package named py3.12-lxml. This week, you'd end up with py3.13-lxml and its dependencies installed.

There are several packages that have an un-versioned python package listed as a build-depends, runtime-dependency or test dependency. Many of these packages will now be broken either in runtime or in the next build for reasons such as:

  1. they installed python modules (.py files) into /usr/lib/python3.12/site-packages or a virtual env's lib/python3.12/site-packages using 'python' or 'python3' to do the install.
  2. their build uses a python3 or python3-dev package , but does not support python-3.13

I searched through existing melange files for use of un-versioned python 3 packages.
There are very few cases where an un-versioned python dependency (python3-dev rather than python-3.XX-dev) should be used. Those are mostly limited to:

  1. build systems that use only python3 and no other python packages.
  2. runtime dependencies that use only the python executable and standard library. byobu is one such example.

To update packages, follow these bullet points:

  • Remove python-3 and python-3-dev references. Either replace them with python3 or python3-dev. In almost all cases, python3 should not be listed in the runtime or test dependencies.

  • Packages that simply use python or python3 in their build environment but do not produce packages with python files can continue to use python3.

  • If python3 is listed only in the build environment (not elsewhere), you may well be able to just delete it. If another package in the build environment needs python, it should be pulled in.

  • If a package has more than one python package (py3-) in its build environment, it should pick a specific version (3.13 if supported) and list the dependencies with py${{vars.py-version}}-pkg. This will guarantee that the dependencies are installed at a consistent version.

  • If there are more than one python build package (setuptools, wheel...) listed in the build environment, then replace those with py3.XX-build-base, then use the build-base package.

  • If a package has anything other than python3 listed in its runtime depends, or it uses a virtual env, then it needs to pick a specific python version.

  • If the build uses python3 or python by name, the easiest thing to do is
    to use a python-as-wrapper package to provide that executable.
    Add python-as-wrapper package and use it in asciidoc build. #36891 . It ensures that shbang will get
    the correct executable value, and melange will then generate the correct
    dependendency.

Where 'pick a specific version' is mentioned above, the preferred mechanism for
doing so is to use py-version variable.

vars:
  py-version: 3.13
...

environment:
  contents:
  packages:
   ..
   - py${{py-version}}-build-base
   - py${{py-version}}-
@smoser
Copy link
Member Author

smoser commented Dec 16, 2024 •
edited by dannf
Loading

I put together a little python script find-pythons that searches build/run/test depends for un-versiondd python packages. The packages below were found to have some. The full output of that tool is python-rdeps.txt . The output lists where the dependency is (run/test/build as well as how many).

@smoser
Copy link
Member Author

smoser commented Dec 17, 2024

Just a though... It would be nice to have a way to indicate "this actually correctly uses any-old-python".

@Ritish134
Copy link

May I take up this task and work on updating the remaining packages with the correct Python versioning?

Here’s the approach I’ve understood so far—please let me know if I’ve missed anything, as I’m still familiarizing myself with the codebase:

  • I’ll start by using the provided find-pythons script to identify packages that have python3 or python3-dev in their dependencies but lack versioning.
  • If Python is required, replace python3 with the appropriate version (py3.13, py3.12, etc.).
  • Remove unnecessary Python dependencies if the package doesn't need it.
  • Using melange compile to build the updated packages and ensure the changes work as expected.

or if there’s anything else I should take into consideration.

@smoser
Copy link
Member Author

smoser commented Dec 17, 2024

May I take up this task and work on updating the remaining packages with the correct Python versioning?

Sure, glad to have help.

Here’s the approach I’ve understood so far—please let me know if I’ve missed anything, as I’m still familiarizing myself with the codebase:

* I’ll start by using the provided find-pythons script to identify packages that have python3 or python3-dev in their dependencies but lack versioning.

I attached its output, you can run it again, you should hopefully get mostly the same output. The script also shows packages that have 'py3-' rather than py3.XX-foo as dependencies.

* If Python is required, replace python3 with the appropriate version (py3.13, py3.12, etc.).

preferring 3.13.

* Remove unnecessary Python dependencies if the package doesn't need it.

* Using melange compile to build the updated packages and ensure the changes work as expected.

'melange build' . easiest way to do this is:

make package/your-pkgname
make test/your-pkgname

last, use py-version variable so that we can easily find and bump those packages in the future. see example #37277

@dannf dannf mentioned this issue Dec 18, 2024
Closed
@sil2100 sil2100 mentioned this issue Dec 18, 2024
Closed
@Ritish134
Copy link

Ritish134 commented Dec 18, 2024 •
edited
Loading

I pulled the melange image

docker pull cgr.dev/chainguard/melange:latest

and updated the rust-1.69.yaml with python versioning #37426
while using melange build I mounted with the current directory

docker run --rm -v "/root/os:/work" cgr.dev/chainguard/melange build /work/rust-1.69.yaml

At first it was throwing this error mentioned here : chainguard-dev/melange#1020

So I updated the file adding this


environment:
  contents:
    keyring:
      - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
    repositories:
      - https://packages.wolfi.dev/os
    packages:
      - busybox
      - ca-certificates-bundle
      - go
  environment:
    CGO_ENABLED: "0"

again using melange build, I got this :

image

I think I found a similar issue here: chainguard-dev/melange#1645
@smoser

@smoser
Copy link
Member Author

smoser commented Dec 18, 2024

I generally just use melange and make from the host, and I do not invoke 'melange' directly, but rather through make package/rust-1.69.yaml.

If you want to use docker, I would suggest ghcr.io/wolfi-dev/sdk. and the easiest way to get into that is just make dev-container from a git clone of wolfi-dev/os.

So, assuming you have docker installed. what I would suggest you do is:

$ git clone ssh://git@github.com/wolfi-dev/os wolfi-os
$ cd wolfi-os
$ make dev-container

That should put you into a environment that you can then:

$ make py3/aio-http.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smoser smoser

@dannf dannf

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@smoser @dannf @Ritish134