Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DEV: Evaluate dependency update concept for CI #2574

Open
stefan6419846 opened this issue Apr 2, 2024 · 7 comments
Open

DEV: Evaluate dependency update concept for CI #2574

stefan6419846 opened this issue Apr 2, 2024 · 7 comments
Labels
Meta nf-ci Non-functional change: Continuous Integration

Comments

@stefan6419846
Copy link
Collaborator

At the moment, the requirement files used by the CI are only updated partially and if required. The same applies to the pre-commit configuration.

There have been some related issues where I stumbled upon such cases:

  • In DEV: Ensure testing against the latest versions #2455, I would like to ensure that our package does not break with the latest package changes as it did for pytest some weeks ago and for Pillow with yesterdays' release.
  • In DEV: Remove unused dependencies #2572, I stumbled upon some dependencies which nobody is using anymore and thus could be cleaned.
  • In DEV: Review pinned requirements in include files #2573, I had a look at explicitly pinned packages, where the reason sometimes is not documented appropriately. Most of these just require some unpinning and small fixes. But there also is the pre-commit framework which has a separate set of updates to maintain as well, although I do not think that this is actively used indeed(?)

I would appreciate if we would come up with an appropriate process for handling such cases and document it accordingly. Some immediately apparent issue might already be that updating the requirement files is not very easy due to requiring different Python versions on a system, which requires additional local setups and might depend on the OS/distribution.

@MartinThoma
Copy link
Member

I stumbled upon some dependencies which nobody is using anymore and thus could be cleaned.

A partial solution for that might be to comment explicitly why a dependency is there within the requirements.in file.

@MartinThoma
Copy link
Member

I do use pre-commit and I think it's pretty valuable. We could also run pre-commit in CI. That could even replace the way we run ruff / mypy.

To me pre-commit is valuable as it's a well-known way to apply linting rules before CI. To me as a contributor, it's nicer to fail earlier (and even nicer if auto-fixes are applied).

At the same time, if somebody does not know pre-commit it's not necessary. CI will ensure the coding standards are fulfilled.

@MartinThoma
Copy link
Member

requiring different Python versions

I was thinking about dropping the next Python version at the end of 2024: #2418

I'm not sure if that makes things simpler, though.

@stefan6419846
Copy link
Collaborator Author

A partial solution for that might be to comment explicitly why a dependency is there within the requirements.in file.

This is correct (or at least inside the history to allow blaming with an additional issue to track the progress of resolving this). An ideal solution to this would still be to have no pins at all and ensure that each update applies cleanly similar to https://github.com/stefan6419846/pypdf_recent (although I am ignoring linting issues there for now due to some larger changes enforced by ruff).

I do use pre-commit and I think it's pretty valuable. We could also run pre-commit in CI. That could even replace the way we run ruff / mypy.

The standalone calls in CI are completely fine in my opinion as this separates the different tools in a clean manner through separate steps. We should still keep in mind that the versions there need some love as well ;)

I was thinking about dropping the next Python version at the end of 2024: #2418

I'm not sure if that makes things simpler, though.

I do not think so unless we are going to drop both Python 3.7 and 3.8 at the end of the year. In October 2024, Python 3.8 will go EOL as well and Python 3.13 is planned to be released: https://devguide.python.org/versions/#supported-versions With the current approach, we are always lagging one version behind, possibly complicating dependency handling in CI due to newer releases not supporting Python 3.7 any more for example (like recent Pillow versions).

@MartinThoma MartinThoma added the nf-ci Non-functional change: Continuous Integration label Jul 14, 2024
@j-t-1
Copy link
Contributor

j-t-1 commented Jul 26, 2024

Although currently in beta, I think it is possible to add 3.13 to tests now.

@pubpub-zz
Copy link
Collaborator

Although currently in beta, I think it is possible to add 3.13 to tests now.

At some time also it would be interesting to test python 3.13 with NOGIL

@stefan6419846
Copy link
Collaborator Author

I have just added Python 3.13 to our test matrix. AFAIK the new GIL handling requires a build parameter to be set and only affects threading, which (if ever) would be relevant for the parallel test runner in CI only, but not for regular pypdf usage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Meta nf-ci Non-functional change: Continuous Integration
Projects
None yet
Development

No branches or pull requests

4 participants