Add development support for python 3.12

[kjaymiller]

Thanks for wanting to report an issue you've found in valkey-py. Please delete this text and fill in the template below.
It is of course not always possible to reduce your code to a small test case, but it's highly appreciated to have as much data as possible. Thank you!

Version: What valkey-py and what valkey version is the issue happening on?

Platform: Python 3.12 on MacOS (M-Series ARM devices)

Description: There are some package changes that need to be made to make development possible on python 3.12

• unpinning the version of invoke from 1.7.3 allowing version >=2.0.0 which removes usage of the imp module which was removed in Python 3.12
• replacing distutils which is being used to check for truthy-like strings using strtobool with a dedicated module still being maintained like dirty-equals

[aiven-sal]

Is this just to make development possible or is there any production need?
If it's just for development, you may consider using pyenv. This is what I use as well, because my distro also has 3.12 by default and it works quite well (on Linux at least).

[kjaymiller]

development is possible with some work but in order to run invoke devenv with the Python 3.12 followed by invoke tests you run into several module not found errors that you can either import yourself likepyyaml or in some cases they have been replaced in updated versions of the existing requirements.

I would be interested in understanding why the versions are pinned to the versions selected. Other that tests failing outside the scope of my work, I've seen no issues with making the replacements.

distutils has been removed from Python as of 3.12 and is only used in a single point for testing. That can be replaced with a new module (again my suggestion is dirty_equals as it seems the most up to date and could have some other utilities that would be helpful in testing.

FWIW I'm using pyenv which when I switched the environment to 3.11 and saw the deprecation warnings, that is how I begun the digging for updated versions and replacements.

[mkmkme]

Hey @kjaymiller,

From my experience, docker-compose is quite broken these days. I personally use docker compose (note the space instead of a dash) that comes with docker.io installation.

The only thing devenv does is starting the docker containers needed for testing, though. You can easily do this via calling docker compose --profile all up -d manually.

There's another set of issues with Python 3.12, though. There seems to be an issue related to ssl module which we are still yet to investigate.

[kjaymiller]

I am limited on bandwidth for working on this one currently, but I think that perhaps the swap away from the two modules that we know are a problem would be a start. I could possibly look at the ssl issue with 3.12 but it would be at least a week before I can get to it.

[aiven-sal]

There should be no attempt to make development possible with any specific version of Python.

valkey-py supported versions are those that are used to run tests in CI, if it isn't in CI it is not supported and it doesn't make sense to develop against an unsupported version.

I can see the value of adding proper 3.12 support in general, which means enabling the CI and making sure that all tests pass. This is something that we should probably do before the end of the year.

But this should be done carefully. A rushed 3.12 support that causes problems later is not something I want.

[kjaymiller]

I can see the value of adding proper 3.12 support in general, which means enabling the CI and making sure that all tests pass. This is something that we should probably do before the end of the year.

I agree with you but we can't test against 3.12 because we're using modules that are no longer supported in 3.12 (and future versions). Work should be done to replace those unsoppurted modules across the board to ensure support for currently supported versions, as well as future versions as this problem is not going away.

Also FWIW Python 3.13 should also be tested against.

[aiven-sal]

I agree with you but we can't test against 3.12 because we're using modules that are no longer supported in 3.12 (and future versions). Work should be done to replace those unsoppurted modules across the board to ensure support for currently supported versions, as well as future versions as this problem is not going away.

My primary concern is to not break valkey-py for any of its users or potential users (i.e. people using redis-py that might want to migrate). Lack of support for 3.12 hasn't been a blocker for any of the redis-py users so far.

As I said, I think 3.12 support should be added before the end of the year, but there is no need to rush things.
Doing it now has the potential of creating some extra maintenance burden that I want to avoid.

Also FWIW Python 3.13 should also be tested against.

No, we don't have to support a version that isn't even stable. We could support it, but we don't really have time.

[kjaymiller]

My primary concern is to not break valkey-py for any of its users or potential users (i.e. people using redis-py that might want to migrate).

I may have miscommunicated this but the dependencies in questions are for the test suite and not for the primary code-base itself. Users would not be affected.

(Putting on PSF and Python Community Member Hat) Every year core developers ask that maintainers use the betas to report any issues in upcoming releases to ensure compatibility.

My suggested changes would allow for better support around testing newer releases without having to do major updates to the developer environment.

If we wanted to be sure we could add tox as a testing platform I. The devenv so that it can test against as many versions possible.

We can also include a matrix of tests in GitHub actions to ensure no breaking changes occur across supported versions (which Python 3.8+ is EOS at the end of October) but that's a completely different convo.

[kjaymiller]

If I haven't made it clear, I would be open to doing this work or helping find others to potentially support it.

I need to adjust my environment to include the sign off settings and avoid these PR issues but that's a quick fix I'm prepared to make.

[aiven-sal]

My primary concern is to not break valkey-py for any of its users or potential users (i.e. people using redis-py that might want to migrate).

I may have miscommunicated this but the dependencies in questions are for the test suite and not for the primary code-base itself. Users would not be affected.

I'm not sure I understand.
In a previous comment you mentioned (correctly) that distutils is a problem for 3.12 because it was removed from that version. But distutils is not used in any test. It is, in fact, used only once in the actual library. I also don't understand why you proposed to replace it with dirty_equals, because it doesn't seem to have anything to do with the current usage of distutils in the code base.
So changes need to be done to the actual library, not just the test code.
And there is more that needs to be done.

(Putting on PSF and Python Community Member Hat) Every year core developers ask that maintainers use the betas to report any issues in upcoming releases to ensure compatibility.

Following this line of thought we should also probably test against IronPython, Jyton, Stackless Python, Micropython etc
and maybe one day we will, but now we need to work with the resources we have and we need to prioritize the need of this project and its users.

My suggested changes would allow for better support around testing newer releases without having to do major updates to the developer environment.

I don't understand the problem here. Correctly setup the environment is not a big deal, maybe we can improve the documentation to make it clearer, but I don't see a problem with the current system. I have never used invoke before, I didn't even know that it existed before I started working on this project, but it's pretty simple and straightforward.

If we wanted to be sure we could add tox as a testing platform I. The devenv so that it can test against as many versions possible.

Please don't. At the moment we can't really afford to undergo any major change or refactoring just for the sake of it.

We can also include a matrix of tests in GitHub actions to ensure no breaking changes occur across supported versions

This is already in place and it already works.

(which Python 3.8+ is EOS at the end of October) but that's a completely different convo.

This is irrelevant. There are distributions that people use in the real world (e.g. Ubuntu 20.04 LTS that will be supported for another year at least) that still have 3.8. The upstream Python release cycle is something that we need to keep in mind, but we also need to target what real people use.

If I haven't made it clear, I would be open to doing this work or helping find others to potentially support it.

Thanks! Any help is always appreciated, but this one can be tricky.

As I said before, full support for 3.12 is something that we should add before the end of this year. Me or Mikhail will do it as soon as we have time.

[mkmkme]

Hey @kjaymiller,

I'm starting working on this issue now. There's a chance it's much larger than we expect, but let's see

[kjaymiller]

I'm not sure I understand.
In a previous comment you mentioned (correctly) that distutils is a problem for 3.12 because it was removed from that version. But distutils is not used in any test.

As this thread has grown in the last couple weeks I may have mistaken its placement as I only bumped into the issue when deploying the devenv (testing locally with 3.12 seemed to work for me but I may have manually updated some things.)

[kjaymiller]

@aiven-sal the issue around the devenv was with invoke's version being pinned.

I noticed had a dep the was also deprecated and already replaced it in newer versions of invoke.

[kjaymiller]

But distutils is not used in any test. It is, in fact, used only once in the actual library. I also don't understand why you proposed to replace it with dirty_equals

I was confusing a test that I saw in a previous PR.

You are correct that there is one place where distutils is used and it's for the imported function strtobool.

Most suggestions for replacing this is more verbose detection of values (e.g. if you know what the string will always be then look for it.) another option is dirty_equals which you can check if a value is giving a true-like string ("1", "true" "y" "yes", etc).

This would do the same thing without adding another piece of code to maintain and update.