Drop older Python support, require Python >=3.7

[albertz]

Require Python version >=3.7.

See #1326 for a follow-up.

---

This is just to start some discussion, specifically what we want to drop. Surely <=Python2, but probably also <Python3.x for some x (e.g. 7). We could also say it might make more sense maybe in 6 months or so. This issue implies then to clean up the code accordingly.

The purpose is to clean up some code where Python 2 vs 3 needs extra separate logic, and also where we could make use of newer syntax. I want to collect some of these arguments (and their needed Python version) here.
Sometimes some things can also be fixed by from __future__ import ..., which is ok (but also not for all features).

Then there might be practical issues on some systems we use, in some conditions, which just doesn't provide newer Python versions, or would make it very inconvenient. E.g. for some time, RASR was linked to Python 2 by default, and also did not support Python 3. I think that is fixed now? Then Ubuntu 16 has some older Python, but maybe we can just agree to use anyway a newer Python version?

---

Now some arguments, or relevant features:

• Type annotations. Like def func(a: str) -> Data or whatever. I think since Python 3.5.
  However, there is the issue with forward declaration. E.g.:

  class Data:
     def bla(self) -> Data:
       ...
  

  This requires >=Python 3.10 I think.
  Or it requires from __future__ import annotations since Python 3.7 (doc).
  The earlier workaround was to use a string as annotation instead, i.e. def bla(...) -> "Data" (referred to as forward declaration). But I think this support was always somewhat limited (e.g. not sure if PyCharm will correctly pick this up), and will also likely go away at some point, or being deprecated, so I would argue to avoid this (because otherwise we will have a similar discussion soon again). But this means we need at least Python 3.7.
  (There are ongoing discussions around this. E.g. see pydantic #2678 (HN), PEP 649 (python-dev discussion).)
  I would also argue, in any case, type inference must work in PyCharm. If that doesn't work, we should not use it.
  In any case, I think these type annotations are actually one of the major reasons for us to drop older support.

• dict.items() or dict.keys() returns a non-deterministic order in earlier Python versions.
  Because you usually want to have this deterministic, you would use sorted(dict.items()).
  Since Python 3.6, this is now deterministic (using the insertion order) (see here).
  So, from Python 3.6 on, you can drop the sorted, or explicit usage of OrderedDict.
  We should still be a bit careful here though. Some issues:

  • When the user now runs such code with Python 3.5, there will not be any error. But it might produce strange very hard to debug issues. We always should avoid hard-to-debug issues at all cost. So If we adopt this, we should also have some assert py_version >= (3,6) somewhere.
  • I can't imagine such a situation now, but maybe there could be cases where we must match the old behavior using sorted, i.e. sorted by alphabet, or maybe to stay compatible with some old data?
  • I can't imagine such a situation now, but maybe there could be other cases where this would be bad? E.g. you remove some key, then add some key. Then list(dict.keys()) would be different (whereas sorted would always be the same). Not sure if this could cause any issues.

• f-strings, since Python 3.6.

• Ubuntu 16.04 (so far still the standard on many of our systems) comes with Python 3.5.
  Ubuntu 20.04 comes with Python 3.8.
  Ubuntu 22.04 comes with Python 3.10.

• typing.Protocol requires Python >=3.8.

• Slash (/) in function argument list to mark positional-only arguments (doc) requires Python >=3.8.

[curufinwe]

We use RETURNN also on some Centos 7 / Amazon EC2 Linux machines with python 3.6. It would be nice if support for python 3.6 remained let's say until the end of the year.

Regarding the sorted issue: this is not performance critical, so I think it's best to just leave the sorted call there if a deterministic order is required.

[albertz]

Yea, this whole issue is mostly not about performance at all, but really about cleaning up code. And for that, using sorted might be more annoying. But sure, it's just a small thing.

Python 3.6 might be problematic for the annotation arguments above (specifically forward declarations).

But so maybe we just wait with this code cleaning up effort and dropping earlier Python support until end of year then. So you expect that the Amazon EC2 machines will come with newer Python then?

[curufinwe]

Actually the one EC2-Linux machine I checked had python3.7 but that one was ARM based. I'm more concerned about the whole Centos situation. Redhat stopped support for Centos stable releases and now we'll have to decide what distribution we want for our prod environment. For now we stick to Centos 7. Maybe we can start by dropping support for python 2.7? But maybe you prefer to do all changes in one go?

[JackTemaki]

Does the current style of using rST docstring annotations and type hinting as in https://www.python.org/dev/peps/pep-0484 any restrictions on where a type hinting is not possible? I think it will be quite the effort to change all docstrings and functions, so what is the main reason to do this? Or do you want to do a gradual change, so that only new code uses the new annotation style?

[albertz]

Maybe we can start by dropping support for python 2.7? But maybe you prefer to do all changes in one go?

I think it's easier to do in one go. For only dropping Python 2, there is actually also not too much to be done, this is really minor then. That is why I also hesitate to do just that, because it's really only minor, and having Python 2 support might outweigh the 5 lines of code we safe or so.

I think it will be quite the effort to change all docstrings and functions, so what is the main reason to do this?

Sure, this will be some effort. But I think it's required at some point in time to keep the code style modern. The Python world is slowly moving towards having more and more static typing.

Or do you want to do a gradual change, so that only new code uses the new annotation style?

Sure, we have to be pragmatic about this. We would have a rule that all new code must use annotations.

And we would slowly change the old code, but these changes would not be a single commit.

I think with some automation, we can also get quite far, and then need to check where it is incorrect or broken, which can also be automated (e.g. with help of PyCharm and/or MyPy).

[albertz]

@mihajenko How can I understand your downvote?

[michelwi]

Bevor dropping the support for python2 we should make sure that in the instances where returnn has to talk to rasr (sequence training, Baum-Welch training, feature-extraction) this still works.

I.e. finally make rasr compatible with python3 or handle possible incompatibilities of the used pickle protocols.

[albertz]

I.e. finally make rasr compatible with python3

I thought it already is compatible with Python 3? What is the state here?

handle possible incompatibilities of the used pickle protocols.

I think there were just problems when we mixed Python 2 and Python 3, although we even resolved these problems with some workarounds.

But when we drop Python 2, we can clean this up. But then RASR must support Python 3 as well.

Or maybe we could also say that the modules which are imported by RASR (e.g. SprintInterface) must still be importable and compatible with Python 2?

[curufinwe]

There is a version of RASR that uses python3 instead of python2 and I can port it to the public version if there is demand for that.

[albertz]

There is a version of RASR that uses python3 instead of python2 and I can port it to the public version if there is demand for that.

I think this is a must such that we can safely drop Python 2.

[albertz]

Sorry, the issue is too broad. FastAPI is used as a model serving API by a decent number of ML applications; PyTorch barely got support out for Python 3.8, TensorRT barely graduated from past 3.6 ... In this context, suggesting to drop support for 3.6, 3,7 is not well-thought through.

@mihajenko You realize this here is not about pydantic/FastAPI but about the returnn project?
Also, I suggested to drop support for Python 2. For other versions, that's exactly what I wanted to discuss here.
You refer to this TF TensorRT integration? But Python >=3.7 is supported there or not? But I'm not sure how much this is relevant for us anyway.

[albertz]

Ah, my bad. Very sorry, my reply was the one not throught through.

@mihajenko No problem. I assumed so. Your references to PyTorch / TensorRT are interesting anyway.