Support Python 3.10 -- find replacement for stdlib_list

[szabi]

stdlib_list is an archived project. The last Python version supported is 3.9; and >=3.10 will never be added.

Currently, using pydeps with Python 3.10 falls back to using the 3.9 standard lib listing. That's a viable workaround for now, but with growing stdlib differences with every Python version (the actual current stable version is already 3.11 -- two versions newer) the fallback will be less and less 'correct' and will contain more and more errors.

For pydeps to stay useful and relevant, it seems utmost important to find a replacement for stdlib_list or take over the maintenance of the closed/archived project python-stdlib-list.

[thebjorn]

This is actually not a huge problem for pydeps. The current stdlib_list package is (and will probably always be) on PyPI and is quite comprehensive for the python versions it supports. Pydeps will use the lists from the latest supported python version from stdlib_list, and since new stdlib modules are added rather infrequently to python this works well.

The modules that are in 3.11 but not in stdlib_list's 3.9 are

{'_contextvars', '_tkinter', '_socket', '_tokenize', 'tomllib', '_symtable', '_warnings', '_md5', '_posixsubprocess', '_random', '_sre', '_sha256', '_struct', '_locale', '_codecs', '_codecs_hk', '_xxsubinterpreters', '_bisect', '_posixshmem', '_sha1', '_operator', '_sqlite3', '_pickle', '_string', 'pyexpat', '_codecs_cn', '_codecs_jp', '_functools', '_io', '_codecs_tw', '_statistics', '_winapi', '_bz2', '_curses_panel', '_abc', '_dbm', '_stat', '_decimal', '_codecs_kr', '_signal', 'xxsubtype', '_msi', '_multibytecodec', 'nt', '_crypt', '_frozen_importlib', '_overlapped', '_csv', '_sha512', '_ssl', '_tracemalloc', '_queue', '_ast', '_curses', '_lzma', '_opcode', '_blake2', '_scproxy', '_zoneinfo', '_uuid', '_imp', '_typing', '_sha3', '_ctypes', '_heapq', '_gdbm', '_weakref', '_frozen_importlib_external', '_multiprocessing', '_lsprof', '_collections', '_hashlib', '_asyncio', '_elementtree', '_datetime', '_codecs_iso2022', '_json'}

Since the list returned by list(sys.stdlib_module_names | set(sys.builtin_module_names)) only contains top-level modules, there are 656 modules that are listed in stdlib_list's 3.9 modules that are not returned from py3.11. I'm not sure there is an efficient way of populating these.

The dev/py310-native-stdlib-list branch (https://github.com/thebjorn/pydeps/tree/dev/py310-native-stdlib-list) implements the py310+ PR to stdlib_list (pypi/stdlib-list#56).

[szabi]

Given that stdlib-list was archved with "This repo is now archived. I no longer have the spoons to maintain this in my spare time." (from README.md) I guess, even PRs won't be merged.

You could try contacting @jackmaney and maybe fork stdlib-list and learn how he compiled the complete list (or even take over maintenance).

The actual effort to maintaining it might be well worth for pydeps to exist: new Python versions come around once a year, in October, so the maintenance effort likely is limited.

[thebjorn]

I'm afraid you might seriously underestimate the effort it takes to maintain a project like stdlib_list, which needs to always be ahead of new python versions and the challenges that poses with installations of alpha/beta/rc versions both locally and in the CI.

If you want to take it on though, I'm sure everyone would be delighted :-) Documentation for how the list was compiled, as well as code for doing it is in the repo.

For pydeps, I'm much more likely to vendorize the resulting lists than taking over maintenance - but since Python versions < 3.10 are already covered (and unlikely to change, since it goes against python's non-current micro-version release policy) there really is no practical point.

Of course, if you have a real issue, and not just a fear of future viability, I would be happy to look at it.

[szabi]

I absolutely believe that I am underestimating it. However, I would have assumed that for keeping pydeps productive, as a "MVP" it would suffice to keep up to date with stable versions, which come out once a year with the current schedule.

Vendorizing the list does not solve the problem in so far, that the list still has to be compiled somehow (and stdlib-list seems to be up to the task per se, unless it falls behind and stays unmaintained for too long a time).

I don't quite get the "there really is no practical point", as neither the current nor the last-but-current versions are supported: The last supported version, 3.9 is already beyond it's end of full support date (2022-05-17).

I fully understand that you don't want to take over the maintenance of it, I am certainly not trying to convince or urge you, but then, I guess, one has to think about other ways to procure the list (for vendorizing).

[stefan6419846]

As already mentioned in #95 (comment), Python since version 3.10 does have in-built support for retrieving such a list - it just has to be evaluated to which extent this actually matches with the stdlib_list data (3.10.txt 3.11.txt).

[thebjorn]

@szabi @stefan6419846 micro-releases (the x in 3.9.x) of old Python versions, i.e. <3.10, do not grow new standard library modules (see https://devguide.python.org/developer-workflow/development-cycle/ and https://devguide.python.org/developer-workflow/stdlib/), ergo, the lists that stdlib_list has for those versions are correct now, and will continue to be correct in the future. The machinery for maintaining those lists will fail though, as soon as the documentation indexes for those versions disappear (that is how the lists are generated). It is therefore more reasonable to vendorize those lists than any other approach to work with older versions of Python.

stdlib_list does not currently support Python>=3.10, and is not likely to do so in the future.

The proposed, but not merged, patch for >=3.10 support generates lists simply from

list(sys.stdlib_module_names | set(sys.builtin_module_names))

which has also been suggested numerous times for pydeps. Per my comment above, this list is missing 656 items that are in the Python 3.9 list for stdlib_list.

All of the new modules in 3.10, 3.11 are underscore-prefixed, except two: nt and tomllib. The nt module is not picked up by stdlib_list due to the way its list is generated, however it is manually added in pydeps (https://github.com/thebjorn/pydeps/blob/master/pydeps/pystdlib.py#L36)).

It would therefore give worse 3.10, 3.11 support today to use the proposed stdlib_list 3.10/1 patch, and it would give long-time issues to try to support stdlib_list's generation mechanism for older Python versions.

Pydeps will therefore continue to use the current published version of stdlib_list, and if needed add extra modules in https://github.com/thebjorn/pydeps/blob/master/pydeps/pystdlib.py#L36 depending on Python version (i.e. tomllib should only be added if the Python version is >=3.10). If stdlib_list disappears in the future, pydeps will only copy out (i.e. vendorize) the lists that stdlib_list has generated for older versions (as explained above these will continue to be valid and unchanged for older Python versions.

Pydeps might grow a feature where any module starting with a prefix generated from list(sys.stdlib_module_names | set(sys.builtin_module_names)) to more automatically support Python versions > 3.11 but there is currently no need, and it will require rewriting https://github.com/thebjorn/pydeps/blob/master/pydeps/py2depgraph.py#L253 to be a prefix match.

PR are always very welcome if anyone wants to do this.

[szabi]

Just for reference, stdlib_list is no longer archived and unmaintained, but has been transferred to the pypi team: https://github.com/pypi/stdlib-list

[thebjorn]

@szabi yes, I'm aware :-)

ba358d0