outdated recipes

[obfusk]

Because many of the recipes I use have outdated versions (which may also have security implications), I'm currently building the following recipes with a patch to use the latest version (as of a week or so ago):
flask hostpython3 libffi openssl pyjnius python3 setuptools six sqlite3.

So far everything works for me without any problems (other than #2264).

I'd be happy to make PRs for those, but the latest versions may break for some users (and build environments).
What's the best/preferred way to handle this?

[AndreMiras]

Yes please go-ahead with the update. The best practice is for users to pin recipe version in their buildozer.spec if they want to be sure it doesn't get updated 😄

[obfusk]

The best practice is for users to pin recipe version in their buildozer.spec if they want to be sure it doesn't get updated

I don't recall seeing an option to pin recipes. How does one do that?

[AndreMiras]

The same way you pip install a pinned package. So basically users should pin versions in their requirements list if they want to, e.g. https://github.com/AndreMiras/EtherollApp/blob/v2020.0322/buildozer.spec#L40

[obfusk]

The same way you pip install a pinned package.

Ah. I mistakenly assumed that only worked for pip dependencies, not for p4a recipes. Good to know!

[obfusk]

I've started working on a script to check for updates in e.g. p4a recipes.
It currently outputs this:

$ cd python-for-android/pythonforandroid/recipes
$ checkupdates p4a-recipes
[USES HTTP] cdecimal
[USES HTTP] freetype
[USES HTTP] harfbuzz
[USES HTTP] libmysqlclient
[USES HTTP] libogg
[USES HTTP] libpq
[USES HTTP] libvorbis
[USES HTTP] libx264
[USES HTTP] libxml2
[USES HTTP] libxslt
[USES HTTP] zope
[CHECK FAILED] cdecimal
[CHECK FAILED] ethash
[CHECK FAILED] ffpyplayer_codecs
[CHECK FAILED] fontconfig
[CHECK FAILED] freetype
[CHECK FAILED] harfbuzz
[CHECK FAILED] icu
[CHECK FAILED] ifaddrs
[CHECK FAILED] libbz2
[CHECK FAILED] libcurl
[CHECK FAILED] libglob
[CHECK FAILED] libiconv
[CHECK FAILED] liblzma
[CHECK FAILED] libmysqlclient
[CHECK FAILED] libogg
[CHECK FAILED] libpq
[CHECK FAILED] librt
[CHECK FAILED] libsecp256k1
[CHECK FAILED] libtribler
[CHECK FAILED] libvorbis
[CHECK FAILED] libx264
[CHECK FAILED] matplotlib
[CHECK FAILED] netifaces
[CHECK FAILED] openal
[CHECK FAILED] preppy
[CHECK FAILED] pyicu
[CHECK FAILED] pyleveldb
[CHECK FAILED] pyogg
[CHECK FAILED] pyopenal
[CHECK FAILED] pyrxp
[CHECK FAILED] pysdl2
[CHECK FAILED] reportlab
[CHECK FAILED] scrypt
[CHECK FAILED] sdl2_image
[CHECK FAILED] sdl2_mixer
[CHECK FAILED] sdl2_ttf
[CHECK FAILED] storm
[CHECK FAILED] vlc
[POSSIBLY OUTDATED] Pillow = 7.0.0 -> 7.2.0
[POSSIBLY OUTDATED] apsw = 3.15.0-r1 -> 3.32.2-r1
[POSSIBLY OUTDATED] atom = 0.3.10 -> 0.4.1
[POSSIBLY OUTDATED] babel = 2.2.0 -> 2.8.0
[POSSIBLY OUTDATED] bcrypt = 3.1.7 -> 1.1.1
[POSSIBLY OUTDATED] boost = 1.69.0 -> 1.73.0
[POSSIBLY OUTDATED] cffi = 1.13.2 -> 1.14.1
[POSSIBLY OUTDATED] coverage = 4.1 -> 5.2.1
[POSSIBLY OUTDATED] cppy = 4e0b956 -> 1.1.0
[POSSIBLY OUTDATED] cryptography = 2.8 -> 3.0
[POSSIBLY OUTDATED] cython = 0.29.15 -> 0.29.21
[POSSIBLY OUTDATED] dateutil = 2.6.0 -> 2.8.1
[POSSIBLY OUTDATED] decorator = 4.2.1 -> 4.4.2
[POSSIBLY OUTDATED] enaml = 0.9.8 -> 0.10.2
[POSSIBLY OUTDATED] evdev = 0.4.7 -> 0.7.0
[POSSIBLY OUTDATED] feedparser = 5.2.1 -> 5.1.3
[POSSIBLY OUTDATED] ffmpeg = 007e03348dbd8d3de3eb09022d72c734a8608144 -> n2.8.5
[POSSIBLY OUTDATED] ffpyplayer = c99913f2317bf3840eeacf1c1c3db3b3d1f78007 -> 4.3.1
[POSSIBLY OUTDATED] gevent = 1.4.0 -> 20.6.2
[POSSIBLY OUTDATED] greenlet = 0.4.15 -> 0.4.16
[POSSIBLY OUTDATED] groestlcoin_hash = 1.0.1 -> 1.0.3
[POSSIBLY OUTDATED] jedi = 0.9.0 -> 0.17.2
[POSSIBLY OUTDATED] jpeg = 2.0.1 -> 2.0.5
[POSSIBLY OUTDATED] kiwisolver = 0846189 -> 1.2.0
[POSSIBLY OUTDATED] libexpat = master -> R_2_2_9
[POSSIBLY OUTDATED] libgeos = 3.7.1 -> 3.8.1
[POSSIBLY OUTDATED] libshine = c72aba9031bde18a0995e7c01c9b53f2e08a0e46 -> 3.1.1
[POSSIBLY OUTDATED] libsodium = 1.0.16 -> 1.0.18-RELEASE
[POSSIBLY OUTDATED] libtorrent = 1_2_1 -> 1.2.8
[POSSIBLY OUTDATED] libxml2 = 2.9.8 -> 2.9.10
[POSSIBLY OUTDATED] libxslt = 1.1.32 -> 1.1.34
[POSSIBLY OUTDATED] lxml = 4.2.5 -> 4.5.2
[POSSIBLY OUTDATED] m2crypto = 0.30.1 -> 0.36.0
[POSSIBLY OUTDATED] msgpack-python = 0.4.7 -> 0.5.6
[POSSIBLY OUTDATED] numpy = 1.18.1 -> 1.19.1
[POSSIBLY OUTDATED] omemo = 0.11.0 -> 0.12.0
[POSSIBLY OUTDATED] omemo-backend-signal = 0.2.5 -> 0.2.6
[POSSIBLY OUTDATED] opencv = 4.0.1 -> 4.4.0
[POSSIBLY OUTDATED] opencv_extras = 4.0.1 -> 4.4.0
[POSSIBLY OUTDATED] pandas = 1.0.3 -> 1.1.0
[POSSIBLY OUTDATED] png = 1.6.37 -> 1.6.35
[POSSIBLY OUTDATED] protobuf_cpp = 3.6.1 -> 3.12.4
[POSSIBLY OUTDATED] pyaml = 15.8.2 -> 20.4.0
[POSSIBLY OUTDATED] pycparser = 2.14 -> 2.20
[POSSIBLY OUTDATED] pycrypto = 2.7a1 -> 2.6.1
[POSSIBLY OUTDATED] pycryptodome = 3.6.3 -> 3.9.8
[POSSIBLY OUTDATED] pygame = 2.0.0-dev7 -> 1.9.6
[POSSIBLY OUTDATED] pymunk = 5.5.0 -> 5.6.0
[POSSIBLY OUTDATED] pynacl = 1.3.0 -> 1.4.0
[POSSIBLY OUTDATED] pyopenssl = 19.0.0 -> 19.1.0
[POSSIBLY OUTDATED] pyproj = 1.9.6 -> 2.6.1rel
[POSSIBLY OUTDATED] pytz = 2019.3 -> 2020.1
[POSSIBLY OUTDATED] pyusb = 1.0.0b1 -> 1.0.2
[POSSIBLY OUTDATED] pyzbar = 0.1.7 -> 0.1.8
[POSSIBLY OUTDATED] pyzmq = master -> 14.1.1
[POSSIBLY OUTDATED] regex = 2019.06.08 -> 2020.7.14
[POSSIBLY OUTDATED] ruamel.yaml = 0.15.77 -> 0.16.10
[POSSIBLY OUTDATED] sdl2 = 2.0.9 (f2ecfba915c54f7200f504d8b48a5dfe) -> 2.0.12
[POSSIBLY OUTDATED] setuptools = 49.2.1 -> 49.3.1
[POSSIBLY OUTDATED] shapely = 1.7a1 -> 1.7.0
[POSSIBLY OUTDATED] snappy = 1.1.7 -> 1.1.8
[POSSIBLY OUTDATED] sqlalchemy = 1.3.3 -> 1.3.18
[POSSIBLY OUTDATED] sympy = 1.1.1 -> 1.6.2
[POSSIBLY OUTDATED] ujson = 1.35 -> 3.1.0
[POSSIBLY OUTDATED] vispy = 0.4.0 -> 0.6.4
[POSSIBLY OUTDATED] websocket-client = 0.40.0 -> 0.57.0
[POSSIBLY OUTDATED] x3dh = 0.5.3 -> 0.5.9
[POSSIBLY OUTDATED] xeddsa = 0.4.4 -> 0.6.0
[POSSIBLY OUTDATED] zbarlight = 2.1 -> 3.0
[POSSIBLY OUTDATED] zeroconf = 0.24.5 -> 0.28.0
[POSSIBLY OUTDATED] zope = 4.1.3 -> 5.1.0
[POSSIBLY OUTDATED] zope_interface = 4.1.3 -> 5.1.0
24 ok, 72 outdated, 38 failed

[obfusk]

I'd be happy to make some PRs for newer versions.
But maybe not everything 😅.
Anything considered important that I should prioritise?

Currently outputs:

[USES HTTP] cdecimal
[USES HTTP] freetype
[USES HTTP] harfbuzz
[USES HTTP] libmysqlclient
[USES HTTP] libogg
[USES HTTP] libpq
[USES HTTP] libvorbis
[USES HTTP] libx264
[USES HTTP] libxml2
[USES HTTP] libxslt
[CHECK FAILED] audiostream
[CHECK FAILED] cdecimal
[CHECK FAILED] ethash
[CHECK FAILED] ffpyplayer_codecs
[CHECK FAILED] fontconfig
[CHECK FAILED] freetype
[CHECK FAILED] harfbuzz
[CHECK FAILED] icu
[CHECK FAILED] ifaddrs
[CHECK FAILED] libbz2
[CHECK FAILED] libcurl
[CHECK FAILED] libglob
[CHECK FAILED] libiconv
[CHECK FAILED] liblzma
[CHECK FAILED] libmysqlclient
[CHECK FAILED] libogg
[CHECK FAILED] libpq
[CHECK FAILED] librt
[CHECK FAILED] libsecp256k1
[CHECK FAILED] libtribler
[CHECK FAILED] libvorbis
[CHECK FAILED] libwebp
[CHECK FAILED] libx264
[CHECK FAILED] matplotlib
[CHECK FAILED] netifaces
[CHECK FAILED] openal
[CHECK FAILED] preppy
[CHECK FAILED] pyicu
[CHECK FAILED] pyleveldb
[CHECK FAILED] pyogg
[CHECK FAILED] pyopenal
[CHECK FAILED] pyrxp
[CHECK FAILED] pysdl2
[CHECK FAILED] reportlab
[CHECK FAILED] scrypt
[CHECK FAILED] sdl2_image
[CHECK FAILED] sdl2_mixer
[CHECK FAILED] sdl2_ttf
[CHECK FAILED] storm
[CHECK FAILED] vlc
[POSSIBLY OUTDATED] Pillow = 7.0.0 -> 8.0.1
[POSSIBLY OUTDATED] aiohttp = 3.6.2 -> 3.7.3
[POSSIBLY OUTDATED] apsw = 3.15.0-r1 -> 3.34.0-r1
[POSSIBLY OUTDATED] atom = 0.3.10 -> 0.6.0
[POSSIBLY OUTDATED] babel = 2.2.0 -> 2.9.0
[POSSIBLY OUTDATED] bcrypt = 3.1.7 -> 1.1.1
[POSSIBLY OUTDATED] boost = 1.69.0 -> 1.75.0
[POSSIBLY OUTDATED] cffi = 1.13.2 -> 1.14.4
[POSSIBLY OUTDATED] coverage = 4.1 -> 5.3.1
[POSSIBLY OUTDATED] cppy = 4e0b956 -> 1.1.0
[POSSIBLY OUTDATED] cryptography = 2.8 -> 3.3.1
[POSSIBLY OUTDATED] cython = 0.29.15 -> 0.29.21
[POSSIBLY OUTDATED] dateutil = 2.6.0 -> 2.8.1
[POSSIBLY OUTDATED] decorator = 4.2.1 -> 4.4.2
[POSSIBLY OUTDATED] enaml = 0.9.8 -> 0.12.0
[POSSIBLY OUTDATED] evdev = 0.4.7 -> 0.7.0
[POSSIBLY OUTDATED] feedparser = 5.2.1 -> 5.1.3
[POSSIBLY OUTDATED] ffmpeg = n4.3.1 -> n2.8.5
[POSSIBLY OUTDATED] gevent = 1.4.0 -> 20.12.1
[POSSIBLY OUTDATED] greenlet = 0.4.15 -> 0.4.17
[POSSIBLY OUTDATED] groestlcoin_hash = 1.0.1 -> 1.0.3
[POSSIBLY OUTDATED] hostpython3 = 3.8.5 -> 3.9.1
[POSSIBLY OUTDATED] jedi = 0.9.0 -> 0.18.0
[POSSIBLY OUTDATED] jpeg = 2.0.1 -> 2.0.90
[POSSIBLY OUTDATED] kiwisolver = 0846189 -> 1.3.1
[POSSIBLY OUTDATED] libexpat = master -> R_2_2_10
[POSSIBLY OUTDATED] libgeos = 3.7.1 -> 3.9.0
[POSSIBLY OUTDATED] libshine = c72aba9031bde18a0995e7c01c9b53f2e08a0e46 -> 3.1.1
[POSSIBLY OUTDATED] libsodium = 1.0.16 -> 1.0.18-RELEASE
[POSSIBLY OUTDATED] libtorrent = 1_2_1 -> 2.0.1
[POSSIBLY OUTDATED] libxml2 = 2.9.8 -> 2.9.10
[POSSIBLY OUTDATED] libxslt = 1.1.32 -> 1.1.34
[POSSIBLY OUTDATED] libzmq = 4.3.2 -> 4.3.3
[POSSIBLY OUTDATED] lxml = 4.2.5 -> 4.6.2
[POSSIBLY OUTDATED] m2crypto = 0.30.1 -> 0.37.1
[POSSIBLY OUTDATED] msgpack-python = 0.4.7 -> 0.5.6
[POSSIBLY OUTDATED] numpy = 1.18.1 -> 1.19.4
[POSSIBLY OUTDATED] omemo = 0.11.0 -> 0.12.0
[POSSIBLY OUTDATED] omemo-backend-signal = 0.2.5 -> 0.2.6
[POSSIBLY OUTDATED] opencv = 4.0.1 -> 4.5.1
[POSSIBLY OUTDATED] opencv_extras = 4.0.1 -> 4.5.1
[POSSIBLY OUTDATED] openssl = 1.1.1g -> 1.1.1i
[POSSIBLY OUTDATED] pandas = 1.0.3 -> 1.2.0
[POSSIBLY OUTDATED] png = 1.6.37 -> 1.6.35
[POSSIBLY OUTDATED] protobuf_cpp = 3.6.1 -> 3.14.0
[POSSIBLY OUTDATED] psycopg2 = 2.8.5 -> 2.8.6
[POSSIBLY OUTDATED] pyaml = 15.8.2 -> 20.4.0
[POSSIBLY OUTDATED] pycparser = 2.14 -> 2.20
[POSSIBLY OUTDATED] pycrypto = 2.7a1 -> 2.6.1
[POSSIBLY OUTDATED] pycryptodome = 3.6.3 -> 3.9.8
[POSSIBLY OUTDATED] pygame = 2.0.0-dev7 -> 2.0.1
[POSSIBLY OUTDATED] pynacl = 1.3.0 -> 1.4.0
[POSSIBLY OUTDATED] pyopenssl = 19.0.0 -> 20.0.1
[POSSIBLY OUTDATED] pyproj = 1.9.6 -> 3.0.0.post1
[POSSIBLY OUTDATED] python3 = 3.8.5 -> 3.9.1
[POSSIBLY OUTDATED] pytz = 2019.3 -> 2020.5
[POSSIBLY OUTDATED] pyusb = 1.0.0b1 -> 1.1.0
[POSSIBLY OUTDATED] pyzbar = 0.1.7 -> 0.1.8
[POSSIBLY OUTDATED] pyzmq = master -> 14.1.1
[POSSIBLY OUTDATED] regex = 2019.06.08 -> 2020.11.13
[POSSIBLY OUTDATED] ruamel.yaml = 0.15.77 -> 0.16.12
[POSSIBLY OUTDATED] sdl2 = 2.0.9 (f2ecfba915c54f7200f504d8b48a5dfe) -> 2.0.14
[POSSIBLY OUTDATED] setuptools = 49.2.1 -> 51.1.1
[POSSIBLY OUTDATED] shapely = 1.7a1 -> 1.7.1
[POSSIBLY OUTDATED] snappy = 1.1.7 -> 1.1.8
[POSSIBLY OUTDATED] sqlalchemy = 1.3.3 -> 1.3.22
[POSSIBLY OUTDATED] sqlite3 = 3.32.3 -> 3.34.0
[POSSIBLY OUTDATED] sympy = 1.1.1 -> 1.7.1
[POSSIBLY OUTDATED] ujson = 1.35 -> 4.0.1
[POSSIBLY OUTDATED] vispy = 0.4.0 -> 0.6.6
[POSSIBLY OUTDATED] websocket-client = 0.40.0 -> 0.57.0
[POSSIBLY OUTDATED] wsaccel = 0.6.2 -> 0.6.3
[POSSIBLY OUTDATED] x3dh = 0.5.3 -> 0.5.9
[POSSIBLY OUTDATED] xeddsa = 0.4.4 -> 0.6.0
[POSSIBLY OUTDATED] zbarlight = 2.1 -> 3.0
[POSSIBLY OUTDATED] zeroconf = 0.24.5 -> 0.28.7
[POSSIBLY OUTDATED] zope = 4.1.3 -> 5.2.0
[POSSIBLY OUTDATED] zope_interface = 4.1.3 -> 5.2.0
18 ok, 78 outdated, 40 failed

[AndreMiras]

Sweet! Can't think of any specific priority.
However try to not make it into a single pull request because our CI.
I'm asking because our CI will try to build updated recipes only and it has at least the following two limitations:

1. doesn't deal with two conflicting recipes in the same PR
2. would timeout after 30m or so

[obfusk]

Sweet! Can't think of any specific priority.

Any way to know which recipes are used a lot?
I've sent PRs for the ones I'm using myself.

[obfusk]

I've also added a command to check for updates to (pinned) recipes (and pypi packages without recipes) in a buildozer.spec:

$ checkupdates buildozer-spec -r my-local-p4a-recipes -r /path/to/python-for-android/pythonforandroid/recipes 
[POSSIBLY OUTDATED] setuptools = 51.0.0 -> 51.1.1
16 ok, 1 outdated, 0 failed

Unfortunately it might be a while before I have time to properly document & package the script so others can use it easily.
But it is available for anyone to use.

[AndreMiras]

Sweet! Can't think of any specific priority.

Any way to know which recipes are used a lot?
I've sent PRs for the ones I'm using myself.

We don't have a stats on used recipes yet, so I also think updating the ones you use yourself is a good approach.
Also very nice that you shared your script and improve it over time

[Julian-O]

Closing - there is good discussion here, but no action required by python-for-android team.