Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file

[Zen-CODE]

Versions

• Python: 2.7.2
• OS: ubuntu 16.04, android 6
• Kivy: 1.10.0
• Cython: 0.25.2

Description

This is my setup.

cython 0.25.2
ubuntu 16.04
android ndk r16b
android sdk 28
api=27
python-for-android - using branch set_app_platform_19

the requirements contains pyjnius, and other calls to it works e.g. starting the service.

The app runs and imports work fine. The service starts, but it fails on the first import. The import is valid (from the root of the app folder, not the service folder), but it crashes with a jnius.so error. Even if the error is trapped, it just crashes without going into the exception clause.

The error is logged here.

https://gist.github.com/Zen-CODE/0a11a5881bb80caeeee17b8cb5a2d48e

buildozer.spec

Command:

buildozer android debug deploy run

Spec file:

[app]

# (str) Title of your application
title = CAMI-Apps

# (str) Package name
package.name = camiweb.com

# (str) Package domain (needed for android/ios packaging)
package.domain = speedtest.maths

# (str) Source code where the main.py live
source.dir = .

# (list) Source files to include (let empty to include all the files)
# source.include_exts = py,png,jpg,kv,atlas,ini

# (list) Source files to exclude (let empty to not exclude anything)
#source.exclude_exts = spec

# (list) List of directory to exclude (let empty to not exclude anything)
#source.exclude_dirs = working

# (list) List of exclusions using pattern matching
#source.exclude_patterns = license,images/*/*.jpg

# (str) Application versioning (method 1)
version = 2.1.10

# (str) Application versioning (method 2)
# version.regex = __version__ = ['"](.*)['"]O
# version.filename = %(source.dir)s/main.py

# (list) Application requirements
# comma seperated e.g. requirements = sqlite3,kivy
# requirements = python, sqlite3, kivy==1.10.0, openssl
# For new toolchain
requirements = sqlite3, kivy==1.10.0, openssl, pyjnius, android

# (str) Custom source folders for requirements
# Sets custom source for any requirements with recipes
# requirements.source.kivy = ../../kivy

# (list) Garden requirements
#garden_requirements =

# (str) Presplash of the application
presplash.filename = working/splash/presplash.jpg

# (str) Icon of the application
icon.filename = %(source.dir)s/icon.png

# (str) Supported orientation (one of landscape, portrait or all)
orientation = landscape

# (list) List of service to declare
#services = NAME:ENTRYPOINT_TO_PY,NAME2:ENTRYPOINT2_TO_PY
services = camiapps:service/main.py

#
# OSX Specific
#

#
# author = © Copyright Info

#
# Android specific
#

# (bool) Indicate if the application should be fullscreen or not
fullscreen = 1

# (list) Permissions
android.permissions = INTERNET

# (int) Android API to use
android.api = 27

# (int) Minimum API required
android.minapi = 19

# (int) Android SDK version to use
# android.sdk = 20
android.sdk = 28

# (str) Android NDK version to use
#android.ndk = 9c

# (bool) Use --private data storage (True) or --dir public storage (False)
#android.private_storage = True

# (str) Android NDK directory (if empty, it will be automatically downloaded.)
android.ndk_path = /home/kivy/Android/android-ndk-r16b/

# (str) Android SDK directory (if empty, it will be automatically downloaded.)
#android.sdk_path =

# (str) ANT directory (if empty, it will be automatically downloaded.)
#android.ant_path =

# (str) python-for-android git clone directory (if empty, it will be automatically cloned from github)
#android.p4a_dir =
p4a.source_dir = /home/kivy/Repos/python-for-android/

# (list) python-for-android whitelist
#android.p4a_whitelist =

# (str) Android entry point, default is ok for Kivy-based app
#android.entrypoint = org.renpy.android.PythonActivity

# (list) List of Java .jar files to add to the libs so that pyjnius can access
# their classes. Don't add jars that you do not need, since extra jars can slow
# down the build process. Allows wildcards matching, for example:
# OUYA-ODK/libs/*.jar
#android.add_jars = foo.jar,bar.jar,path/to/more/*.jar

# (list) List of Java files to add to the android project (can be java or a
# directory containing the files)
#android.add_src =
#android.add_src = java/com

# (str) python-for-android branch to use, if not master, useful to try
# not yet merged features.
#android.branch = master

# (str) OUYA Console category. Should be one of GAME or APP
# If you leave this blank, OUYA support will not be enabled
#android.ouya.category = GAME

# (str) Filename of OUYA Console icon. It must be a 732x412 png image.
#android.ouya.icon.filename = %(source.dir)s/data/ouya_icon.png

# (str) XML file to include as an intent filters in <activity> tag
#android.manifest.intent_filters =

# (list) Android additionnal libraries to copy into libs/armeabi
#android.add_libs_armeabi = libs/android/*.so
#android.add_libs_armeabi_v7a = libs/android-v7/*.so
#android.add_libs_x86 = libs/android-x86/*.so
#android.add_libs_mips = libs/android-mips/*.so

# (bool) Indicate whether the screen should stay on
# Don't forget to add the WAKE_LOCK permission if you set this to True
#android.wakelock = False

# (list) Android application meta-data to set (key=value format)
#android.meta_data =

# (list) Android library project to add (will be added in the
# project.properties automatically.)
#android.library_references =

# (str) Android logcat filters to use
#android.logcat_filters = *:S python:D

# (bool) Copy library instead of making a libpymodules.so
#android.copy_libs = 1

#
# iOS specific
#

# (str) Name of the certificate to use for signing the debug version
# Get a list of available identities: buildozer ios list_identities
#ios.codesign.debug = "iPhone Developer: <lastname> <firstname> (<hexstring>)"

# (str) Name of the certificate to use for signing the release version
#ios.codesign.release = %(ios.codesign.debug)s


[buildozer]

# (int) Log level (0 = error only, 1 = info, 2 = debug (with command output))
log_level = 2

# (int) Display warning if buildozer is run as root (0 = False, 1 = True)
warn_on_root = 1


#    -----------------------------------------------------------------------------
#    List as sections
#
#    You can define all the "list" as [section:key].
#    Each line will be considered as a option to the list.
#    Let's take [app] / source.exclude_patterns.
#    Instead of doing:
#
#[app]
#source.exclude_patterns = license,data/audio/*.wav,data/images/original/*
#
#    This can be translated into:
#
#[app:source.exclude_patterns]
#license
#data/audio/*.wav
#data/images/original/*
#


#    -----------------------------------------------------------------------------
#    Profiles
#
#    You can extend section / key with a profile
#    For example, you want to deploy a demo version of your application without
#    HD content. You could first change the title to add "(demo)" in the name
#    and extend the excluded directories to remove the HD content.
#
#[app@demo]
#title = My Application (demo)
#
#[app:source.exclude_patterns@demo]
#images/hd/*
#
#    Then, invoke the command line with the "demo" profile:
#
#buildozer --profile demo android debug

Logs

https://gist.github.com/Zen-CODE/0a11a5881bb80caeeee17b8cb5a2d48e

[Zen-CODE]

It seems to be that this issue is related to variable values not being shared by the main app and the service. To elaborate.

There is a dictionary in a file "helpers/localos.py". This dictionary is used as a singleton and populated on the first call to some of the module functions. From there on, the module functions return values loaded from the stored dictonary. This means that we only call the autoclass function once.

Using the old toolchain, the dictionary persisted between the app and the service.

Using the new toolchain, the dictionary filled by the app is empty for the service. The service then tries to populate it and crashes on the 2nd call of:

activity = autoclass('org.kivy.android.PythonActivity')

Is there any known reason when this might be happening? The loss of state between the app and the service is cetainly different behaviour.

Dictionary persists between app and service
    buildozer command: buidlozer android_old
    python-for-android branch: old_toolchain
    android sdk: 23
    android ndk: 9c

Dictionary does not persist between app and service
    buildozer command: buidlozer android
    python-for-android branch: set_app_platform_19
    android sdk: 28
    android ndk: r16b

Would it help if I prepared a minimal example demonstrating the issue? Or is there a known cause?

[Zen-CODE]

Could this be due to the changed "Class-loading behavior" described in the changes here?

https://developer.android.com/about/versions/oreo/android-8.0-changes

[Zen-CODE]

Adding code to avoid the above issues, it seems that using the new toolchain, the zipfile module cannot open a valid zipfile. It throws the error

    File is not a zip file

But I have verified it is by opening it with another tool. And the same file can be opened by zipfile using the android_old toolchain....

[AndreMiras]

Thanks for investigating 👍

Would it help if I prepared a minimal example demonstrating the issue? Or is there a known cause?

Yes please do. I'm not sure I'm following, but know that at least on the new toolchain the service and the main app are two different process, so the memory is not shared.
See ee58dee#diff-419ff95ffcc5c8bb0c2b40a76a023bb2R12

[Zen-CODE]

It turns out that neither the service not the app itself can open a normal zipfile. The exact same zip was open-able using the android_old toolchain and the app opens the zip no problem. It only fails when packaged as an APK. The error log is here:

https://gist.github.com/Zen-CODE/867c8585d29e54a638063f2964366fdc

The minimal project that demonstrates the issue is here.

https://github.com/Zen-CODE/kivybits/tree/master/Examples/Services

Although the app has storage permissions, I suspect that it has do with the new protocol of requesting runtime permissions, and the 'zipfile.BadZipfile: File is not a zip file' message really means 'permission denied'. It will test this by seeing if I can open the file as a binary and read bits from it....

Update. Nope. The file can be read as a binary file using open(my_zip, 'rb') no problem. So it's not a permissions problem. So why is the zipfile module failing to open it?

[rasmuskreiner]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class:
https://pastebin.com/pxJnEiR7

[OptimusGREEN]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class:
https://pastebin.com/pxJnEiR7

This link isn't working but yes I have the same issue. tar file doesn't extract either.

[rasmuskreiner]

I had the exact same issue, and decided to do a work around with pyjnius. See this java-class:
https://pastebin.com/pxJnEiR7

This link isn't working but yes I have the same issue. tar file doesn't extract either.

Link should be working now....
BTW usage is as follows:

unzip = autoclass("unzipFile")
unzipper = unzip()
unzipper.extractFile(filePathToZipFile, filePathToDestinationFolder)

[Zen-CODE]

@rasmuskreiner. Thanks. I'll keep that in mind. But what worries me is that other things are also failing. E.g. If I try and autoclass in the service, it also crashes. So there is somethings more sinister afoot methinks...

[Zen-CODE]

@rasmuskreiner. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks

[Zen-CODE]

@OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that might be the cause of my issues. Still investigating....

[OptimusGREEN]

Hi, No I don't think I was using threading.  Sent from my Samsung Galaxy smartphone. -------- Original message --------From: Richard Larkin <notifications@github.com> Date: 06/01/2019 13:19 (GMT+00:00) To: kivy/python-for-android <python-for-android@noreply.github.com> Cc: OptimusGREEN <johnmcbridegreen@gmail.com>, Mention <mention@noreply.github.com> Subject: Re: [kivy/python-for-android] Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file (#1388) @OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that might be the cause of my issues. Still investigating.... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kivy/python-for-android","title":"kivy/python-for-android","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kivy/python-for-android"}},"updates":{"snippets":[{"icon":"PERSON","message":"@Zen-CODE in #1388: @OptimusGREEN. Were you trying to extract on a background thread by any chance? Looks like that *might* be the cause of my issues. Still investigating...."}],"action":{"name":"View Issue","url":"#1388 (comment)"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#1388 (comment)", "url": "#1388 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[rasmuskreiner]

Sorry the late response. I’m on paternity leave hence the slow reply rate… As I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src". Eg. android.add_src = unzipFile.java.

…

Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin ***@***.***>: @rasmuskreiner <https://github.com/rasmuskreiner>. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1388 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw>.

[OptimusGREEN]

That java files don't always get copied into the build folder by buildozer. There's obviously a bug so need to manually copy them into the path that buildozer looks for them which will be in the build error, then it works ok.  Sent from my Samsung Galaxy smartphone. -------- Original message --------From: rasmuskreiner <notifications@github.com> Date: 16/01/2019 19:36 (GMT+00:00) To: kivy/python-for-android <python-for-android@noreply.github.com> Cc: OptimusGREEN <johnmcbridegreen@gmail.com>, Mention <mention@noreply.github.com> Subject: Re: [kivy/python-for-android] Unable to open a zip on Android: zipfile.BadZipfile: File is not a zip file (#1388) Sorry the late response. I’m on paternity leave hence the slow reply rate… As I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src". Eg. android.add_src = unzipFile.java.

Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin ***@***.***>: @rasmuskreiner <https://github.com/rasmuskreiner>. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using "android.add_src" or "android.add_jars"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1388 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw>.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread. {"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/kivy/python-for-android","title":"kivy/python-for-android","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/kivy/python-for-android"}},"updates":{"snippets":[{"icon":"PERSON","message":"@rasmuskreiner in #1388: Sorry the late response. I’m on paternity leave hence the slow reply rate…\n\nAs I recall I was also struggling with this part. Ultimately I ended up putting the java files in the root directory, and adding them to the “android.add_src\". Eg. android.add_src = unzipFile.java.\n\n\u003e Den 30. dec. 2018 kl. 04.58 skrev Richard Larkin \u003cnotifications@github.com\u003e:\n\u003e \n\u003e @rasmuskreiner \u003chttps://github.com/rasmuskreiner\u003e. I'm struggling to get the java class detected. How do you include you java class file in your buildozer.spec file? Using \"android.add_src\" or \"android.add_jars\"? Do you have to do anything extra to register this class on android? e.g. set the CLASSPATH? Thanks\n\u003e \n\u003e —\n\u003e You are receiving this because you were mentioned.\n\u003e Reply to this email directly, view it on GitHub \u003chttps://github.com/kivy/python-for-android/issues/1388#issuecomment-450537618\u003e, or mute the thread \u003chttps://github.com/notifications/unsubscribe-auth/AFHUKzOjSnqhSHEifS0fPNyx4tBz2Xdqks5u-DoBgaJpZM4XEJmw\u003e.\n\u003e \n\n"}],"action":{"name":"View Issue","url":"#1388 (comment)"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "#1388 (comment)", "url": "#1388 (comment)", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]