"Illegal Instruction" by googlesamples-assistant-pushtotalk in Pi Zero(grpc issue)

[Dimeiza]

Problem:

I installed Googlesamples-assistant-pushtotalk(Google Assistant Service) in Pi Zero recently , but it displays "Illegal instruction" error message and I can't execute it.

This messages didn't appear when I installed before(3/25/2018).

Environment:

Raspberry Pi Zero W
RASPBIAN STRETCH WITH DESKTOP (2018-03-13)

procedure:

I executed official installation procedure.

https://developers.google.com/assistant/sdk/guides/service/python/embed/install-sample

cause:

I think that it is caused by grpcio pip package .

I found grpcio package installation with "grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl" from log of installation Google Assistant. This file name looks like to compliant armv6 architecture, but "Illegal Instruction" error was occurred.

So, I tried to install grpcio from source before Google Assistant installation.

python -m pip install --upgrade --no-binary :all: grpcio
python -m pip install --upgrade google-assistant-library
python -m pip install --upgrade google-assistant-sdk[samples]
python -m pip install --upgrade google-auth-oauthlib[tool]

...

Then, "Illegal instruction" error was not occurred and I can use pushtotalk well.

In Pi Zero, latest version of grpcio(1.11.0) is installed from binary package. but previous version(1.10.0) is installed from source.

I think this is why "Illegal Instruction" is occurred recently.

Request:

Would you check this problem please?
If I should send this issue gRPC team, please tell me. And I will do so.

Thank you for taking your time.

[proppy]

/cc @mehrdada @kpayson64

[gerardwr]

Just to report that I had this issue too when I installed Google Assistant on my ZeroW.

Thought it was related to the ReSpeaker sound card I used, So I reported it here:
respeaker/seeed-voicecard#62

Excellent that you have found the possible source of the error.

[mehrdada]

Not sure if this happens to be the root cause or not but just by looking at PyPI index, I can confirm that none of gRPC ARM packages are “arm6”. All of them are “arm7l” designated. https://pypi.org/project/grpcio/1.11.0/#files

I’m curious how you got the arm6 installed from a binary wheel?

[Dimeiza]

according a installation log of google assistant...

Requirement not upgraded as not directly required: six>=1.9.0 in ./env/lib/python3.5/site-packages (from tenacity<5,>=4.1.0; extra == "samples"->google-assistant-sdk[samples]) (1.11.0)
Collecting grpcio>=1.3.5 (from google-assistant-grpc==0.1.0; extra == "samples"->google-assistant-sdk[samples])
  Downloading https://www.piwheels.org/simple/grpcio/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl (7.9MB)
    100% |????????????????????????????????| 7.9MB 317kB/s

I have confirmed this file can be downloaded now.

It seems that this file is a binary file to complient armv7 architecture, because I heard other user could run Google Assistant with this package in Raspberry Pi 3.

I don't grasp what happened because I don't know well about python package system.

[mehrdada]

I don't know who operates https://www.piwheels.org but for sure it is not an officially supported gRPC artifact. Some third party must have built it.

The official packages are all here: https://pypi.org/project/grpcio/#files

[proppy]

/cc @bennuttall who operate piwheels: is piwheels repo set by default as a PyPi mirror on Raspbian? Does it takes precedence over PyPi if arm6 wheels are missing are there is only arm7 wheels?

[bennuttall]

Ah, I can see what's happened here. It hasn't come up before.

Before piwheels, PyPI didn't accept Arm wheel uploads, so we automated building and hosting Arm wheels built from sdist of all packages on PyPI.

Since PyPI uploads moved to warehouse's back-end (long before the very recent front-end switch), Arm wheel uploads have been permitted.

Since Raspbian userland is 32-bit only, regardless on Pi model/CPU, all wheels built on a Pi 2/3 are identical to those built on Pi 1/Zero, despite the platform tag being armv6l/armv7l. But they're actually armv6 because of the agnostic userland.

We build "armv7l" wheels and provide duplicate files tagged as "armv6l" because they are compatible with all Pis in Raspbian. See RTIMULib for example.

When a compatible wheel is available on PyPI, all we do is download it, not build it from source. If this happens it is usually a pure python wheel which is not problem. However, in this case Armv7 wheels have been provided but not Armv6 (note: PyPI/warehouse does not allow uploading multiple identical files even with different tags like we do on piwheels).

is piwheels repo set by default as a PyPi mirror on Raspbian?

It's configured as an additional index in /etc/pip.conf. If you remove this file, your requests will only hit PyPI. pip will choose wheels over sdist, so more often than not, pip will chose files from piwheels over PyPI.

Does it takes precedence over PyPi if arm6 wheels are missing are there is only arm7 wheels?

Precedence is determined by the provider of the best matching file. Usually (e.g numpy) piwheels is preferred due to providing a wheel over PyPI's sdist, but in the case of a pure python package providing a wheel on PyPI (e.g. gpiozero), they are equal and PyPI is preferred as the primary index.

If the user is on an "armv7" platform (Pi 2 or 3), pip will only match armv7 wheels. armv6 is simply not a match, the same way x86 isn't. So in this case, the user would get the armv7 wheel from pypi.

If the user is on an armv6 platform (Pi 1/Zero) and no armv7 wheel is provided on PyPI, pip's choice is between a seemingly valid platform wheel from piwheels vs. sdist from PyPI. piwheels wins.

However, if the original wheel is true armv7 (as opposed to armv6 userland posing as armv7) we have a problem, because piwheels copied it to "armv6l" and is providing that to Pi 1/Zero users.

How was the armv7 wheel built? QEMU maybe? If it contains Armv7 instructions it will not work on Pi 1/Zero (and I'm not even sure about Pi 2/3).

Resolution:

• If the wheel is true armv7, does it definitely work on a Pi 2/3? If it was actually built by Raspbian, I have no idea what happened.
• If it works on a Pi 2/3 but not on Pi 1/Zero, can you rebuild on a Pi 1 and try to upload it to PyPI? If it differs from the armv7 wheel that will work fine - just let me know and I'll delete it from piwheels.
• If you remove /etc/pip.conf and pip install from PyPI on a Pi 1/Zero, does building from source work? If so, we can just blacklist this package from piwheels and not get in the way by trying to provide wheels.
• If building on a Pi 3 works, and a duplicate wheel for armv6 also works on Pi 1 then can you just upload sdist only in future and leave it to piwheels to build Arm wheels?

I will test and report 👍

[bennuttall]

I have successfully built an Arm wheel on a Pi 3, and tested installing it on another Pi 3. It seems to work but I'm not sure how to test it.

I have also duplicated it as armv6l.

You can test:

pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv7l.whl  # pi 2/3
pip install http://bennuttall.com/files/grpcio-1.11.0-cp35-cp35m-linux_armv6l.whl  # pi 1/zero

If those work ok, I'd suggest the maintainers don't bother uploading emulated platform wheels to PyPI, and let piwheels build them natively. Any reason not to?

[mehrdada]

@bennuttall I am not 100% sure who asked us (@kpayson64 should have more context), but I think we have had people pinging us a couple times we forgot to upload the ARM wheels simultaneously with the rest of our wheels. I suspect the reason might be not all ARM users are on Raspbian or Raspberry Pi? Frankly, we are not in love with building on ARM either, and we would love to defer to other people if we could, but I strongly suspect that's not an option. @proppy, do you think that is a viable solution for everyone?

[bennuttall]

Is this the right place for this issue? Should it be raised with grpc/grpc perhaps? Is gprc a google project or not?

Frankly, we are not in love with building on ARM either, and we would love to defer to other people if we could, but I strongly suspect that's not an option. @proppy, do you think that is a viable solution for everyone?

Yeah it's a complex beast. piwheels solves one problem for one set of users really well, i.e. raspbian / raspberry pi users. We don't claim or pretend to be trying to solve a wider Arm issue. But maintainers providing Arm wheels upstream (i.e. PyPI) must ensure they are compatible with other Arm boards. We (piwheels) do not recommend users of other Arm boards use piwheels as their repo! (they can replicate the effort if they want!)

If piwheels is breaking something for Pi 1/Zero users then I'll remove it, but moving forward if the authors can agree on a coherent solution, that would be great 👍

[mehrdada]

@bennuttall gRPC was started by Google and donated to and currently owned by CNCF. I don't think there's much to be discussed from the gRPC side. I just brought it up because you suggested completely outsourcing the ARM wheel production to pywheels, and I wanted to get a quick assessment on its fundamental feasibility from the community's perspective, as people on this thread are probably more familiar with the community needs around the ARM distribution than most, but beyond that there's not much we can do from gRPC's side. What I think makes sense regardless of the upstream is for pywheels to either serve a correct arm6 build or not serve one at all so users end up fetching and building from source.

[Dimeiza]

Sorry, I can't test it because I'm so busy.
As I wrote above, I have already found out a solution for me.

I think Google Assistant team has a responsibility to test a solution and fix this problem.

There are many people that failed to install Google Assistant in own Pi Zero even if they followed official install procedure. It isn't user's problem and it is a problem of official guide, isn't it?

As a result, it seems that these people have had misunderstanding about "Google Assistant is not supported Pi Zero." I would appreciate if you write clearly support status of Pi Zero in official document to avoid misunderstanding of these users.

And as outside opinion from user-side third person, I think it is a good way that build from source if you can't cope with ARM complex environment properly. It may requires a lot of time, but it is better than halt caused by illegal binary.

I have already reported all information I have. Please handle this issue as you wish.

Thank you for your support.

[proppy]

@bennuttall is there a way the piwheels infra can always rebuild arm6 wheels, even if there is one one PyPi? That seems to be the root cause of the issue (I can imagine it can happens for other wheels than the grpc one).

[bennuttall]

I've replaced our wheels with the ones I built last night, so Pi 2/3 users will still get the armv7 wheel from PyPI, but Pi 1/Zero users will get the armv6 one from piwheels (whereas without piwheels they'd get sdist and would have to build - it took about 20 mins to build on a Pi 3, which could be more like 2 hours on a Pi 1).

[mehrdada]

I just discussed the matter with @kpayson64 who was trying to build the ARMv6 builds with QEMU before and he told me the generated artifacts for ARM6 from QEMU were broken and that's why we never published ARM6 ones on PyPI.

@bennuttall Can you modify the pywheels process such that it builds from source instead of renaming the ARMv7 version? I can imagine this might cause problems not just for gRPC but also other packages.