grpc streaming example crashes and hangs

[thecloudist]

Just cloned the latest repo for grpc -
running this on Raspbian (Raspberry Pi)

Cannot even break out of the program at all.
The sample rate of my usb mic is limited to 44100 and 48000. I tried changing those in the example but I get the same results.

I have pyaudio installed up to date.
Sound recording and playback both work with pyaudio.

The ~/.asoundrc script is here:

pcm.!default {
    type asym
    playback.pcm "hw:0,0"
    capture.pcm "hw:1,1"
}

# pcm.!default {
#        type hw
#        card 0
# }

ctl.!default {
        type hw
        card 0
}

#pcm.record {
#    type plug;
#    slave {
#        pcm "hw 1,1"
#    }
# }

Here is the output of the program.

(grpc) pi@raspberrypi:~/python-docs-samples/speech/grpc $ python transcribe_streaming.py
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/pi/gopigo/Projects/Awareness/voice/python-docs-samples/speech/grpc/grpc/local/lib/python2.7/site-packages/grpc/_channel.py", line 184, in consume_request_iterator
    for request in request_iterator:
  File "transcribe_streaming.py", line 112, in request_stream
    data = audio_stream.read(chunk)
  File "/home/pi/gopigo/Projects/Awareness/voice/python-docs-samples/speech/grpc/grpc/local/lib/python2.7/site-packages/pyaudio.py", line 608, in read
    return pa.read_stream(self._stream, num_frames, exception_on_overflow)
IOError: [Errno -9981] Input overflowed

[jerjou]

Would you mind posting a snippet of the code that works with pyaudio, and the output that you get?

[thecloudist]

Here is what I have working:

pyaudio_examples_steve.zip

[jerjou]

Two things:

• Can you post the output that you get when you run the code that works?
• Can you post the output of git diff on the directory where you're running transcribe_streaming.py?

[thecloudist]

will do. thanks

[thecloudist]

The output when it is successful:

pi@raspberrypi:~/gopigo/Projects/Awareness/voice/pyaudio_test $ python pyaudio_test.py
ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.rear
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.center_lfe
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.side
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.hdmi
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.modem
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pcm.c:2239:(snd_pcm_open_noupdate) Unknown PCM cards.pcm.phoneline
ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pulse.c:243:(pulse_connect) PulseAudio: Unable to connect: Connection refused

ALSA lib pcm_dmix.c:1022:(snd_pcm_dmix_open) unable to open slave
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
jack server is not running or cannot be started

• recording
• done recording
  pi@raspberrypi:~/gopigo/Projects/Awareness/voice/pyaudio_test $ aplay pyaudio-output.wav
  Playing WAVE 'pyaudio-output.wav' : Signed 16 bit Little Endian, Rate 48000 Hz, Mono

The diff is in the attachment:

grpc_diff.txt

[thecloudist]

By the way:

This works --- >

(grpc) pi@raspberrypi:/python-docs-samples/speech/grpc $ python transcribe.py gs://voice_test/audio.flac
alternatives {
transcript: "how old is the Brooklyn Bridge"
confidence: 0.982678949833
}
pi@raspberrypi:/python-docs-samples/speech/grpc $

[jerjou]

I'm not seeing the diff? I'm seeing a dump of your audio devices in the attachment..

[thecloudist]

Sorry: 1 sec --
grpc_diff.txt

[jerjou]

Okay, I can't seem to reproduce the error on my box (I don't have a raspberry pi handy at the moment).

Googling around, it looks like it happens because audio data is coming in faster than you're using it, and the portaudio buffer overflows. I'd guess that this is because, on the raspberry pi, the hardware is just a tad too slow / constrained to keep up with the sampling.

I would try playing with your chunk size (increasing it, perhaps?) and using the lowest sample rate your audio device supports (44100, from what you said). There are also a number of workarounds posted in that stackoverflow thread that you might try.

Notably, since you know what platform you're running it on (I used pyaudio in this sample mostly because it's cross-platform), one answer suggests using the alsaaudio library instead of pyaudio, which may get you the efficiency needed to keep up with the input stream.

Alternatively, you could spin up a thread to consume and store the input stream, and then have the request_stream function grab whatever's been buffered and send that to the api. You'll have to deal with locking a shared object between threads (which is always a pain), but this is probably the Right Way to do it.

Let us know what you try / what results you get!

[theacodes]

@jerjou if you'll remind me next time you come to Seattle I can bring in my RPi2.

[thecloudist]

@jerjou - I concur with your analysis. If the alsaaudio example works, that is it can keep up with the sampling, then it does suggest that this is an efficiency issue. I am also looking at what the enormously popular (but somewhat limited) Amazon Echo 'on yur pi' project does to guarantee audio can get to the service and responses come back pretty quickly on a Raspi.

I am building a voice-controlled robot car, not a virtual assistant. Although, conceptually, I could with the GCP/Voice, NLP, etc, etc which I strive to do eventually.

What I need to do first if I want to use the streaming grpc service is to get a test program working. Once there, I can slice and dice things, build a simple library which my mains can call and I know it will work each time. Right now, I cannot get past this problem.

FYI - Googlers: Kas Sata and Glen Shires @io16 built a gopigo voice-controlled car with vision based on GCP that was the inspiration for my project. If you have access to them, they may know how to get this working on a Pi.

Cheers!

[jerjou]

I just pushed a quick-and-dirty version that fills an input buffer in a separate thread. See if this works for you:

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/threaded-speech/speech/grpc/transcribe_streaming.py