IoT: connection drop after around 24 hours

[kilabyte]

Our raspi connection to google iot via mqtt fails to persist past the 24hour mark. I have the JWT refreshing about every 20 mins or so.

Running on a raspi 3 with mqtt and code is identical from the examples posted here just changed my device id and project id and other vars to match my instance.

[gguuss]

Thanks for reporting this issue, I'll see if I can figure out what's going on.

[kilabyte]

If i use the pubsub client library as also on your documentation this works flawlessly but then i get no information within the IoT console for things like last seen, heartbeat, config sent etc

[gguuss]

There are a lot of potential causes for a disconnect. If you go to the console and look up your device are you seeing any error details?

[kilabyte]

I get either "The connection was closed or broken by the client" or i get "The authorization token expired" Which either doesnt make sense. The raspi is also hardwired to my router and im refreshing the token every 20 minutes.

[kilabyte]

Or if i use the client library instead is there anyway to specify the device id so i can maintain status of each iot device?

[gguuss]

I'm able to reproduce the console error issue now. There's a chance the device is not correctly refreshing the JWT, but it looks like to me like it should work as the JWT is refreshed in the call to get_client.

There's a chance on the reconnect we need to call:

client.disconnect()
client.loop()

To cleanly disconnect after this line to avoid the error. But I believe I have tested this before and it worked, despite the console errors. If you're doing a lot of calculations between the calls to mqtt.loop() the client loses its connection and behavior after that could get inconsistent.

[kilabyte]

ok i can update and try that. what is the difference between client.loop(), client.stop_loop() and client.loop_start() ?

[gguuss]

client.loop is synchronous, client.loop_stop / client.loop_start are asynchronous helpers that create a background thread. The asynchronous methods run smoother in practice but can be difficult to test.

Don't mix client.loop() with client.loop_stop() and client.loop_start().

[gguuss]

Of note, adding the disconnect seems to be the right thing to do, I'm working on a PR to set it up.

[kilabyte]

Also can evey ioT device use the same public key? Or should each device have a unique one? and does client.loop() can be called without calling client.loop_stop()?

[kilabyte]

To cleanly disconnect after this line to avoid the error.

Ok so i added the disconnect here and i get this just repeating:
on_disconnect 1: Out of memory. Connection: Connection Accepted. on_disconnect 1: Out of memory. Connection: Connection Accepted.

Im using client.loop_start() every time the loop is entered as i get better performance with the async call. Should i be calling client.loop_stop() somewhere?

If i just use client.loop() i see performance issues but it looks like the token refresh works but now my 30 second loop takes 60 seconds

[kilabyte]

OK i think i figured it out. I kept with the async loop calls. I put a client.loop_start() within my getClient method right before the return. Then when the JWT refresh is called i call client.loop_stop(). Once the connection is recreated it starts the loop again since i have it abstracted into a connect method. Im going to run this for a bit and see what i get

[gguuss]

Sounds good, @kilabyte - I have created a new branch where we cleanly disconnect the MQTT sample. Feel free to reopen the issue if you're still seeing the client disconnecting or crashing.

[kilabyte]

@gguuss can you just explain why you used .loop() and .loop_stop() in the same file? and i think for performance your PR would be better off with using the async calls instead that way you dont have to implicitly disconnect as i have found this has issues with the backoff in the disconnect callback

[gguuss]

@kilabyte Which file are you referring to? This should be a bug if it's going on. In my anecdotal tests, there is no harm in calling loop in the mix with loop_start() and loop_stop() but if you call loop_start without ever calling loop_stop later on, the program will hang.

If you're referring to the removed lines from manager_test.py those were a correction of the issue I'm referring to. In the cloud_iot_mqtt.py example I'm using the synchronous / explicit methods because they are easier to debug. To replace them with the loop_start / loop_stop methods would be the first thing I would do in real-world code.

[kilabyte]

Gotcha! Thanks for the explanation :) much appreciated. For handling like network offline and disconnects that is where the backoff comes in right? And then any offline message that was "published" is stored locally until the network is back in which case the messages will all dump out? (sorry i know this is off topic to the issue)

[gguuss]

@kilabyte Yes, you got it. The back off is to ensure devices behave well under a number of circumstances. The sample/demo/example code itself does not buffer messages or do anything intelligent in terms of online / offline. There are other solutions built on top of Cloud IoT Core that are more robust.

[kilabyte]

do you have links to said examples?

[gguuss]

A few come to mind:

• Mongoose OS
• FogLamp

There's also the device SDK written in C.

[kilabyte]

thank you!