Getting a request.py error in the code for the "MagTag Lists From Google Spreadsheets" tutorial

[wavesailor]

I'm getting an error running the following code: https://github.com/adafruit/Adafruit_Learning_System_Guides/blob/main/MagTag_Google_Sheets/weekly_planner/code.py

Connecting to AP MyWiFI
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Traceback (most recent call last):
  File "code.py", line 101, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

I use wget on my Pi to retrieve my URL just fine, but it does not work on the MagTag

I'm using the latest libraries and Circuit Python for MagTag

adafruit-circuitpython-adafruit_magtag_2.9_grayscale-en_US-7.2.3.uf2
adafruit-circuitpython-bundle-7.x-mpy-20220321.zip

[wavesailor]

I tested the code with a dummy URL: https://api.publicapis.org/entries
and I don't get an error.

but using the google spreadsheets URL (https://docs.google.com/spreadsheets/d/e/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/pub?output=tsv)
I get the error.

Could it be how the data is being returned?

[anecdata]

Google may have changed up the URLs and something may be getting lost in a redirect. Can you test with curl, something like:

curl -iL -A "Adafruit CircuitPython" --http1.1 "https://docs.google.com/spreadsheets/d/e/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/pub?output=tsv"

and see what the sequence of URLs is.

edit: found the uRL, and it is getting a 307 Temporary Redirect. Requests should handle routine redirects, so I'm not sure if or why this one would be an issue.

[anecdata]

Looks to me like Requests will do redirects within the same host:

Adafruit_CircuitPython_Requests/adafruit_requests.py

Line 737 in 2705656

if "location" in resp.headers and 300 <= resp.status_code <= 399:

but Google does the 307 to a new subdomain & domain.

[wavesailor]

I kinda follow what you saying .... my initial request is redirected (Temporary redirect 307) to another host but still Google.

But does this mean requests.py needs to be corrected/fixed?

[anecdata]

I believe adafruit_requests would need a change, since the host changes on redirect. You might be able to put the final https://doc-14-2g-sheets.googleusercontent.com/... URL into the CircuitPython code for it to work. It's a temporary redirect, so it could change again in the future.

[wavesailor]

Okay thanks. Do I have to log a request somewhere to get it fixed?

PS. I did try use the redirected URL in but it expires are a certain period :-(

[anecdata]

Yeah, I couldn't get it to work in curl either. A bit unfriendly to require a heavy redirect to access an asset.

This issue serves as the request to get the issue addressed, but it is dependent on resources and priorities, or someone motivated enough to fix it sooner ;-). I'm going to also file a new issue in the Guide to keep track there... maybe there's an alternate way to serve up the spreadsheet.

[anecdata]

@wavesailor It might work to get the response headers and parse out the new temporary Location URL and then turn around and fetch that.

[wavesailor]

The results I get are definitely a bit strange. I changed the code to retry every minute. This happens if it is successful or if there is a runtime error.

It would not work for a few tries, then it would miraculously work, then it would fail again and finally it would crash circuit python.

This is the output I got:

Connecting to AP MyWifi
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...OK
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...OK
Updating time
Getting time for timezone America/New_York
Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Retrying in 1 min -  Sending request failed
Updating time
Getting time for timezone America/New_York
Updating tasks
Retrieving data...Traceback (most recent call last):
  File "code.py", line 105, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

[wavesailor]

Now I cannot even get it to run at all :-(

I only get this:

Retrieving data...Traceback (most recent call last):
  File "code.py", line 105, in <module>
  File "adafruit_portalbase/network.py", line 478, in fetch
  File "adafruit_requests.py", line 769, in get
  File "adafruit_requests.py", line 758, in request
  File "adafruit_requests.py", line 702, in request
  File "adafruit_requests.py", line 378, in close
ValueError: invalid syntax for integer with base 16

Code done running.

So I cannot even try get the redirected URL in because it is failing in the close function of the adafruit_requests.py

[wavesailor]

Besides the Temporary redirect 307 of the URL, I feel some error or value is not being checked in the close function of the response class in adafruit_requests.py

[dhalbert]

The invalid syntax for integer with base 16 is because it thinks it's reading a chunk header with a hex size and trying to parse it as a hex number. So probably it's reading something else that does not look like a number. You could print chunk_header in close() and see what it's trying to parse.