Odd (literally) issue with nested dictionaries from weather.gov API

[mattdm]

I'm trying to fetch and parse https://api.weather.gov/gridpoints/BOX/71,90/forecast/hourly with a MatrixPortal, which does not have a lot of spare RAM. Things are basically working, but when I tried to add probability of precipitation to the data I'm fetching, I got a surprise — it's skipping every other list item.

The json in question is a dictionary with the data I want under the key properties. That key's value is another dictionary, which contains the key periods, which is a list of more dictionaries. Parsing all this works fine as long as I'm simply reading key/value pairs in the right order:

hourly_file = io.open("hourly.json",'rb')
json_data = adafruit_json_stream.load(hourly_file)
periods = json_data['properties']['periods']

for period in periods:
    print(f"Number:   {period['number']:03}")
    print(f"Start:    {period['startTime']}")
    print(f"End:      {period['endTime']}")
    print(f"Temp:     {period['temperature']} {period['temperatureUnit']}")
    print(f"Forecast: {period['shortForecast']}")

For testing, I'm using the system python on Fedora Linux, with hourly.json pre-downloaded. But this is exactly the same problem I'm seeing on the MatrixPortal with CircuitPython 9.1. What problem? Well, each period looks something like this:

{
  "number": 1,
  "name": "",
  "startTime": "2024-05-30T12:00:00-04:00",
  "endTime": "2024-05-30T13:00:00-04:00",
  "isDaytime": true,
  "temperature": 57,
  "temperatureUnit": "F",
  "temperatureTrend": null,
  "probabilityOfPrecipitation": {
    "unitCode": "wmoUnit:percent",
    "value": 80
  },
  "dewpoint": {
    "unitCode": "wmoUnit:degC",
    "value": 10
  },
  "relativeHumidity": {
    "unitCode": "wmoUnit:percent",
    "value": 77
  },
  "windSpeed": "9 mph",
  "windDirection": "N",
  "icon": "https://api.weather.gov/icons/land/day/rain_showers,80?size=small",
  "shortForecast": "Rain Showers",
  "detailedForecast": ""
}

and if try to get at one of the further-nested values, that's when stuff gets weird. For example:

for period in periods:
    print(f"Number:   {period['number']:03}")
    print(f"Start:    {period['startTime']}")
    print(f"End:      {period['endTime']}")
    print(f"Temp:     {period['temperature']} {period['temperatureUnit']}")
    print(f"Rain%:    {period['probabilityOfPrecipitation']['value']}")
    print(f"Forecast: {period['shortForecast']}")

... skips every other period, printing (in this example) just the odd-numbered ones.

Or, if I remove any lookups for keys after the nested item, like:

for period in periods:
    print(f"Number:   {period['number']:03}")
    print(f"Start:    {period['startTime']}")
    print(f"End:      {period['endTime']}")
    print(f"Temp:     {period['temperature']} {period['temperatureUnit']}")
    print(f"Rain%:    {period['probabilityOfPrecipitation']['value']}")
    #print(f"Forecast: {period['shortForecast']}")

... it stops after the first period.

Is there a better way to do this?

Is there a way to turn the Transient "period" into a real dictionary on each step of the loop? That'll use more memory, but only temporarily. Or, for that matter, just period['probabilityOfPrecipitation']?

Or should I be doing something else altogether?

[mattdm]

FWIW, this works with https://github.com/daggaz/json-stream (on Fedora Linux). I suppose I could try to bring that into CircuitPython rather than using this, but that sends me on a yak-shaving quest I'd like to avoid for my lunchtime project. :)

[tannewt]

Is there a better way to do this?

I don't think so. I'd expect it to work. This looks like a bug.

Is there a way to turn the Transient "period" into a real dictionary on each step of the loop? That'll use more memory, but only temporarily. Or, for that matter, just period['probabilityOfPrecipitation']?

I don't think so. We'd need to add an item iterator to TransientObject I think.

Or should I be doing something else altogether?

I think you are on the right track. You just hit a bug in the implementation.

[mattdm]

I think you are on the right track. You just hit a bug in the implementation.

Thanks. I took a stab at addressing it, but ended up going in circles.

In the meantime, this horrible thing works when I'm pretty sure it shouldn't:

period = next(periods)
while True:
    try:
        print(f"Rain%:    {period['probabilityOfPrecipitation']['value']}")
    except KeyError:
        break

That is, despite only nominally loading the first period from the list, it then proceeds to continue through each when searching for the key probabilityOfPrecipitation. (The KeyError I'm catching only happens when the whole file is exhausted.)

I'm going to put this horrible hack into place in my project now, and later perhaps owe a $BEVERAGE for fixing this. :)

[justmobilize]

@mattdm we just merged in #7 which does 2 things:

1. fixes an issue reading the last record in the object (could be a dict or list)
2. added the ability to get an entire dict/list with .as_object()