Error 429 connecting to solcast on HA restart, zapped all solcast sensors

[gcoan]

Describe the bug
I restarted Home Assistant and when the Solcast integration started up it got an error 429 I believe from retrieving the rooftop sites info:

This caused the rooftop sensor in HA to change to Unknown:

And ALL the solcast PV forecast sensors to change to zero:

Fortunately I spotted this pretty quickly and was able to resolve it by reloading the Solcast integration and all the sensors populated correctly

To Reproduce
Restart HA and if Solcast is busy its likely that the same will occur

Expected behavior
On HA start, if an error 429 occurs, use the previous solcast sensor values. Do not set any sensors to Unknown or zero.

Desktop (please complete the following information):
HA 2024.5.5
HAOS 12.3
Solcast v4.0.25

[oleg-d]

i don't know if this will help you but this helped me when i had the 429 error.

wait around a day without making a call to solcast. and then refresh at a random time, not on the hour.

if you have automation to refresh data from solcast exactly on the hour then add a delay to it. for example i am using 2 min 45 secs delay on my automation that updates solcast data.

[gcoan]

wait around a day without making a call to solcast. and then refresh at a random time, not on the hour.

yes I have this in my solcast automation and I'm getting pretty reliable solcast polls now.

This problem occurred when Home Assistant was restarted (at a random time), the integration reached out to solcast to get the site data, got a 429 error and then that zero'd out all the solcast sensors. The enhancement request is to either prevent the initial solcast call, or trap the 429 error and not over
write the sensors if it occurs

[autoSteve]

The enhancement request is to either prevent the initial solcast call, or trap the 429 error

I am looking into adding a configure option to suppress the initial call to Solcast... With two PV arrays I run out of available calls real quick doing restarts.

The initial call is not easily suppressed, as from what I can tell it returns the 'sites' along with forecast data. So I'll need to cache the sites data to survive the restart, which will result in data still showing up on the Energy dashboard right after restart. Doable, but it'll take time.

[oleg-d]

One other integration I use has a toggle that you can enable/disable cloud update at integration startup and HA reboots. Obviously dont know if this could be translated over to this integration but worth mentioning.

For someone who knows about the technical side of things and python can check the PR here from the other integration to see if something can be borrowed into here.

[timverlander]

So I'll need to cache the sites data to survive the restart

Just an alternative thought.. as hobbyist accounts can only have two rooftops, it might be simpler to capture two additional user-entered resource_ids in the configuration flow and if either/both are present then skip the startup discovery call to Solcast and use the supplied resource_id(s) instead?

[autoSteve]

I did some code examining only to find that the sites data is rigged to be definitely already cached by solcastapi.py. Then I did a few restarts of HA and checked API use count at solcast.com.au. It did not go up, so getting the sites must not contribute to API calls confirming the cache read of sites data.

However earlier today I did notice a 429 from Solcast after restarting, with forecast data unavailable in HA. The solar production yellow bars were also missing. Trying to check the log later did not have a 429 present because log clear on restart, so I need to wait until it happens again to see if it's the same issue reported at line 144.

An immediate restart of HA fixed the forecast and solar display.

So in summary, this should never happen on HA start with /config/sites.json (not solcast.json) present and the class variable apiCacheEnabled set True, because that section of code in the try/except block should read the cache and not do a REST call but apiCacheEnabled is always False.

Debug logging for individual components also seems to turn off on HA restart, so I might need to enable system-wide debug logging for a while. Ugh. I got logging sorted.

[autoSteve]

Delving further:

On startup, apiCacheEnabled is always False, hard-coded, so it's not caching the sites collection. It is caching forecasts only.

But it doesn't really need to cache sites collection, because it doesn't impact the count of daily API calls to do so. So no impact, you'd think? Kind of.

What I think is happening here is that Solcast limit API calls when things get too busy, probably only when you have a hobbiest account, and there is already much said by Oziee on that by guiding folks to not gather exactly on the hour. Many folks don't listen, so if your restart occurred near the top of the hour it might fail for that reason.

But it didn't. It failed at a quarter to five.

I am wondering whether some form of retry is in order on failure. Caching is also probably a better option so that the API call never happens. But what if sites change? Get added? This is probably why Oziee left it permanently disabled, to most likely cut down on the number of issues logged against his repository. Not everyone's a Python coder... And not everyone knows how to find and delete a file in Linux.

But he wrote the code to do so. It'd be amusing to try and make it work in the real world.

So I suspect that API limiting, because busy is behind this. I just did about twenty restarts in a row and none of them failed to get the sites from API, despite me having just one weird and seemingly random failure earlier today.

[autoSteve]

@gcoan, you could try out 490494c, which adds a retry mechanism, but testing my suspicion is going to be tough.