Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

M1 EXPORTED/IMPORTED KWH resetting to zero periodically #60

Closed
ministryofsillywalks opened this issue Oct 15, 2021 · 35 comments
Closed

Comments

@ministryofsillywalks
Copy link

ministryofsillywalks commented Oct 15, 2021

I have an issue where my sensor.solaredge_m1_exported_kwh & sensor.solaredge_m1_imported_kwh are occasionally resetting "0"
image
image

This is somehow happening at pretty similar times:
20:22
06:22
20:15
06:15

It happens to both sensors however not to any other sensors from the modbus integration.
I have no idea whats going on. HA doesn't reboot at these times and also all other network activity seems normal.
These "0" values of course royally screw up my energy dashboard
image

Any idea whats going on?

@ministryofsillywalks ministryofsillywalks changed the title M1 EXPORTED/IMPORTED KWH resetting to zero preiodically M1 EXPORTED/IMPORTED KWH resetting to zero periodically Oct 15, 2021
@mpredfearn
Copy link
Contributor

@ministryofsillywalks yeah sometimes we get bad data back from the inverter. #44 should fix it once upstream.

@nerfherder
Copy link
Contributor

This should now be fixed

@ChristophCaina
Copy link
Contributor

Hi, I've just noticed the same behave today - but not periodically...
Is there some way to detect, why the sensor was at zero?

grafik
grafik
grafik

@ChristophCaina
Copy link
Contributor

ok, I have recalculated the stats in the Database and changed the SUM... for some time, the Dashboard showed the correct data... but then, suddenly, the issue came back, without another "dip" in the sensors log / history...

grafik

Import Sensor
grafik

Export Sensor
grafik

Stats from 07:00 to 09:30
grafik

the previous dip to 0 [the same than in the comment above]
grafik

How can I fix this in the table - and why did it come back, after recalculating the sum in the DB ?

@nerfherder
Copy link
Contributor

A new release hasn't gone out yet but the code to fix the 0 values has been merged into master. Just double checking if you have got the latest code?

I'm not sure about the stats issues you are facing, I've been looking into this as well as I got some bad battery charged stats in my history which is impacting my long term stats as well. I believe you can delete the entity from the database and it will rebuild when new stars are added.

@ChristophCaina
Copy link
Contributor

hm... I had some issues whenever I deleted somehting from the stats table...
I did this a few times, when I had bad readings from the default SolarEdge integration (API call)... and whenever I deleted something, the complete DB was rebuilt with empty stats.

The strange thing here is, that whenever I had this in the past, recalculating the SUM did work - and the graph was then OK (until the next bad reading / connection loss, or whatever)

Only this time, after recalculating the sum, the issue came back with the next calculation.
Not sure, if the integration does save the last Sum somewhere... or if this might be related to a change in the core?

@nerfherder
Copy link
Contributor

I don't believe that issue would be coming from this integration from my understanding of the code. There might have been a change in the core.

@gmacioso
Copy link

@ChristophCaina
Appreciate, if you find solution how to correct the stats in DB, to share :)
I stucked with same issue...

@ChristophCaina
Copy link
Contributor

only recalculating and editing manually... :-(
Unfortunately, this could cause some other side effects...

I did this right now - trying to fix the values, but since my solar installation did start to produce a small amount of energy, the dashboard graph is now showing some values in the wrong "direction" ...
:-(

grafik

honestly speaking:
I haven't seen such a "delicate" thing for a while now... within the last month, I had so many issues in this direction that I really considering to disable the energy dashboard... I can't see any benefit until these kind of issues are fixed... (either in the many integrations that could be related - or the core)

@nerfherder
Copy link
Contributor

nerfherder commented Oct 18, 2021

What are you editing the db with out of interest?

@ChristophCaina
Copy link
Contributor

For editing the DB, I am using a Supervisor Addon called: SQLite Web
It is a simple SQLite Viewer / Editor...

after the recent change from above, I did refresh the site... now, it seems that the statistics have been summed up again:
grafik

  • frustrated *

@mpredfearn
Copy link
Contributor

mpredfearn commented Oct 18, 2021

honestly speaking:
I haven't seen such a "delicate" thing for a while now... within the last month, I had so many issues in this direction that I really > considering to disable the energy dashboard... I can't see any benefit until these kind of issues are fixed... (either in the many integrations that could be related - or the core)

Agreed - the energy dashboard is definitely pretty fragile at the moment - but it's still a young feature so some growing pains are kinda to be expected. These issues need to be reported against home assistant core though. I believe we've now solved the "zero problem" with the energy stats from this integration - zero is not tolerated as a valid value for either the inverter total generation nor the import / export meters. So as long as you have the latest version of this integration, any problems observed must be coming from the core.

@ChristophCaina
Copy link
Contributor

So as long as you have the latest version of this integration, any problems observed must be coming from the core.

Currently, I am running Version 1.5.0 of this integration.
I guess, the fix will be in one of the next releases, right?

@WillCodeForCats
Copy link

Out of curiosity, what's your integration polling interval and how are you connecting to the inverter?

@ChristophCaina
Copy link
Contributor

Out of curiosity, what's your integration polling interval and how are you connecting to the inverter?

?
default settings of the integration... connected via IP with the inverter...
The inverter is set to "SolarEdge Device / Master" as far as I remember (the settings are in german - and I haven't confirmed the exact wording)...

Polling Interval is just as the default setting of the app... don't remind the frequency.

@WillCodeForCats
Copy link

It looks like the default is 30 seconds. Can I suggest try changing that to 60 seconds?

@ChristophCaina
Copy link
Contributor

hm... should that change anything?
While checking the statistics, I could only see the behave two times (last friday in the evening and today in the morning).
I would expect, that 30seconds in my home network should be quite ok... but - yes, I can change the settings to 60 seconds.

@WillCodeForCats
Copy link

So the reason I suggest 60 seconds is because that's what I use and have not seen any of these problems with meters or energy dashboard (although I am using my fork, because I have multiple inverters this fork does not yet support, but my fork is also still back on the previous modbus library). My M1 kWh values have never reset or returned zero.

It makes me wonder if some of the data problems are from polling too fast. I'm not sure how robust the inverter modbus/tcp is for frequent polling, after all it only supports one connection at a time.

@WillCodeForCats
Copy link

Actually the modbus library might still be the same in my fork, for some reason I thought that change was merged here but it doesn't look like it was yet. So the major difference then is polling interval and all the extra stuff for control support (I don't have batteries).

@TheLordVader
Copy link

Suffering exact same issue starting today, it has never occurred before. Might be core related? Polling interval 5 seconds, working very well before, Ethernet cable and local IP. Messed my energy dashboard completely, did a full restore of backup and deleted Maria dB. Tonight or does to zero again. A half days work and problem is back.
Looking forward to you guys investigating, it is out of my knowledge I'm afraid.
Screenshot_2021-10-18-20-41-58-072_io homeassistant companion android

@binsentsu
Copy link
Owner

#44 Is now released in version 1.5.1. With this fix, the zero flats should be skipped.

@ChristophCaina
Copy link
Contributor

hm... can you please check the exact time-frame when this issue happened?
I've compared the two issues I've found... and in both cases, it seems, that the dip happened for one "cycle" - means from one poll to the next one.

nearly 30 secs, which is the default setting in my case:
interestingly, in both cases around xx:xx:15 - xx:xx:45

grafik
grafik

and - sure. both sensors have been affected.
What I've also found out, only the sensor values for the meter (most of them) show this drop (or climb) to 0.0
The other sensors from the inverter itself are normal.
This means, it is in fact an issue caused by the meter - and not the connection / poll against the inverter.

@WillCodeForCats
Copy link

If you're willing to try a 60 second or longer interval before you try the patched version, I'm still curious if it's a polling frequency related thing or something else. I'd try it myself, but since I haven't had this issue I don't want to risk getting zeros in my data.

@ChristophCaina
Copy link
Contributor

#44 Is now released in version 1.5.1. With this fix, the zero flats should be skipped.

Thanks :)

If you're willing to try a 60 second or longer interval before you try the patched version, I'm still curious if it's a polling frequency related thing or something else. I'd try it myself, but since I haven't had this issue I don't want to risk getting zeros in my data.

I would prefer the Update.
If it is related to the polling frequency I would consider that this ist should also be visible for the Sensors of the Inverter - which was mit the Case for me

@WillCodeForCats
Copy link

Ok. I won't do any further digging since hiding the problem is the preferred solution.

@nerfherder
Copy link
Contributor

@WillCodeForCats I have faced issues with higher frequency polling (5 seconds) but I run mine on wifi a decent distance from my router. I have found that #39 solves this issue as the open/close of connections are handled differently.

However this is a different issue to #60 so maybe best discussed in another issue like #37

@WillCodeForCats
Copy link

Rapid polling ideally should hold the modbus/tcp connection open, but I don't know if HA is even capable of that. The inverter only supports a single TCP connection at a time, which gives me gut feeling that connection setup/teardown too often is asking for trouble. Getting random zeros and just masking it, well, I don't like it because the problem still exists. I'd rather have accurate data than masked bad data, but that's just me and I guess the typical HA user doesn't care about that level of accuracy.

The inverter can certainly handle real time polling i.e. make a simple client with libmodbus, loop and sleep between reads, but the connection is always held open so there's no setup/teardown overhead between read cycles.

@gmacioso
Copy link

gmacioso commented Oct 19, 2021

Just for clarification:

This are not 0 (zeroes), this are nulls (no value)

So it could be either meter either inverter issue with rs485 communication.
It happened to me while I had 15s, 30s as well as 60s polling.
And, what's more important...

It's for sure not an integration problem, as I run for a year on Domoticz with my own integration. 4 month using Perl script, then switched to Python.
I had same issues there.
Interesting is, that Domoticz was handling those nulls, I found an issue while I started to store data in Influx (same scripts were pushing data to both API Domoticz & Influx)

So the only thing we can do is to handle this fault by ommiting 0 and null while it comes from SE

@ChristophCaina
Copy link
Contributor

Just for clarification:

This are not 0 (zeroes), this are nulls (no value)
So the only thing we can do is to handle this fault by ommiting 0 and null while it comes from SE

Hi,
Thanks for your reply :)
In fact, based on the History Graph, the sensor got zeros.
When it become NULL or invalid, you should see a small gap in the chart, but not a dip to a zero-value, like in the screenshots above.

And NULLs could be from an connection / polling issue... like described in #50

@gmacioso
Copy link

So...
When I looked directly into db I found nulls
You can see it too on pics in my #50 on data comming from modbus registes directly (Wh counter from meter)
But if you look next to integrated/recalculated kWh then in shows 0 on history graph
PLS, take a look at those pics from #50

@ChristophCaina
Copy link
Contributor

@gmacioso:

This case is another one.
The Gap in the chart happens, when the sensor became "unknown"...
grafik

Here's the corresponding entry in the DB (please note, I am in UTC+02, therefore, there is a difference in the Time of the UI and the DB)
grafik

But in the case reported below, the sensor got a zero value, not a NULL...
This screenshot does show both times when I can see the dip to ZERO value in my UI...
grafik

The question now is:
The sensor becomes UNKNOWN - for example, when HA is restarting.
When the connection gets lost - does it receive a NULL - and this will be shown as "0.0" ?
In this case, I would suggest, that a NULL should also be set to unknown by default - because unknown will be ignored and does not cause any issues with the energy dashboard...

@gmacioso
Copy link

@ChristophCaina

OK, I cannot verify now, as my HA recorder is set to 2 weeks storage (long term is in Influx)
For sure in my case there was empty value/state field - nothing, just empty

@nerfherder
Copy link
Contributor

FYI, I have cleared historical data by deleting the entry from the states, statistics_meta, statistics_short_term and statistics tables. The trick was to lookup the meta_id in statistics_meta and then delete relevant entries from statistics_short_term and statistics.

I just deleted everything so it could start again fresh, but I suppose you could try and manually remove the incorrect figures and update the sums etc.

@ChristophCaina
Copy link
Contributor

Hi @nerfherder, in the past, updating the sum has worked well for me (while I've used the solar edge api integration)... but when I tried it this time - the sum was recalculated with the wrong value on the next statistic calculation.

I am still not sure, if this is related to a change in one of the recent core updates or not... haven't got any further reply or comment on my raised issue on the core repository.

But I've updated this integration now and will observe the issue :)

@ChristophCaina
Copy link
Contributor

can be closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants