Unmatched entities in MQTT payload

[glitter-ball]

Bug description
EMS-ESP32 regularly sending the same few points with no matching data.
2021-11-13 23:00:05 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'tempautotemp' when rendering '{{value_json.hc1.tempautotemp}}'

At one level, this is trivial! But...

• It's generating megabytes of log file every day - in 20 hours, my HA instance has logged this error 17,000 times!
• It's hammering the SSD on HA
• It's usually the same few points, so maybe there's a bug somewhere causing this.

I can delete the point with MQTT Explorer, but it will be back - presumably when I restart EMS-ESP32? If the point has no associated data, can it be stopped from sending the template? 'tempautotemp' is far and away the biggest culprit! Is there something special about the definition of this point?

Steps to reproduce
Run EMS-ESP32 and HA : HA log file fills up with this error.

Expected behavior
No HA errors!

Screenshots

Device information```

EMS-ESP version 3.3.0b2

Boiler: Worcester Logamax Plus/GB192/Condens GC9000 (DeviceID:0x08 ProductID:208, Version:01.01)
heating active: off
warm water active: off
selected flow temperature: 28 °C
burner selected max power: 100 %
heating pump modulation: 96 %
outside temperature: 11.3 °C
current flow temperature: 26.5 °C
gas: off
flame current: 0.0 uA
heating pump: on
fan: off
ignition: off
heating activated: on
heating temperature: 75 °C
burner pump max power: 100 %
burner pump min power: 60 %
pump delay: 3
burner min period: 5
burner min power: 0 %
burner max power: 100 %
hysteresis on temperature: -10 °C
hysteresis off temperature: 6 °C
burner current power: 0 %
burner starts: 19281 times
total burner operating time: 173 days 6 hours 51 minutes
total heat operating time: 132 days 16 hours 41 minutes
total UBA operating time: 1200 days 18 hours 11 minutes
last error code: EA(227) 09.10.2021 10:27
service code number: 204
maintenance message:
maintenance scheduled: off
maintenance set time: 6000 hours
maintenance set date: 01.01.2014
ww selected temperature: 60
ww set temperature: 60
ww type: buffer
ww comfort: hot
ww flow temperature offset: 40
ww max power: 100
ww circulation pump available: off
ww charging type: 3-way valve
ww hysteresis on temperature: -5
ww hysteresis off temperature: 0
ww disinfection temperature: 70
ww circulation pump frequency: off
ww circulation active: off
ww current intern temperature: 55.2
ww current extern temperature: 55.2
ww current tap water flow: 0.0
ww storage extern temperature: 55.2
ww activated: on
ww one time charging: off
ww disinfection: off
ww charging: off
ww recharging: off
ww temperature ok: on
ww active: off
ww heating: off
ww starts: 5233
ww active time: 40 days 14 hours 10 minutes

Thermostat: RC300/RC310/Moduline 3000/1010H/CW400/Sense II (DeviceID:0x10, ProductID:158, Version:33.03)
error code: (0)
date/time: 18:52:15 14/11/2021
floor drying: off
damped outdoor temperature: 11.8 °C
floor drying temperature: 0 °C
building: medium
minimal external temperature: -5 °C
ww set temperature: 60
ww mode: own_prog
ww set low temperature: 45
ww circulation pump frequency: own_prog
ww charge duration: 60
ww charge: off
ww circuit 1 extra: 0
ww disinfection: off
ww disinfection day: tu
ww disinfection time: 120
hc1 selected room temperature: 18.5
hc1 current room temperature: 18.7
hc1 mode: auto
hc1 mode type: comfort
hc1 eco temperature: 15.0
hc1 manual temperature: 18.0
hc1 comfort temperature: 21.0
hc1 summer temperature: 17
hc1 design temperature: 45
hc1 offset temperature: 0
hc1 min flow temperature: 25
hc1 max flow temperature: 82
hc1 room influence: 9
hc1 current room influence: -1.7
hc1 nofrost temperature: 5
hc1 target flow temperature: 28
hc1 heating type: radiator
hc1 set summer mode: auto
hc1 summer mode: off
hc1 control mode: optimized
hc1 program: prog_1
hc1 fast heatup: 0

Controller: ErP (DeviceID:0x09, ProductID:209, Version:01.03)

Dallas temperature sensors:
Sensor 1, ID: 28-BE2A-7791-0402, Temperature: 26.2 °C (offset 0.0)

[proddy]

I know it's annoying, it happens when there is a homeassistant/sensor/ems-esp/<name>/config topic in the MQTT broker (which is retained with the retain flag set) but the actual data in the ems-esp/thermostat-data topic doesn't yet exist. It can usually happen during the first few seconds when EMS-ESP is booted. But then the warnings should stop, unless somehow that old topic is still lingering about. So I'm puzzled...

if you fire up MQTTExplorer and in the Search bar at the top type in tempautotemp what do you see? Hopefully 3 topics...?

[proddy]

and if you're worried about the size of the log file (not a solution I know but a temporary workaround) disable the logging in HA with

logger:
  default: critical
  logs:
    homeassistant.components.mqtt: error

https://www.home-assistant.io/integrations/logger/

[glitter-ball]

I get a little flurry at start-up as things sort themselves out and templates get populated with all the necessary variables. But, this one just keeps on coming! I only get the one topic...

There's something weird/different about this one point. Loads of disk space, so log file not a catastrophic issue, but I'd rather HA wasn't wasting time writing out 20k+ errors every day *8)

[proddy]

that's odd. EMS-ESP wouldn't create that homeassistant topic if there wasn't a value. Are you sure it sneaks back in after you delete it in MQTTExplorer and restart EMS-ESP?

[glitter-ball]

Now the fun. I deleted the 'tempautotemp' topic above, so no entries in MQTT Explorer. I then went to my thermostat and changes the temperature by spinning the wheel - nothing else. So now we have a TEMPorary AUTO TEMPerature set.

MQTT Explorer now shows...

So now I have matched data and no HA errors. I suspect things get interesting when the temporary temperature period ends. At that point, tempautotemp probably has no definition but the template is still being sent...

[glitter-ball]

I realised I turned the temperature down last night when I was the last to bed, relatively early! The errors started started at 23:00:05 when the thermostat would have gone from my temporary temperature back to the time programme. At that point, there was no temporary info to send any more.

I've just tried resetting the temp and reverting to AUTO but this hasn't replicated the problem, so it may depend on reaching the next programme time/temperature change.

[glitter-ball]

So... the logic is there to define the point when the stat begins to publish it, but when it stops publishing it (and the point disappears from the payload), the definition payload remains. Weird. Nasty.

[proddy]

yes, that's it! If there is no value for tempautotemp then it will not be included in the thermostat_data payload, and HA will throw the warning. I need to consult with Michael as this entity is something he recently added so I'm not that confident I fully understand how it works.

I think the solution is either

• we add a new attribute type to a device value (like tempautotemp) identifying it as something that is volatile, i.e. it can come and go. Then we remember the state, and if it has no value (i.e. it goes) we delete the HA config topic
• or, we just use a dummy value for these volatile types so its gets included in the _data payloads, and tell HA to ignore them somehow.

[glitter-ball]

Great - glad we at least understood what's going on pretty quick. I think there might be other points doing other weird stuff like this. I'll try and collect evidence this week. One was something to do with ww but I'll wait and check.

I guess the first option above is technically better, but I don't know about the coding practicalities.

[glitter-ball]

Shame the thermostat doesn't handle it better but I suppose there's no point in it sending data that doesn't exist any more!

[glitter-ball]

Yep, the errors are back and only the definition payload is being sent. The data payload disappears when...

• The next programmed time is reached
• I flick the stat from Auto to Manual back to Auto

[glitter-ball]

Going back in the logs, one of the changes I made also caused a bigger burp, albeit short-lived. Thermostat is obviously having a good cough then...

2021-11-14 19:56:53 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'tempautotemp' when rendering '{{value_json.hc1.tempautotemp}}'
2021-11-14 19:56:53 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'tempautotemp' when rendering '{{value_json.hc1.tempautotemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{ value_json.hc1.mode }}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.summermode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.hc1.summermode}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwcharge}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.wwcharge}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwdisinfect}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.wwdisinfect}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.mode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.modetype}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.ecotemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.manualtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.comforttemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.summertemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.id}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.errorcode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.building}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.floordry}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.floordrytemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.seltemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.designtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.offsettemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.minflowtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.maxflowtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.roominfluence}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.nofrosttemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.targetflowtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.controlmode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.program}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.dampedoutdoortemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.currtemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.heatingtype}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.datetime}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwmode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwcircmode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwsettemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwsettemplow}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwextra1}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.minexttemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.summersetmode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.curroominfl}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwchargeduration}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.fastheatup}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwdisinfectday}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.wwdisinfecttime}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.hatemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.hc1.hatemp}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.components.mqtt.climate] Could not parse temperature from 
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.seltemp}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.hc1.seltemp}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.components.mqtt.climate] Could not parse temperature from 
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.hamode}}'
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Error parsing value: 'value_json' is undefined (value: , template: {{value_json.hc1.hamode}})
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.components.mqtt.climate] Invalid modes mode: 
2021-11-14 19:57:03 ERROR (MainThread) [homeassistant.helpers.template] Template variable error: 'value_json' is undefined when rendering '{{value_json.hc1.tempautotemp}}'
2021-11-14 19:57:04 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'tempautotemp' when rendering '{{value_json.hc1.tempautotemp}}'
2021-11-14 19:57:04 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'tempautotemp' when rendering '{{value_json.hc1.tempautotemp}}'

[proddy]

I've implemented my option 1 (delete the HA config when the value is bogus). I'll test it for a bit and see how well it stands up