Enhancement/nested telemetry messages

[oscgonfer]

A new proposal for nesting protobuf messages in sensor data structures, to add information for the sensor producing the data (not just the metric). Currently, only with AQ telemetry module.

⚠️ This is a breaking change for clients and potentially best to keep two formats in parallel for a while.

The result reads, decoded on the other end:

{ 'time': 1769781580,
  'air_quality_metrics': {
                         'sen5xdata': {
                                       'pm10_standard': 1,
                                       'pm25_standard': 2,
                                       'pm40_standard': 2,
                                       'pm100_standard': 2,
                                       'particles_05um': 1100,
                                       'particles_10um': 300,
                                       'particles_25um': 0,
                                       'particles_40um': 0,
                                       'particles_100um': 0,
                                       'pm_temperature': 17.615,
                                       'pm_humidity': 69.84,
                                       'pm_voc_idx': 100.0,
                                       'pm_nox_idx': 1.0,
                                       'particles_tps': 0.667},
                        'scd4xdata': { 'co2': 1344,
                                       'co2_temperature': 15.322728,
                                       'co2_humidity': 81.573204}
 }
}

[Tomas-Kuchta-FPV]

Hi. Nice job. I've noticed one thing.
'sen5xdata': {pm_temperature': 17.615}
'scd4xdata': {co2_temperature': 15.322728

We already know which sensor is sending the metric, so we don't need to add the co2_ prefix to the metric, and we can keep it simple like temperature.
Rather than being dependent on the prefix, it should be dependent on the sensor.

Note that I'm just a HW person, and I'm slowly getting into coding and have no clue about building this stuff.

[oscgonfer]

That's an easy fix. Let's see if the proposal moves forward. There are many other implications for this on the client side.

[Tomas-Kuchta-FPV]

Hi @oscgonfer, I’ve been thinking about a possibly more additive approach.

What if we keep the current flat Telemetry → metric structure exactly as it is, and introduce an optional, separate message (e.g. TelemetryProvenance) that provides sensor context: sensor ID / model (BME280, SHT3x, etc.) and which metrics it supplies.

This message could be sent at longer intervals (or on change / boot), so normal telemetry remains lightweight and unchanged, while clients that care about provenance can use the extra information.

This would avoid breaking existing clients and seems like it could work nicely with sensor selection logic as discussed in meshtastic/firmware#9474.

Curious what you think.

[oscgonfer]

Hi @Tomas-Kuchta-FPV

From my gut "feeling" I'd say the approach with the nested protobuf is better, as it doesn't substantially increase the size of the protobuf (since encoding/decoding happens on both ends, and we're adding a tag and a length varint -> + 2 bytes) and it has it all on the message itself. However, it's true that the approach you suggest wouldn't involve changes on clients and that it could be a good solution (although not the best case, if there's a chance to consider the nested mesaage approach). I'd say an input from the development roadmappoint of view should be considered in this discussion. Maybe @caveman99 or other maintainers/admns can chip in.

The tricky thing is the case where multiple senors provide similar metrics. In smartcitizen.me, for instance, its quite typical to have 3-4 temperature sensors in the same unit. The nested approach guarantees a safe 1-1 relationship, avoiding the wait (and potentially missing) the Telemetry Provenance packet. In our case, having this less frequent Telemetry Provenance wouldn't be ideal, because we'd need to wait for that packet to know what we're receiving, hence storage of messages, and expecting nothing changes in between Provenance packets.