Patchy BG data in Loopalyzer report

[bjornoleh]

Describe the bug
For some days, the BG curve in the Loopalyzer report is very patchy, although the main NS BG chart is continuous.

To Reproduce
Steps to reproduce the behavior:

1. Go to Reports/Loopalyzer
2. Click on 'Today' and "Show"

Unknown if this is related, but I might see a pattern:
The BG readings from the G6 transmittor/app arrives every five minutes at 0, 5, 10, 15 minutes etc. Around the missing BG readings in the Loopalyzer chart, I see that some of the readings are at "unexpected" times, eg
03:00 - 03:04 - 03:10 - 03:15 - 03:19 - 03:25 - 03:30

Expected behavior
I would expect a continuous BG curve whenever that is present in the main NS view.

Your setup information

• Nightscout 14.0.7 dev
• Heroku, recently migrated from mLab to Atlas
• Loop/FreeAPS (upload data = ON)
• Dexcom G6 --> Loop
• Tested in PC/Chrome

Additional context
Possibly related to this issue?

Loopalyzer report has missing Active insulin rendering for some days #6028

[sulkaharo]

@lixgbg can you take a look?

[lixgbg]

@bjornoleh Can you please go to the Day to day report and select the same day and see if you have a continuous line there or if it is patchy as in Loopalyzer?

They pull data from Nightscout in the same way (at least did when Loopalyzer was developed) so I would like to see if this is a Loopalyzer-specific issue or if it's a Nightscout reporting in general thing.

[bjornoleh]

@lixgbg Hi, it looks continuous in Day-to-day as well

[lixgbg]

Hi, ok that looks like a more continuous line. It could be, just as you say, that for some reason you sometimes get BG data which are not timestamped properly and thus misaligned and thus skipped by Loopalyzer. In Loopalyzer I tried to avoid modifying the data, or recalculate/align too much, and keep the data as it is. This in order to be easier to CE/FDA approve if that would one day be of interest.

I have not seen these gaps in the graph on our own graphs or anyone else's. Not saying it's a problem you have identified, but wonder what the root cause is. Could it be you're using FreeAPS and that it is sending data differently than Loop? It's a long shot but that could be one thing. But then why would it do that only sometimes. Can you see a pattern connected to networking, e.g. if you're on a reliable WiFi vs mobile 4G far out in the woods?

How often does this occur?
Do some days show a completely continuous line while some days are patchy like this? Any pattern there?

[bjornoleh]

This was by far the most obvious one. But smaller gaps are seen from time to time. FreeAPS has no changes in this respect, so that can't be an issue. I don't think wifi / network quality is the issue, but will pay attention to this. I can have a look at the exact timestamps in the entries collection, perhaps that would reveal something. tor. 15. okt. 2020, 07:59 skrev lixgbg <notifications@github.com>:

…

Hi, ok that looks like a more continuous line. It could be, just as you say, that for some reason you sometimes get BG data which are not timestamped properly and thus misaligned and thus skipped by Loopalyzer. In Loopalyzer I tried to avoid modifying the data, or recalculate/align too much, and keep the data as it is. This in order to be easier to CE/FDA approve if that would one day be of interest. I have not seen these gaps in the graph on our own graphs or anyone else's. Not saying it's a problem you have identified, but wonder what the root cause is. Could it be you're using FreeAPS and that it is sending data differently than Loop? It's a long shot but that could be one thing. But then why would it do that only sometimes. Can you see a pattern connected to networking, e.g. if you're on a reliable WiFi vs mobile 4G far out in the woods? How often does this occur? Do some days show a completely continuous line while some days are patchy like this? Any pattern there? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6236 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEZWMYFCP7CG5XZ3B5EACTSK2FVLANCNFSM4SOV63ZQ> .

[bjornoleh]

Here is a sample from the patchy period

The timestamps are at exactly the five minute mark, or one second before.:

In NS

[bjornoleh]

Some of the SGV entries today look like this, and the Loopalyzer report is continuous at this time. The timestamp is 2-3 seconds before the 5-minute marks.

These are displayed in NS with timestamps at 04, 09, 14, 19 minutes etc:

We changed the sensor last night, I dont know if the timing of SGVs are affected by the sensor start operation in the G6 app, or if that is a fixed timing for each transmittor.

[bjornoleh]

So it seems like Loopalyzer do not play well with data that are arriving at around the whole minutes mark, since NS seems to round the values down to the nearest full minute, and can't deal with any other than 5 minute intervals.

[sulkaharo]

Nightscout only rounds the time for display, the actual logic in Nightscout uses millisecond precision for the data. @bjornoleh in your sample, it's purely incidental the the timestamps align to a full five minute mark - this just means you happened to start the CGM transmitter session at at that moment. FWIW Loopalyzer should allow at least +/- 10 second difference to the CGM data timestamps, as a lot of CGM sources don't have the original CGM device timestamp available and the data generation timestamp varies due to a lot of reasons, such as the time it takes for a Bluetooth connection handshake to happen, which can easily cause a few seconds variation to the timestamps. Additionally we don't even know the full internals of how apps such as the official Dexcom app generates the timestamps and it's possible even that source uses the logic on the phone to generate the timestamp, which would cause it to vary.

[bjornoleh]

Ok, yes I would expect the rounding to be for display purposes only. I still suspect a timing issue with Loopalyzer. But of course it could be incidental. tor. 15. okt. 2020, 10:25 skrev Sulka Haro <notifications@github.com>:

…

Nightscout only rounds the time for display, the actual logic in Nightscout uses millisecond precision for the data. @bjornoleh <https://github.com/bjornoleh> in your sample, it's purely incidental the the timestamps align to a full five minute mark - this just means you happened to start the CGM transmitter session at at that moment. FWIW Loopalyzer should allow at least +/- 10 second difference to the CGM data timestamps, as a lot of CGM sources don't have the original CGM device timestamp available and the data generation timestamp varies due to a lot of reasons, such as the time it takes for a Bluetooth connection handshake to happen, which can easily cause a few seconds variation to the timestamps. Additionally we don't even know the full internals of how apps such as the official Dexcom app generates the timestamps and it's possible even that source uses the logic on the phone to generate the timestamp, which would cause it to vary. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6236 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APEZWM3RTXHIQGA2ZGBQNADSK2WXNANCNFSM4SOV63ZQ> .

[bjornoleh]

@lixgbg I don't know if my reasoning in the previous posts are worth much, but I thought I could add some data.

Here is the Loopalyzer for a backup instance of NS which is only receiving Dexcom SGVs (Bridge). I just wanted to know if there was a difference between BG data coming from Loop ("Upload readings") and Dexcom.

Edit: its not at all the same picture with Dexcom only (see next post)

[bjornoleh]

Dexcom only

The day before, also Dexcom only, can be patchy here as well: