Excessive profile syncing to Nightscout

[sulkaharo]

Hi! I got reports from some Loop/FreeAPS users that they're having issues with Nightscout profiles. Checking their profile data, I noticed they have a huge amount of profile data in their system, where in some cases the profile has been synched in three times within one second, resulting in the profile collection bloating to be huge (in one case, recent profiles resulting in a 33 MB JSON object).

I doubt the profile actually changes this often, so I wonder where the glitch is that causes this. One suspicious part of thee data was that the loopSettings section seems to have "scheduleOverride" section, which seems to be part of the runtime data and not part of the profile, where if the next scheduled basal change is sometimes entered into the profile data and sync is caused by a difference in the profile, this would trigger an unintended profile sync. I'll check if there's a simple fix for the database code in NS, but checking if there's an issue in Loop would be great.

[bartsopers]

I actually also experienced this on AndroidAPS yesterday night. If it helps, I think I can provide you with logs.

[loyalchallenger]

mLab mondgoDB is being discontinued on Heroku. What else can I use? I’ve been unable to run autotune reports because I think nightscout is not storing my data. Thanks!

[sulkaharo]

@loyalchallenger see https://github.com/nightscout/cgm-remote-monitor/wiki/mLab-discontinuation-FAQ

[loyalchallenger]

@loyalchallenger see https://github.com/nightscout/cgm-remote-monitor/wiki/mLab-discontinuation-FAQ

Thank you so much!

[loyalchallenger]

@sulkaharo I think I broke it. I updated Nightscout like it said and did everything. But now Nightscout is saying my API secret is incorrect even though it is correct. I tried all the tips I found on google, but nothing is working.

[loyalchallenger]

I was able to get nightscout back up and migrate to Atlas. I still can’t get autotune to run though.

[Nightfoxy]

Jon Fawcett has created change that reduces the profile updates, but isn't a complete fix.
The basic issue is that loop uploads a new profile whenever any settings are changed or overrides are run or modified. This can add up quickly, even if the settings in NS are not what was updated.

Its causing issues on my NS site as well. I don't know of a quick way to purge old profiles.

06b8a11

[sulkaharo]

You mean a new profile is sent every time Loop sets a basal rate? The profile data is very much not intended for that purpose - it should only be used to store the long term dosing profile. There's multiple other ways to store temporal data, please describe the scenarios to me that you'd like to store and I can help figure out something that works in concert with Nightscout design.

[Kdisimone]

@sulkaharo the problem is that loop doesn't save profiles itself for users to switch between. So it's a little hampered of a system to integrate with NS yet in a way that would be most clean. Ideally, the integration would be like remote overrides are done currently. Loop saves profiles, those are uploaded to NS and the looper or a NS user (with api authorization) could switch between profiles like you can on most pumps.

Without loop saving profiles, the benefit of loop uploading current schedule for basal, cr, and isf is that a remote parent can have a look at existing settings before making suggestions to school nurse or Looper about a direction to change settings as needed. Unfortunately, that benefit is limited to CURRENT settings, so loop does need to upload every time those would change in loop settings by the user

[sulkaharo]

If I understand correctly what the data is, the correct way to store that would be a devicestatus record. That collection is intended for the temporal data and the Loop pill will have access to the recent statuses at all times in a way that scales. This is what Openaps and Androidaps uses, in addition to treatment records. I understand the desire to communicate the information but as is, the way Loop uses the profiles breaks Nightscout and with the current architecture and storage limits most users have, there's no simple way to chance Nightscout to support the way Loop works right now.

[sascha-egerer]

@Nightfoxy @loyalchallenger

I've just deleted all the profiles to get Autotune working.
You can also do that by running this script on your nightscout page.
BUT BE AWARE! THIS WILL DELETE ALL YOU PROFILES BUT THE LAST ONE!

1. Use Chrome and open your nightscout
2. Open the Console to run JavaScript (https://developers.google.com/web/tools/chrome-devtools/open#console)
3. Paste this script into the console input field at the bottom of you browser

var url = window.location.protocol + '//' + window.location.host;
fetch(url + '/api/v1/profile.json', {
    credentials: 'same-origin',
    headers: {'api-secret': localStorage.apisecrethash}
}).then(response => response.json()).then(data => {
    data.forEach(function (item, index) {
        if (index > 0) {
            fetch(url + '/api/v1/profile/' + item._id, {
                method: 'DELETE',
                credentials: 'same-origin',
                headers: {'api-secret': localStorage.apisecrethash}
            }).then(response => response.json());
        }
    });
});

4. BE AWARE IT WILL DELETE ALL YOU PROFILES!
5. Hit Enter

[loyalchallenger]

@sascha-egerer , I had the time zone wrong in Atlas. 🤦‍♀️Mine is working now. Thank you!

[sulkaharo]

Related ticket in FreeAPS ivalkou/LoopWorkspace#6

[sulkaharo]

Would love some suggestions from Loop developers on how to fix this. @ps2, can you point someone at this? I'm in a slightly difficult position here, where I don't want to break NS for Loop users, but simultaneously the current Loop behaviour is breaking NS for said users who automatically assume this is a bug in Nightscout which adds to NS support load, plus this is possibly causing issues (maybe even wrong dosing suggestions) for Loop users who use Autotune, so something needs doing. Given I don't use Loop and don't have any other developers actively participating in Nightscout development, it's even hard to test any potential performance improvements that could mitigate this.