query for profile.json

[zreptil]

There is currently a problem with iOS loop, which fills the profile table with an enormous amount of identical profiles and additional "loopSettings" data. I think iOS loop is misusing the profiles table.

The problem with this is, that it is not possible to query the data from the profile table in such a way as it is possible with entries and treatments. If the data is requested, one gets all the data in the table and this can hit the timeout of the server so that nothing is sent back over the api.

It would be good, if there were a way to restrict the data just like with entries and treatments with find option. For example something like this:

xxx/api/v1/profile.json?find[millis][$gte]=1600639200000&find[millis][$lte]=1600725599999

This should return only the profileentries between two dates.

The source of the problem is iOS loop, that is misusing the profile table for storing information that would be better placed in treatments or devicestatus. Since there is no query for profile up to now, i think it was never considered to hold such large amounts of data. I got testdata from users that have several thousands entries in the profile table. Most of them are identical except for the loopSettings part.

[sulkaharo]

Yes. This has been discussed earlier; please raise the issue with Loop developer so they'd start using the API as intended.

[zreptil]

Unfortunately I use neither iOS loop nor do I have any connection to the developers. So this solution with the query seems for me to be currently the best way to have a workaround.

[bjornoleh]

@zreptil Here is one of the discussions on the topic, as well as some temporary solutions for tidying up the profile collection, and a patch that can be applied to Loop to prevent unnecessary profile entries. At least something to do as we wait for a proper solution.

The culprit is mainly the profile entries which are triggered by every activation (and possibly deactivation) of temporary overrides in Loop.

NS issue report #5801

I suppose the problem at hand is that NS/Atlas is timing out while attempting to get the data needed to generate reports at https://nightscout-reporter.zreptil.de/ ?

Thanks for the nice reports, by the way!

[sulkaharo]

@bjornoleh Also note the profile code in NS is highly performance critical and uploading profiles when the actual dosing data hasn't changed is highly problematic for responsiveness of UI and reporting in NS.

[zreptil]

@bjornoleh

thanks for the link to the discussion. After now implementing a not very good solution to this problem in nightscout reporter and thinking about it a little bit deeper, I think also the query would not solve the problem. When you don't have a crappy uploader that floods your data with unneccessary and unwanted data, than you can have a profile that works for months and you will not get any profile if querying only data for the given period (e.g last three weeks). Having all profiles loaded at once is the best solution, but leads to problems as long as the api is not used correctly by the uploader. I think, somebody at the iOS Loop development team should take a deep look in the code, read the api documentation of cgm-remote-monitor and fix this bug as soon as possible. It is not the fault of nightscout or of apps that use the api correctly. But the users of the apps have clearly a problem with this. I tried my best to circumvent this bug in nightscout reporter, but this solution is still a patchwork.

I suppose the problem at hand is that NS/Atlas is timing out while attempting to get the data needed to generate reports at https://nightscout-reporter.zreptil.de/ ?

Yes, exactly. Users, that use iOS Loop are not able to get the full profile list. So they are limited to a smaller amount of data, that is redundant and that will have a big chance for missing the needed data when generating reports of former times.

Thanks for the nice reports, by the way!

You and every one else are very welcome :)

[sulkaharo]

Oh also - @zreptil you're probably looking to query the /profiles API (note the s) which supports the date parameters.

[zreptil]

Oh also - @zreptil you're probably looking to query the /profiles API (note the s) which supports the date parameters.

@sulkaharo I tried it with profiles, but it gives me an empty response. Also I cannot find documentation for the profiles API. How do you use it?

[sulkaharo]

@zreptil load /api/v1/profiles.json and use the same query parameter format as with the other API endpoints. FWIW I don't think it's really possible to work with the data Loop is sending, so if you want to support Loop users, it might be worth discussing with Loop devs on making a release that starts only storing profiles in the profile collection

[zreptil]

load /api/v1/profiles.json and use the same query parameter format as with the other API endpoints.

sounds nice, doesn't work. I dont get any content with profiles.json or even profiles. Only /api/v1/profile brings some data. And this data cannot be queried. Maybe you use a patched version of nightscout. But in the servers that i access, there is no api/v1/profiles.json available.

[sulkaharo]

Which Nightscout version do you have? This works for me on 14.0.6 release and dev. Can you post the query URLs you're using?

[zreptil]

I also use 14.0.6. You can try your best at https://zreptil20200627.herokuapp.com. I really don't get results with api/v1/profiles.json. If you can get results, please send me the link with your successful request.

[sulkaharo]

Ah right - what's going on is the API endpoints that support query parameters constrain loads that don't include date parameters to recent data to reduce the database load, and the data in the instance you have is getting pretty old, so no data is found with the default query. If you want to query for older profiles, you can find these dates using for example the following query parametrisation: /api/v1/profiles.json?find[startDate][$gt]=2018-01-01T11%3A30%3A17.694Z

[zreptil]

@sulkaharo Great, thank you for this help. I tried different queries, but I never used startDate and profiles.json together for it. Maybe I can use this for selecting the correct profiles even when Loop on iOS is misusing the data. Do you know since which version of cgm-remote-monitor this query works? I didn't find this in swagger when I started with nightscout reporter and even now I cannot find it there.