NS API will not delete treatments (regression in v0.11.1)

[Pogman]

Using NS v0.11.1

Deleting treatments from the NS db with the http DELETE --> https://xxxxxxxx.herokuapp.com/api/v1/treatments/5c7ff3a24d6b7ce7ffe84284 form works but seems to not actually be deleting treatment items older then a certain period (~4 days) but yet NS api returns as a successful operation.

Testing by reverting to NS v0.10.3 works as expected .

pazaan/600SeriesAndroidUploader#243

[jpcunningh]

This is due to modifying the treatments DELETE API to comply with the query formats used by the other collections and other query API formats. Part of the standard format is that if a date is not provided in the query, a date is added by enforceDatefilter to add a date requiring the matching records be no older than 4 days old.

[PieterGit]

@jpcunningh @Pogman Can either of you summarize what we need to do. Am I right this is a regression, or is it something that @Pogman expected to work, but never worked for treatments. I don't think I fully grasp the issue, and never delete treatments (except for indiviual ones with the Nightscout UI and that works for me)

[Pogman]

The .../api/v1/treatments/id has always worked as expected pre v0.11.1 and was in fact the only way to delete a treatment as no other method existed.

This method still works but is now limited to 4 days max and I would recommend changing it so as to not be limited so that deleting directly by id always works. Having a default limit for the new query based stuff seems fine.

One side effect of the v0.11.1 implementation is that the api will always return as a successful complete operation even if it fails due to the date limit.

Going forward for the 600 Uploader I will add a check against NS versions and use the appropriate delete method.

[Pogman]

You might want to check the NS reports treatments section in case the treatment editing and delete functions still work as expected for items older the 4 days.

[jpcunningh]

@PieterGit, should we remove the enforceDatefilter requirement for DELETE method on all of the collections if an ID is provided? This makes sense to me since if you have the ID, you have gone through the data enough to know a specific record you wish to remove, so the risk of accidental deletion is much lower.

[PieterGit]

@Pogman @jpcunningh can one of you PR this change? (and perhaps provide an extra unit test that will test this behaviour?). Please let me know if you can do this soon (and that we merge it with 0.11.2) or that it will take some time (and we'll merge it with 0.12).

[tzachi-dar]

I have tried doing a delete with also giving the date string, but it still did not work.
I give the command:

curl -s -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[_id]=5c8d8c1517444b00048b4aba\&find[created_at]=2019-03-16T23:51:49.138Z

On papertrail I see:
Delete records with query: { find:
Mar 16 17:26:59 snirdar-dev app/web.1: { _id: '5c8d8c1517444b00048b4aba',
Mar 16 17:26:59 snirdar-dev app/web.1: created_at: '2019-03-16T23:51:49.138Z' },
Mar 16 17:26:59 snirdar-dev app/web.1: count: 10 }
Mar 16 17:26:59 snirdar-dev app/web.1: devicestatus records deleted

So I do give a datestring, and it does not work.
Anything I can do to have it working on the latest version as well?

[Pogman]

@tzachi-dar query based deletes work, you have a slash that should not be in there

api/v1/devicestatus.json?find[_id]=5c8d8c1517444b00048b4aba&find[created_at]=2019-03-16T23:51:49.138Z

[Pogman]

Misread your comment thinking it was for treatments!

For devicestatus it should be the same but I'm only certain for treatments.

[jpcunningh]

@tzachi-dar, are you able to test PR #4481? I believe it will allow you to delete a record by _id without requiring a date.

[tzachi-dar]

Thanks for the answers:
@Pogman I need to escape two strings because of the shell "&" and $ (for example $lte)
So, this does work:
curl -s -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[created_at][\$lte]=2019-03-10T05:55:10.312Z

It now seems that I can delete things while specifying both datestring and id. For example the following line works:
curl -X DELETE -H 'API-SECRET: 911eea185f5c758ac275687da8b7eab50ad7xxxx' -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[_id]=5c85ee8871982a000474998a\&find[created_at]=2019-03-11T05:13:44.830Z

@jpcunningh I'll try to check your pr tonight, although I think that specifying the date-string, is a reasonable workaround.

[tzachi-dar]

By the way, a little of toppic for this bug, but I'll still report it here:
It seems that getting an object from device status by id, does not work.

For example getting by created_at works:
curl -s -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[created_at][\$eq]=2019-02-07T23:38:48.655Z
[{"_id":"5c5cc188ecce760004890141","device":"מיומיו","uploader":{"battery":50},"created_at":"2019-02-07T23:38:48.655Z"}]root@openaps:~#

Getting by created_at and _id works:
curl -s -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[_id]=5c5cc188ecce760004890141\&find[created_at][\$eq]=2019-02-07T23:38:48.655Z
[{"_id":"5c5cc188ecce760004890141","device":"מיומיו","uploader":{"battery":50},"created_at":"2019-02-07T23:38:48.655Z"}]

Trying to get only by device id does not work.
curl -s -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus.json?find[_id]=5c5cc188ecce760004890141
[]

[tzachi-dar]

OK, seems like with the new code, delete works as expected. For example: this works
curl -X DELETE -H 'API-SECRET: 911eea185f5c758ac275687da8b7eab50ad720e3' -g https://snirdar-dev.herokuapp.com/api/v1/devicestatus/5c5e16ba3e7bce00045c94d7

The problem repeated above was also solved.

Thanks for the fast fix.