Pending bolus insulin is zeroed out in some circumstances, leading to wrong prediction

[elnjensen]

In the current code (1.5.0 master and 1.5.1dev as of this date) the start of boluses is confirmed with the pump, and in general this allows that insulin to be tracked correctly (for predictions, temp basals, and subsequent boluses) until a full pull of history from the pump occurs. Most of the time this works as designed.

However, there are circumstances where I have observed this to fail for a short window. What happens is that I issue a bolus, I get the spinning cursor waiting for the bolus to be confirmed, and then the forecast updates to reflect that bolus insulin. However, sometimes during the next few minutes the forecast will then flip back to being very high, and the corresponding recommended temp basal (or recommended bolus if I go to the bolus screen) indicates that Loop has "forgotten" that there is a bolus being delivered (or recently completed).

If someone would like to try to reproduce this, it appears to me that one thing that can trigger this is the arrival of a new BG value shortly after the bolus is initiated. So to reproduce, wait until you need to bolus, then look at the timestamp on the BG value and wait until it is 3 or 4 minutes old. Then enter carbs as needed and deliver a bolus. Then watch the forecast in Loop and see what happens when the new BG arrives a minute or two later. Often the forecast will then flip back up to a predicted very high value. It only stays up briefly, and then soon comes back down again once the bolus is fetched from pump history. But there is a risk here - on Facebook someone reported delivering a second bolus during that window (due to doing two carb entries with different absorption times for different parts of a meal) and thus accidentally delivering too much insulin.

Looking at the code, I have a guess about what may be happening. When a bolus request is issued to the pump, function addRequestedBolus is called, which sets lastRequestedBolus. That variable is then used here in function getPendingInsulin to track that insulin while waiting for bolus confirmation. Then when Loop receives confirmation that the bolus started without error, the function addConfirmedBolus is called. This clears lastRequestedBolus, and calls addPendingPumpEvent in DoseStore. This adds the event to the current doseStore, which allows it to be used in predictions. All good so far.

According to this comment, pending events in the doseStore are cleared when either addPumpEvents or addReservoirValue are called. That seems like it should be fine, since either the history fetch should show that bolus, or the reservoir level should indicate that it has been delivered. But my guess is that this is where the glitch is happening. I don't know when the pump records a completed bolus to its internal history, but it may be that if the history fetch occurs while the bolus is still occurring, it could miss the current bolus, but still clear the pending bolus record from the doseStore. If this is true, a possible solution could be to check the pump status and not pull history if the pump is currently bolusing. Similarly, I don't know when mySentry packets are sent, but if the pump sends a packet with a reservoir value before a bolus is complete, possibly addReservoirValue could get a value that doesn't fully reflect the bolus delivery, but in the process might still clear the pending event from the doseStore, again leading to a temporarily-wrong amount of pending insulin.

Again, this is a narrow risk window, but still I think it would be good to close it if possible. I'll continue to look into some of the ideas I posed above, but I would be interested to hear thoughts from others. If the second comment about mySentry packets and reservoir levels is correct, this should never happen on x22 pumps, so any testing there would be useful.

[dm61]

I think you've correctly identified the root cause of the problem. Except, I do not think reading pump history is feasible during the time pump is bolusing. But, even an attempt to do so may result in Loop defaulting to a prior history which obviously does not yet contain the new bolus entry, so the effect may be the same: the new bolus is "forgotten" for a brief time interval. Timing of mysentry packets could also result in the same problem.

I am about to put together a duplicate system with a 522 pump, so I'll be able to bolus as much as I like - will try to reproduce the timing you suggested.

[elnjensen]

Great, thanks @dm61 . I'm waiting for a backup RileyLink and then I will set up a test system with my backup 522 as well. Right now I'm limited in testing this because I don't bolus all that often. During my most recent bolus I did successfully fetch history using the RL menu, so it does seem to me that that is possible.

[dm61]

@elnjensen I've now reproduced this multiple times on a spare system using a 522 pump. It looks like the sequence of events is exactly as you've described: bolus is entered; as soon as pump starts delivering bolus, its effects are reflected on both the insulin delivery chart and in the bg prediction, which is fine; but then, if Loop reads pump history during bolus delivery (which it is indeed able to do), the ongoing bolus effects disappear; Loop may further go yellow if it attempts enacting temp basal and the bolus is still in progress; in the end, over the next 5 minutes Loop incorrectly predicts high bg, and will erroneously recommend (and be able to deliver) a duplicate bolus. Even worse, the larger the bolus is, the longer it takes to deliver it, and the bug is more likely to occur.

[elnjensen]

Thanks for testing, @dm61 . So it seems like possible fixes are either to not read history if the pump is bolusing, or (maybe better?) to go ahead and read history but not expire the pending events if the pump is bolusing. @ps2 do you have any thoughts about whether one of these (or something else) would be better? I may try some testing to first just see if I can get code working that just checks on the currently-bolusing status of the pump before fetching history, but not change anything else yet.

[ps2]

There is a fix for this in LoopKit and Loop. To test, checkout the set-pending-bolus-end-date branch of Loop, then update Loop's LoopKit carthage dependency to build the mutable-doses-heuristic branch of LoopKit.

[elnjensen]

Ah, excellent, thanks Pete! @dm61 had pointed out that set-pending-bolus-end-date branch earlier as possibly related to this issue, but at the time I hadn't dug into LoopKit enough to see the connection. Date filtering of the mutable pump events seems like a good solution. One very small observation: you mention x23 pumps in the comment here but it seems like this is an issue for other pumps, too, and actually worse for x22 because of the slower bolus delivery.

Glad to see this taken care of - and it was good experience for me to dig into the code a bit more. I'll try to test it out today.

[dm61]

Thanks @ps2 and @elnjensen. I had (unsuccessfully) tried the set-pending-bolus-end-date branch, not realizing the main part of the fix was in the mutable-doses-heuristic Loopkit. Anyway, the set-pending-bolus-end-date Loop branch with mutable-doses-heuristic Loopkit branch fixes the issue, as far as I can tell after testing a number of different boluses and timing patterns on a 522 pump (the issue is definitely not specific to just x23 pumps, x22 pumps are also affected). A minor issue with the fix is that the pending bolus temporarily shows up as as a large temp spike in Insulin Delivery, and is then updated to the bolus triangle after the next history fetch.

[Kdisimone]

and I'm late to the party...just managed to have her phone in my hand to capture the screens midway through this. Just like you described. the mid-bolus pump read causes the high temp basal to appear on the delivery graph for a small bit and the remaining bit of bolus will still appear in the bolus recommendation until the bolus is over and the pump is read again. Here's what I saw mid-bolus...about half way through a 5.2 unit bolus, Loop said active insulin was about 2 units and a temp basal of 40.2 u/hr (it was interpreting the bolus as temp basal for a bit). At the time of this screen shot, I looked at the bolus screen and it said it was recommending 1.1 units with some still pending.

and then after the bolus ended and the pump was read again, things went back to correct.

[ps2]

Thanks for testing and reporting back, @dm61!

[Kdisimone]

@ps2 I'll rebuild and test it tomorrow too...thanks for the fix

[BK1112]

@Kdisimone hi, did you test this one and did it work? Thx!

[elnjensen]

I've been testing this, too, and seems like I'm still seeing the issue occasionally. I'm wondering if there might be more ways for this to crop up on x23 pumps vs. x22, due to the MySentry packets as well as the history fetch. I'll keep watching it, but I'm curious: @Kdisimone, do you have x22 or x23? I know @dm61 said he's using 522.

It's conceivable that I somehow got a cached version of the wrong LoopKit branch when building, though I don't think so. I'll try to rebuild again and double-check. (Is there any way to check this once the app is built? Hard to see how to do it if there's no version change.)

[dm61]

Yes, I did all testing on a 522 pump. I'd take a look at cartfile.resolved to confirm which LoopKit branch is included with Loop that was installed. I've found myself forgetting to checkout a branch and installing an unintended version. On another note, in the debugger it should be possible to see arrivals of mysentry packets, which may help in testing.

[dm61]

Over the past 3 days, I've been running a 1.5.1dev branch (including set-pending-bolus-end-date fixes) on my 523 (which I am wearing) and, unfortunately, I've seen this issue pop up again on two occasions, which followed the same pattern as described by @elnjensen. In earlier testing of the set-pending-bolus-end-date fix with a spare 522 pump I was unable to reproduce the issue - everything seemed to work fine. Could this be related to MySentry packets, or some x23 specific timing pattern? @ps2, any suggestions what else we could do to find out why the fix may not work for x23 pumps?

[12/19 Edit: after updating to the dev branch as of 12/16 I have not been able to reproduce the problem on my 523. Will continue to keep an eye on this, but it seems that the fix works]

[elnjensen]

@dm61 You could try following the logic through and looking at what triggers both addPumpEvents and addReservoirValue to see if it's possible that both of them could get called in succession, e.g. one from a reservoir read and one from a history fetch (?) and cause the logic to fail. I don't have time to look at it now but that's where I'd start.