G7 backfill overlap fix

[Navid200]

Fixes: #3398

How to Recreate the problem
You will need a G7 session in progress and a test phone.
On the test phone, uninstall xDrip and install the latest Nightly (June 28, 2024).
If you have a G6 session in progress, establish connectivity to it. If you don't set up fake data source.
After establishing connectivity to your G6, wait for another read cycle so that it backfills the past 3 hours.

Now, stop the session. It is important to stop the xDrip session. There is no reason to stop the real session on G6.

After having stopped the xDrip session, change the transmitter ID to your G7 session in progress.
After establishing connectivity to G7, wait for another read cycle so that the backfill is completed. You will see that you will get a 24-hour backfill even though you have had 3 hours of readings from G6. So, you will end up with an overlap.

Tests
All tests were run on an Android 11 test phone.

1- Establishing connectivity to a G7 after a fresh install results in a 24-hour backfill.
2- Establishing connectivity to a G6 after a fresh install results in a 3-hour backfill.
3- Establishing connectivity to a G7 after having stopped a G6 session, a backfill takes place to fill only the empty time after the last G6 reading. In other words, it does not cause an overlap.

[Navid200]

I thought I had solved the problem. But, further tests showed failure.
So, I have taken a different approach, being executing your ToDo comment. That's what this PR does now.

[jamorham]

I'm not sure whether this actually should look at the last sensor stop time and not backtrack before that rather than looking at when readings have been received?

[Navid200]

I don't understand your point.
If the last reading is from a sensor that has not stopped yet, we still should not overlap it.
If that last reading is from Nightscout follow, and we are now switching to become a collector, even though the sensor we were following with Nightscout has not stopped yet, we should not backfill over the readings.

If we switch from a G6 transmitter to another, and the first one has not stopped yet, we should not overlap its readings.

Did I misunderstand your point?

[Navid200]

The overlap actually uploads to Nightscout and AAPS. It is really bad.
Can you give me any scenario where you would like xDrip to backfill over a previous reading?

[Navid200]

There are times a G7 user chooses to switch to their next G7 before their current 10.5 days are completed.
In that case, the previous sensor never stops and they start using the new sensor.
We still should not backfill over it if the new sensor started a few hours before they switch to it.

[jamorham]

If the sensor is stopped then there should not be any readings for the period while the sensor is stopped. Here we are activating the sensor and I'm thinking we should cap how far back we backtrack the start time of this sensor to not before the end of the previous sensor session. That is what my todo says.

If you look only at the readings then you potentially are overlapping sensor periods which I think is the cause of the problem overall. So you can use Sensor.lastStopped() to see if there is a previous sensor and stopped_at to find the time it stopped and use that probably with Math.max() to cap the new sensor start time so it can't be before the previous one. Indeed this could even be a method added to Sensor to allow it to be fed any start time but it then will apply the cap based on any previous sensor and always start it safely which would probably be cleaner overall.

[Navid200]

If the previous session has never stopped in xDrip, your suggestion will fail to avoid a backfill overlap.

This PR will avoid a backfill overlap under all possible conditions.

[Navid200]

If you look only at the readings then you potentially are overlapping sensor periods which I think is the cause of the problem overall.

I respectfully disagree.
Sensor periods overlap. This is nothing new. People have been presoaking G6 for a very long time.
What's new is that Dexcom has offered a 12-hour grace period. Therefore, there are many G7 users who presoak every sensor by 12 hours. So, the sensor periods overlap. This is not a problem. This is a fact of life and xDrip can cope with it easily.

I didn't want to make this solution only for G7 because there have been reports of overlap even for G6.
So, I took a fundamental approach and deals with the cause of the problem at its source.
If we have a reading, we have already uploaded it to Nightscout and AAPS. There is absolutely no justification for overlapping a different reading with it, which would be rewriting history. So, we avoid it. If we have a reading, we never backfill over it.

Can you tell me how this is ever going to allow an overlap?

I have shown the different scenarios that can cause an overlap here: #3398

This PR avoids all of those.
Would you please tell me under what condition this PR can cause a problem?

[Navid200]

Let's say the user started their current G7 at 11:00 am 10 days ago.
Tonight, at 11:00 pm, their session will stop.
But, at 10:00 pm, they decide to go to bed and in order to go to bed, they switch to their new sensor, which they inserted this morning at 8:00 am, right then at 10:00 pm.
This is 1 hour before the current session will expire.
When they switch, there will never be a session stop for the current sensor.

If I use last stopped, it will find the session that stopped 10 days ago. And there will be an overlap.
This PR sees that the last reading was at 9:55 pm tonight. So, it only backfills starting from then avoiding any possibility of overlap.

Would you please tell me what I am overlooking?

[Navid200]

I am sorry that I misunderstood your TODO comment.
I can put that back in.

Regardless, I believe this is the best solution.

Please give me a scenario that you believe this PR will fail, and I will try to test exactly the scenario you are concerned about. Either I will learn that I am missing something and will change this PR back to draft, or I can show you that the scenario works fine after this PR and there should be no concern.

[jamorham]

So I think the situation is that with G7 and possibly with G6 we get that the xDrip sensor session is not tied to the actual sensor (eg if you change transmitter id) - This was not the original intention of the Sensor class but that dates back to G4 and where we did the algorithm internally. In that case you are correct that it is the readings we should be looking for.

The only problem I can see is if we somehow got readings from elsewhere that were not yet synced (via follow perhaps) but I think that is very much an edge case where the user would have to try hard to break things and then they would likely only incorrectly fill in any open gaps in the data.

[Navid200]

Yes, I agree 100%.

Gaps should be avoided.
If a gap is caused accidentally, having the Google Drive daily backup enabled may offer a possible solution. In that case, the user can restore their latest daily backup; then, use their current G7 to backfill the past 24 hours filling all gaps.
But, the best solution is to avoid causing gaps, which will not be backfilled over after this is merged.

[Navid200]

Please, please, don't ask for a switch!
:o)

[Navid200]

Thanks

I will make an announcement on facebook after merge.

[Navid200]

I'm not sure if this is working as intended. I still see overlaps uploaded to Nightscout and also in xDrip history.
This is from 2 days ago using the latest Nightly:

I will be watching very carefully on my next sensor change, in 8 days, and report here.

[Navid200]

I'm sorry I don't think this was a perfect solution.

A problem I have is that after I test a PR and submit it, I will have to switch back to the latest official Nightly so that I can see what everyone else sees.
Most developers can continue to use their PR. I cannot. If I do, I will be isolating myself from what most users use. I am supposed to provide a link between the user community and the developers. So, I cannot use my PR for a long time. There may be a conflict between me as a developer and me as a support staff. I am sorry.

I will have to open a new PR to address this and I am so sorry.