History expiration strategy

[grigoryk]

Depends on ability to delete history. See #1373.

We have limited resources and potentially unlimited browsing history, especially considering syncing against very large profiles.

There are basically two reasons for expiring history:

1. it's taking up too much disk space
2. it's slowing down the browser - more history usually means slower queries that operate over history (like top sites, frecency calculations, etc)

We'll need to investigate both of these reasons. Questions:

• what are the signals OS may send us regarding our disk usage, and how to interpret them? E.g. Fennec relies on ACTION_DEVICE_STORAGE_LOW to aggressively purge history records
• what's the performance profile of our underlying HistoryStorage implementations vis-a-vis expensive queries such as frecency/top sites/etc? This will greatly vary depending on implementation, but probably just care about browser-storage-sync. cc @thomcc and @mhammond to comment on places lib.

For reference, Fennec has two types of automatic history cleanup:

• on receiving ACTION_DEVICE_STORAGE_LOW intent, aggressively drop most history except for the last 500 or so visits
• when Fennec is going into background, drop history outside of the "last 2-3 months" time window

┆Issue is synchronized with this Jira Task

[grigoryk]

In addition to local requirements, we have Sync's point-of-view:

• this device is a replication of user's history, so the more history it has, the better our data-recovery-in-some-edge-cases story is

[buttercookie42]

when Fennec is going into background, drop history outside of the "last 2-3 months" time window

Actually it's four weeks or 2000 history entries, whichever is more.

[thomcc]

Depends on ability to delete history. See #1373.

We probably want to handle this not using the generic history deletion APIs we give you, that API assumes the deletion is synced.

In general, this is harder than it should be due to fundamental limitations of how Sync 1.5 models history (which sucks). This model also makes things like 'delete the last N months' weird and not really how things work.

what's the performance profile of our underlying HistoryStorage implementations vis-a-vis expensive queries such as frecency/top sites/etc

Frecency isn't that expensive to compute in our model, it also is done in automatically in response to changes instead of as part of queries. (It also might get even less expensive if we do mozilla/application-services#610).

I don't know what the top sites query looks like well enough to say for it.

Ultimately, yes, storing all history forever is unlikely to be viable, and clearing it in response to ACTION_DEVICE_STORAGE_LOW seems reasonable (although my desktop history with 250k visits is only ~75MB, I'd hope we clear browser caches before something the user is likely to miss such as history).

2-3mo is extremely low though, and flaws in the sync 1.5 history data model mean that devices that store a subset of history (and expect to write history) can cause problems for all other devices (causing them to delete history unnecessarily depending on how it is done), so hopefully we can enable the things you need with acceptable performance.

[thomcc]

Can we get an idea of how large (in megabytes) of a DB y'all find acceptable?

Specifically:

1. What would you expect expect to be reasonable in normal use when there's no size contention (user has tons of space to spare).
2. What would you expect us to shrink down to after you get an alert about disk size and call us telling us to prune.

Please ignore any concerns about query performance. It's a related concern for certain queries, but assume everything performs fine here.

[mhammond]

Can we get an idea of how large (in megabytes) of a DB y'all find acceptable?

Alternatively, is this something that can be configured at runtime? I'm concerned that if we try and come up with a one-size-fits-all size, we will end up choosing a size that's likely to work on the lowest spec'd device we want to run on - which is a missed opportunity for more modern devices. For example, if the history database takes 2% of the total space taken by the profile, asking us to 1/2 the space we consume doesn't seem the correct tradeoff - the free space will not have increased significantly but the user will be frustrated that some of their favorite sites no longer auto-complete.

How is the space allowed for the http cache configured? ISTM that these are roughly the same problem - the more space taken the better the result for the user - right up until it takes up too much space.

[grigoryk]

a-c have is now exposing prune method from places, which should be the entry point for expiring history.
The rest of the work will happen in mozilla/application-services#674.