Cached assets ignored or stale if filters are specified via managed storage

[jacobmcnamee]

Prerequisites

• I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
• This is NOT a YouTube, Facebook or Twitch report. These sites MUST be reported by clicking their respective links.
• This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
• The issue is not present after disabling uBO in the browser.
• I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

• uBO is the only extension.
• uBO uses default lists and settings.
• using a new, unmodified browser profile.

Description

If filters are specified via managed storage, restoreAdminSettings() calls saveUserFilters() and attempts to remove compiled user filters from the asset cache. This happens prior to cacheStorage.select(), meaning the wrong backend may be used to initialize the asset cache.

The problem occurs when both

1. adminSettings.userFilters or toOverwrite.filters is set in managed storage
2. the cache backend changes when cacheStorage.select() is called
   • occurs with default settings on Firefox on 1.56.0 (hardcoded switch to indexedDB)
   • occurs due to cacheStorageAPI setting on 1.56.1rc0 and beyond (after cache storage redesign)

I intended to open PRs but was unable due to access restrictions:

• Select cache storage backend before processing admin settings: jacobmcnamee/uBlock@681ca96
• Refuse backend changes after initialization: jacobmcnamee/uBlock@8f7b7a7

A specific URL where the issue occurs.

n/a

Steps to Reproduce

Note: these steps will overwrite all user filters

1. Start Firefox
2. Wait at least 2 min for filter lists to update, selfie to be created
3. Restart Firefox and observe baseline startup behavior:

14:16:03.319 [uBO] Start sequence of loading storage-based data 23 ms after launch console.js:38:13
14:16:03.319 [uBO] Admin settings ready 35 ms after launch console.js:38:13
14:16:03.321 [uBO] Backend storage for cache will be indexedDB console.js:38:13
14:16:03.321 [uBO] Hidden settings ready 43 ms after launch console.js:38:13
14:16:03.322 [uBO] WASM modules ready 44 ms after launch console.js:38:13
14:16:03.323 [uBO] Extra admin settings ready 44 ms after launch console.js:38:13
14:16:03.326 [uBO] List selection ready 47 ms after launch console.js:38:13
14:16:03.327 [uBO] Cache magic numbers ready 48 ms after launch console.js:38:13
14:16:03.327 [uBO] WASM PSL ready 48 ms after launch console.js:38:13
14:16:03.328 [uBO] First fetch ready 50 ms after launch console.js:38:13
14:16:03.333 [uBO] PSL ready 54 ms after launch console.js:38:13
14:16:03.333 [uBO] User settings ready 54 ms after launch console.js:38:13
14:16:03.435 [uBO] Selfie ready 156 ms after launch console.js:38:13
14:16:03.437 [uBO] All ready 158 ms (selfie) after launch console.js:38:13

4. Note that further restarts result in similar behavior
5. Create ~/.mozilla/managed-storage/uBlock0\@raymondhill.net.json with contents:

{
  "name": "uBlock0@raymondhill.net",
  "description": "",
  "type": "storage",
  "data": {
    "toOverwrite": {
      "filters": [
        "! test comment"
      ]
    }
  }
}

6. Restart Firefox once and observe similar behavior again (managed storage settings are cached internally and require two restarts to take effect)
7. Restart Firefox again and observe that selfie is not used, loading takes significantly longer, updater starts after 15s, etc.:

14:18:19.281 [uBO] Start sequence of loading storage-based data 23 ms after launch console.js:38:13
14:18:19.281 [uBO] Admin settings ready 34 ms after launch console.js:38:13
14:18:19.282 [uBO] Backend storage for cache will be indexedDB console.js:38:13
14:18:19.282 [uBO] Hidden settings ready 41 ms after launch console.js:38:13
14:18:19.284 [uBO] WASM modules ready 42 ms after launch console.js:38:13
14:18:19.284 [uBO] Extra admin settings ready 43 ms after launch console.js:38:13
14:18:19.290 [uBO] List selection ready 48 ms after launch console.js:38:13
14:18:19.291 [uBO] Cache magic numbers ready 49 ms after launch console.js:38:13
14:18:19.291 [uBO] WASM PSL ready 50 ms after launch console.js:38:13
14:18:19.293 [uBO] First fetch ready 51 ms after launch console.js:38:13
14:18:19.297 [uBO] User settings ready 56 ms after launch console.js:38:13
14:18:19.343 [uBO] PSL ready 102 ms after launch console.js:38:13
14:18:19.389 [uBO] loadFilterLists() Start console.js:38:13
14:18:19.406 [uBO] loadFilterLists() Redirects/scriptlets ready at 16 ms console.js:38:13
14:18:19.543 [uBO] loadFilterLists() All filters removed at 154 ms console.js:38:13
14:18:19.546 [uBO] loadFilterLists() Loading filters from user-filters at 157 ms console.js:38:13
14:18:19.576 [uBO] loadFilterLists() Loading filters from ublock-privacy at 186 ms console.js:38:13
14:18:19.588 [uBO] loadFilterLists() Loading filters from ublock-quick-fixes at 199 ms console.js:38:13
14:18:19.613 [uBO] loadFilterLists() Loading filters from ublock-badware at 223 ms console.js:38:13
14:18:19.631 [uBO] loadFilterLists() Loading filters from plowe-0 at 242 ms console.js:38:13
14:18:19.644 [uBO] loadFilterLists() Loading filters from urlhaus-1 at 254 ms console.js:38:13
14:18:19.685 [uBO] loadFilterLists() Loading filters from ublock-unbreak at 295 ms console.js:38:13
14:18:19.986 [uBO] loadFilterLists() Loading filters from ublock-filters at 596 ms console.js:38:13
14:18:20.205 [uBO] loadFilterLists() Loading filters from easyprivacy at 815 ms console.js:38:13
14:18:20.456 [uBO] loadFilterLists() Loading filters from easylist at 1066 ms console.js:38:13
14:18:20.552 [uBO] loadFilterLists() All filters in memory at 1163 ms console.js:38:13
14:18:20.629 [uBO] loadFilterLists() All filters ready at 1239 ms console.js:38:13
14:18:20.630 [uBO] Filter lists ready 1388 ms after launch console.js:38:13
14:18:20.631 [uBO] All ready 1390 ms after launch console.js:38:13
14:18:35.632 [uBO] Updater: cycle start console.js:38:13
14:18:35.633 [uBO] Updater: prefer CDNs console.js:38:13
14:18:35.878 [uBO] Full updater: successfully updated plowe-0 console.js:38:13
14:18:38.248 [uBO] Full updater: successfully updated urlhaus-1 console.js:38:13

8. Note that further restarts always result in the same degraded performance, updater running, etc., even after waiting for everything to complete

Expected behavior

Similar startup behavior regardless of filter settings in managed storage (only cached compiled user filters should be cleared)

Actual behavior

Slower startup, selfie not used, updater always runs, etc. when filters are specified in managed storage

uBO version

1.56.0

Browser name and version

Firefox 124.0

Operating System and version

Linux 6.6.5-arch1-1

[gorhill]

Thanks for the thorough investigation.

In 1.56.1+, the advanced setting cacheStorageAPI is to be used strictly for development purpose, it's not meant to be used in production. I should probably make that setting present only in the dev build.

Regarding the fix, I think the best way is that when toOverwrite.filters is present, the content should be compared with the current content of user filters, and if no difference is found, avoid invalidating cached selfie and compiled list.