InvalidStateError: A mutation operation was attempted on a database that did not allow mutations. #141
Comments
|
How much free disk space you have? |
|
118,292,480 bytes free on that firefox drive (browser.cache.disk.parent_directory is on a different drive with much more free space) |
|
Not much I will be able to do, apparently Firefox errors when uBO tries to write to the indexedDB. If you search for |
|
No, that setting is still set "true". If I disable EasyList and restart firefox and Update Now, the update completes. If I then Purge Cache and Update Now, it fails with the same error but after a different list (unbreak). Briefly searching for this error mentions the dom.indexedDB.enabled setting (which I have set "true"), but results also suggest it can be caused by not waiting for an objectstore to be created before using it (async operations). |
uBlock-user
added
unable to reproduce
|
Can't reproduce. Filters updating as expected. Reset Firefox settings to default(including the ones you set in about:config) and try again. |
~120 megabytes? I have problems with filters update on my Android when free space is below ~350MB |
|
According to Browser storage limits and eviction criteria, the global limit is 50% of free disk space, so ~60 MB in the current case. From this, there is a group limit, which if I understand correctly is that at most 20% of the global limit can by assigned to uBO, so 12 MB in the current case. So yes, it does seem like an issue with low disk space. |
|
@gorhill it is possible to detect this case and show some dialog? |
|
It's kind of a catastrophic error, the documentation said:
When this occurs, there is no fallback possible. This occurs deep in the code, when using Firefox API, and there might be no uBO UI available, aside the icon in the toolbar (assuming it has not been hidden by the user). At this point, if I am going to throw lot of coding work at this, I rather work on the feasibility of compressing the data so as to reduce the storage footprint -- well it would be nice if Firefox's implementation of indexedDB did this for everybody. At this point however, those suffering that sort of issue will have to free disk space. In the current case, ~120 MB is atypically low, regardless of uBO. |
|
When this happens filter lists are shown as outdated and when I click on "Update now" animation is spinning indefinitely - a way to fix only this case? |
From what I see in the call stack above, this fails at the database level, not at the transaction level. There are error handlers at the transaction level. But it's a bit more complicated at the database level because the handlers at that level do not know about which callers need to be notified. It just means it's possible but not trivial to fix this. If I instead work on compressing (assuming feasability), this would possibly prevent uBO from failing (except in even more extreme cases of low disk space). |
|
I can reproduce after filling my disk by Browser console log |
Isn't a scary thing to try? I fear the desktop/OS is going to start to fail badly before I can even get to try to repro. |
|
How is it a bug if you have to fill your hard disk just to reproduce it ? If your hard disk is at its limit, any software is expected to malfunction. |
|
Well the point is maybe uBO should at least stop spinning endlessly in its UI, though the core issue though would still be there: an essentially defect uBO because of low disk space. |
:) @uBlock-user this happens to me frequently on Android when my free disk space fall bellow ~350MB This is 2014 phone with only 4GB of disk space total and bellow 1GB for user. Updating apps may cause this and you will not even know when this happens until ads starts appearing. And you will not know why because there is no message, no browser console, just animation spinning. One time I left my phone with uBO Dashboard enabled for the night to check if something change. |
|
Ok I will come up with something. Since there is not always a UI available (ex. when auto-update kicks in, or selfie is taken, etc.), I am thinking of showing a red warning on the Settings page (where the "Storage used" area) for when the "mutation operation was attempted" error occurs -- which could be caused by other reasons than just low-disk space I suppose. That warning would be cleared whenever a write operation succeed. |
|
My Firefox profile is on a virtual encrypted disk (vera/true) & portable, therefore not as large as the full drive. (cache is not on the same drive) |
Looking at the code, I am not seeing why this would be. When uBO writes to the storage, it does so without minding the result. Are you sure about "indefinitely"? Can I see a screenshot of the console, I want to see if there is an unhandled exception, so far what I see is all handled by uBO. |
|
I can reproduce now. Found an old 1GB stick, created a new Firefox profile on it, and filled the stick with a 800MB file, there is around ~60 MB left. I get the errors now in the console, so now I have something to work with. |
|
One solution may be to go back to extension storage (should not have limits) - Mozilla will move backend to IndexedDB soon https://bugzilla.mozilla.org/show_bug.cgi?id=1474562
Yes, I even once leave phone for the night with it spinning. |
|
30 minutes later and still spinning. |
Using So far I have resisted adding this permission because of how this was received by some users when Privacy Badger added it to their manifest . In any case, now that I can reproduce I found out that there is an exception thrown, and this is trivially handled. uBO should still work fine without being able to use its cache (up to a point), this will just cause it to launch less efficiently since it might have to re-compile all filter lists every time. The errors will still be written at the console, but there should no longer be unhandled exceptions. |
|
I read the data through a XMLHttpRequest in the demo/benchmark page, and feed an ArrayBuffer to the WebAssembly engine. In the extension code I use I actually did not try XMLHttpRequest in the extension code, I figure this would not be allowed because one could XMLHttpRequest to anywhere and use the bytes from the response to execute code from any origin -- so I stuck with the spirit of executing-code-only-from-the-package, anything else which would happen to work would quite certainly be an issue to fix in Firefox. |
|
Really interesting results! Just out of curiosity, did node-lz4 have slower compression/decompression speeds than lz4-wasm? Also are the benchmarks using LZ4-HC equivalent settings or default/other? |
|
I simplified a bit the compressor implementation by not putting in code for detecting incompressible data, but even without this it's faster than |
|
My findings: Looks like Firefox stores lots of nulls in database: Size of *.sqlite file (only) on my standard Nightly profile with latest uBO rc was ~40MB. I thought something is broken, but filter lists are recent and all seems to work fine. I switched back to rc - *.sqlite 41MB, *.files 7MB. Side note: storage.local is stored in second folder, with And I just confirmed this in fresh profile - lz4 branch uses ~11MB of disk space, latest rc 41MB of *.sqlite and 7MB in *.files without selfie, 12MB with selfie. selectedFilterLists |
…-issues#141) Squashed commit of the following: commit 6a84738 Author: Raymond Hill <rhill@raymondhill.net> Date: Mon Aug 6 10:56:44 2018 -0400 remove remnant of snappyjs and spurious instruction commit 9a4b709 Author: Raymond Hill <rhill@raymondhill.net> Date: Mon Aug 6 09:48:58 2018 -0400 make cache storage compression optionally available on all platforms New advanced setting: `cacheStorageCompression`. Default is `false`. commit 22ee654 Author: Raymond Hill <rhill@raymondhill.net> Date: Sun Aug 5 19:16:26 2018 -0400 remove Chromium from lz4 experiment commit ee3e201 Author: Raymond Hill <rhill@raymondhill.net> Date: Sun Aug 5 18:52:43 2018 -0400 import lz4-block-codec.wasm library commit 883a311 Author: Raymond Hill <rhill@raymondhill.net> Date: Sun Aug 5 18:50:46 2018 -0400 implement storage compression through lz4-wasm [draft] commit 48d1cca Merge: 8ae77e6 b34c897 Author: Raymond Hill <rhill@raymondhill.net> Date: Sat Aug 4 08:56:51 2018 -0400 Merge branch 'master' of github.com:gorhill/uBlock into lz4 commit 8ae77e6 Author: Raymond Hill <rhill@raymondhill.net> Date: Wed Jul 25 18:17:45 2018 -0400 experiment with compression
|
WOW, you actually coded the compression library in assembly?! |
|
Updated to the latest dev build, set |
|
I am showing a 3X size reduction as well, storage size was 15megs and is now 5megs with |
|
Correction (update) to my comment about database size - my Nightly finally decided to put something inside *.sqlite file. Now it's *.sqlite: 16MB, *.files: 12MB No idea why this happened - I cannot reproduce in fresh profile... |
|
Sounds like the load selfie to me. Note that when you enable |
|
Changed setting "cacheStorageCompression" to true, then went to purge and finally added my lists, before 11 megs and now 4 megs (close to 3x size reduction!), using Firefox 62 Beta on Android, works! |
|
I used the Gecko profiler to find out how compression affects load time of uBO. I measured with a selfie available (which I consider the most likely scenario for majority of users). Tested using a desktop computer with an SSD drive. I repeated the disabling/enabling of uBO three times in the profiler, with enough time in between to ensure clearly separate instances in the profiler. I filtered out the profiler output using As a result there were three clear regions of CPU usage, corresponding to uBO being launched and doing its work to load everything in memory. I then focused the profiler timeline to match exactly the start and end of each of those regions, using the magnifier to increase the granularity until the timeline was matching exactly the start/end of uBO doing its launch work. This allowed me to find out the length in time of the focused region. This is the results:
Those figures indicates that despite the added overhead of having to lz4-decode the loaded data, uBO loaded faster. In the Gecko profiler results, I could definitely see an area of idleness in the middle which has been shrunk when using compression, enough to gain faster load despite the lz4-decoding overhead. Below are comparative screenshots, you can see the area of idleness in the middle being reduced when compression is used (bottom graph):
|
Not sure if you know this, these blue "hills" in the graph are not cpu usage. This is stack size. Even low "height" may use lot of cpu. https://perf-html.io/docs/#/./guide-ui-tour?id=timeline39s-thread-stack-graph |
|
There's a performance-wise gain too when force updating filterlists, previously it would take a second or two for it to complete, now it downloads and updates the lists instantaneously and snappier even on Chromium. |
|
The original issue has been addressed, the compression won't ever guarantee that running low on disk space will never affect uBO, so at this point I consider the issue fixed. |
|
@gorhill sometimes filters take long to download with compression on any ideas on how to debug it? |
|
... Apologies if this isn't the proper place to post about my predicament, but I was directed to this issue via the Release Notes of the latest dev build... OS: Windows Vista SP2 32-bit I was updated to the latest dev build from a previous dev build (from memory, must've been 1.16.17.7) which is now removed from the GitHub repo. I opened the dashboard and tried to manually update 3rd-party filters; the "clocks" at the far right of each list all started spinning but after a few (3-4) seconds all turned simultaneously into an orange triangle (with a white exclamation mark inside), then, after another 2-3 seconds, all went back into the "clock" state... This behaviour I had never witnessed previously, after some checks it looked as though my lists hadn't updated 😭 E.g., the "I don't care about cookies" list remained at v2.8.8, while I checked and v2.8.9 is already on line at its site... From looking closer at the release notes of 1.16.21b0, I saw
By launching Firefox Browser Console (CTRL+SHIFT+J), clearing everything inside it and starting a manual 3rd-party filter list update, the console is populated with many
I do hope they're meaningful to someone of you wonderful people... For now, my options to successfully update 3rd-party filter lists on my setup are tested to be:
I understand that I don't have things on my side, as Vista has been EOL'd by MS and Fx 52.9.0 will be EOL'd by Mozilla in over a week's time... However, any insight as to what's the cause of this new issue will be highly appreciated... Many thanks in advance.... |
This is recommended. |
|
I only get this in console in 64bit Firefox:
Filters are not updated, something may be wrong with fallback js code. You may experience this #150
After switching on
BTW, this issue is resolved, you should create new one. |
|
Huge thanks for taking the time looking into my reported issue 👍
I read all the discussion there, but I got the sense - I could be wrong, I know nothing about javascript language - that it did not pertain directly to my problem, despite being a FxESR 52.9.0 issue (but on Linux...); thanks all the same for the reference!
For some obscure reason, I had already inside
... Because of my marginal setup (EOL'd Windows OS, soon-to-be EOL'd Fx version, 32-bitness of OS (hence browser)), I wasn't really sure, to begin with, whether the new "compressing" code was meant to be compatible with my setup (or even tested in a non-Quantum Fx flavour and/or in a 32-bit browser build), so I opted to first post here, apologising in advance if inappropriate 😉 ... I just updated to the newest dev build, v1.16.21b1 and can confirm that filter lists update now works as it should with A sterling job indeed, hugely valued! 👍 🥇 |
Thanks for this tip, I'm sure I knew it already, but (originally) posting late at night and under some frustration doesn't help much in my growing age... Just to confirm, do the filter-list storage files the WE version saves inside |
|
Settings exported by clicking in uBO Settings tab on "Back up to file" button will be saved in special portable text format (json) and will contain all uBO settings and addresses of filter lists you are subscribed to. One thing missing will be your statistics (request blocked since install). |
|
I'm on 1.16.20 on Firefox 61.0.2 on Mac OS 10.11.6, and I'm unable to set |
|
This feature is in beta tested now. You can install uBO development builds for Firefox from here https://github.com/gorhill/ublock/releases |
|
@gwarser if so, then perhaps the wiki documentation (https://github.com/gorhill/uBlock/wiki/Advanced-settings) should be changed? There, it says the feature is for "uBO 1.16.17b and above." |
Prerequisites
Description
Fails to update filters
A specific URL where the issue occurs
moz-extension://d879bf29-1bac-4212-8cbb-be53485d7ab0/dashboard.html
Steps to Reproduce
Open Firefox's "Browser Console" to see the error
Expected behavior:
Filters should update.
Actual behavior:
Filters do not update.
Your environment
uBlock Origin v1.16.14
Firefox 61.0.1
Windows 7 x64
The text was updated successfully, but these errors were encountered: