-
-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Excessive swapping or limited swap space making VMs unresponsive #6659
Comments
How is this different from the memory values the Qube Manager already shows for each VM? |
I have personally never experienced this problem, nor can I recall hearing others report it. Is it because Qubes is installed on an HDD rather than an SSD? In any case, implementing a tool for users to manually monitor the problem sounds much less promising than finding the underlying root cause and fixing it. |
The underlying cause is opening "too many" firefox tabs in a VM because you don't know how many is "too many" (I.E. how much free memory is left inside the machine). The same goes for "too many" applications running in a VM but firefox (and especially torbrowser) are more likely to bloat and become unresponsive while you are away. Increasing the amount of memory does not help because it just increases the number of tabs (or applications) you can have open, the user still doesn't know when they need to stop. |
@ddevz We actually did propose some of what you're suggesting, in two prototypes for a new App Menu custom built for Qubes, tracked in #5677. In the survey to gauge user sentiment regarding feature options (#6573), this feature measured very well with over 2/3 of users responding positively to it. It's not prioritized for an initial release, though, as our goal for that release is to just get the fully-new widget onto users machines for feedback from use. The data for what those prototypes propose, only mirrors what the existing Domains widget (In 4.0 it's the Q menu on the right part of the screen, in the Tray area) provides. Do those existing metrics provide the insight you're looking for (but obviously presented more contextually so you have more natural visibility into it)? |
If I understand your question properly, then no. Those metrics do not provide sufficient insight. The 4.0 "Q" widget on the right shows how much memory the memory balancer has allocated to each VM. This is valuable information as it's part of the question "how many more VMs can I create before I cant start VMs anymore (due to being out of memory)?" and "which VMs do I need to kill to free up enough memory to start the VM I want to start?".
|
I've run into this quite often on SSD in VMs set up primarily for research that tends to require a lot of open tabs. If I also have a few other VMs open, memory pressure can lead to that VM being stuck in perma-swap and unresponsive. It would be nice, particularly for browser heavy VMs based on Linux templates, to be able to control the (max) size of the volatile volume and the swap size from the qubes settings, and have the VM boot process utilize those values. [Above is responsive to your question...but stopping there to return to topic.] |
If I'm understanding correctly, it sounds like we have two different (non-mutually-exclusive) proposals that both aim to address the same underlying problem: the "memory monitoring" approach and the "swap config" approach. |
I do not see how "swap config" would solve the problem. Can you elaborate how one might use swap config to resolve the issue? I'm totally speculating here... maybe your thinking of setting swap to zero so the process dies instead of the VM becoming unresponsive? |
I understand now! my theory is that swapping itself is causing the "have to wait 30+ seconds to get a response" problem, and that running out of swap space is causing the "totally unresponsive (I.E. can't use the terminal anymore and have to kill the VM)" problem. If I am correct, then your idea of expanding the swap space could fix the "totally unresponsive" problem, but not the "have to wait 30+ seconds" problem. If running out of swap space is causing both problem 1 and 2, then expanding the swap space would fix neither as the user would continue to open tabs/launch applications until the same thing happened. However, being able to expand the swap space does sound important for users with limited overall system memory (I.E. the total memory that xen gets to use), when that user wants to run VMs that will go beyond what their system memory can handle. (I added extra memory to my system and had forgotten about this case :) ) So to me it sounds like we solve separate (but related) problems that lead to the same symptoms. |
Adding more swap to a VM that runs Firefox avoids also "have to wait 30+ seconds" in most cases. This helps for Firefox specifically, because a lot of memory allocated by it is never released (aka memory leaks). While ideally applications wouldn't leak that much memory (which is hard, in such complex thing like a web browser...), adding more swap helps to mitigate the issue. Or at least significantly delay it. Each VM has a 10GB volatile volume ( A word of caution: using big swap does help for Firefox, which leaks a lot of memory. But it helps only because most of that memory is not accessed anymore. If an application really uses a lot of memory, then having more swap will actually make the situation worse - if more swap is used (and accessed frequently), it will make the application even slower, resulting in unresponsiveness much longer than 30s. |
I thought Firefox had fixed its leaks many years ago 😞. |
I believe Firefox did fix memory leaks (at least majority). As for today, Firefox consumes way less memory than Chromium/Chrome for having the same amount of tabs loaded and etc. Maybe the problem is actually that Firefox allocates too much memory because it sees that a lot of memory is available (to be faster and utilize this extra free memory), and does not free it aggressively as it may be reused by itself. |
I had been assuming that the memory leaks were in the javascript of the various pages you have open, as it seems highly dependent as to what sites you have open. |
This issue is being closed because:
If anyone believes that this issue should be reopened and reassigned to an active milestone, please leave a brief comment. |
This still is a problem, and it happens quite a lot. Freezing the VM to death if you do not check for a restart of Firefox and free space. swapoff does not help, as VM still can get unresponsive over time. Please advise. |
Am using a relatively recent install of the 4.2.2 image. PC is limited with 16 GB RAM, with all the rest of active VM's I tried to make it work with 2,1GB of RAM. The VM's that get stuck all have 30-50 tabs open without Java active. Turning off swap helped insofar that you would notice running into the bottleneck by response time and could end firefox before, but that does not always work -> when it happens to disposable VM's you loose your work, since no option to restart. With all the VM's I usually uncheck: "include in memory balancing", which probably is a factor. Same use-case on a live systems (porteus) with 8GB RAM / no swap, there it appears to be never an issue, but that much I can not assign. |
Do you also adjust their memory size ("initial memory")? The default for initial memory is 400MB which is definitely not enough to run Firefox or similar, and without memory balancing, VM will not get more RAM. |
Yes. The VM's that are the problem for me are Disposable VM's, with the last attempt I set "initial memory" to 2100 MB in the "default-dvm" with the hope it will do. Kind of a habit to uncheck "include in memory balancing" .. only really thinking of it now .. would have to see how it turns out, but since the RAM is rather limited for me and using 10+ VM's of different RAM sizes at the same time, chance is that the swap space likely still fills up over time, which was the original issue. |
Well, swap will fill up if VM has less RAM than applications running inside try to use. Yes, web browsers are extremely memory hungry apps. Check |
One of the questions was if it can be helped with adjustable swap space, since others have encountered it (older qubes versions but same issue) all with -presumably- "include in memory balancing" enabled - I feel that 1GB of swap may indeed be to little on some VM's (Firefox use) and you would run into it over along enough period of time when you are not able to give it 4-8 GB of RAM -> but with a more swap it could do. Using your solution from on Jun 2, 2021 - will that always be possible in the future? /dev/xvdc is 12 GiB now but just executing it seems to work fine and it shows in top/htop. |
On the chance that you might know how to do bash scripting: it's possible to put a script in cron in the qube that you are worried is going to run out of memory, set it to run every 5 min and notify you when its getting low by using something like: you can get your free memory in the qube with something like: |
@horizon2021: Please try resetting all initial and max memory settings to default (i.e., 400 MB / 4000 MB) and enabling "include in memory balancing" on all qubes where it's possible. (In other words, put all memory-related settings back to the defaults.) Then, reboot the whole system and trying using it for a while to see if the problem persists. |
Thanks. With the tutorials out there I probably could get this done; right now (whatever the security concerns are) just creating a new swap file seems easiest solution to me. sudo swapoff -a sudo fallocate -l 4G /swapfile sudo chmod 0600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile |
Will do a reinstall with the current version and get back to you. Please give it a while. |
FWIW, I don't think a complete reinstall is necessary for the the settings I mentioned. In fact, if you back up all of your qubes first and restore them into the new installation, they may still have the old memory settings, so you may still have to change them manually anyway. |
Alright, just a reboot. It is 11 active VM's in total, two HVM's 400 MB Ram (USB, Network), rest all set to default 400 - 4000. I recreated all VM's but sys-USB and the one that I use Telegram-desktop in. There are three disposable VM's with more or less intense (tabs,java) Firefox use. All booted up fine but there was a noticeable delay in some disp. VM's in the beginning that was unexpected, switching tabs, writing something down in editor all took some delay (this is usually not happening with "fresh VM's" that have just been started to be used), it however did go away after a while and all seemed fine. At one point I killed one disposable VM and started a new one, that one was really slow, due to RAM limitation. You can with this default memory setting, make memory available from elsewhere by closing application in another VM. Which helped here. At last I was running into the situation with the swap filling up in one VM to it's limit again, however without total crash, only unresponsive for maybe 20-30 seconds, which was the VM that runs the Telegram app. It seemed to do ok until starting a video, this is the earliest copy+paste I could do after the problem occurred: top - 23:14:28 up 22:40, 2 users, load average: 1.31, 2.05, 1.18 memory usage due to telegram seems seems to vary a good bit, saw this application run fine below 1000 MB Ram with very little swap for a while, but it can also go up more, right now: MiB Mem : 3230.9 total, 1387.6 free, 1419.4 used, 488.3 buff/cache Also sys-whonix seems to take more than I would expect: MiB Mem : 1560.2 total, 269.9 free, 1092.7 used, 229.1 buff/cache So the issue, as far as I initially reported, can be helped with just using the default (none became totally unresponsive). Memory limitation may still problematic since (the point has been made above) you do not immediately know when you are running into the system limit. |
The problem you're addressing (if any)
2 problems:
Describe the solution you'd like
When clicking the qubes "Q" icon, it shows the memory used for each VM from a memory balancing perspective. One could put a memory number there.
Exactly what number would be best to put there is not entirely clear, as memory stats as displayed in the "free -h" command could be used to show free memory inside the VM devided by available memory inside the VM.
However, due to memory balancing, this is not the whole story. Perhaps free memory inside the VM (as can be displayed by the "free" command) devided by the max memory the memory balancer is willing to provide it?
Where is the value to a user, and who might that user be?
Not having to wait 30+ seconds for responces, and not having to loose data that you hadn't gotten off a disposable VM yet because that VM it is entirely unresponsive.
Describe alternatives you've considered
Normally one would put a system monitor applet in the task bar that shows memory consumption. However, it is unlikely this is a usable solution for qubes because there would end up too many monitors in the task bar and for security reasons
Additional context
Relevant documentation you've consulted
Related, non-duplicate issues
The text was updated successfully, but these errors were encountered: