Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak in v5.0.0 #21502

Open
kieraneglin opened this issue Oct 4, 2024 · 54 comments
Open

Memory leak in v5.0.0 #21502

kieraneglin opened this issue Oct 4, 2024 · 54 comments

Comments

@kieraneglin
Copy link

qBittorrent & operating system versions

qBittorrent: v5.0.0 x64
OS: It's running in an Arch Linux-based Docker container. The host is Unraid 6.12.13
Qt: 6.7.3
Libtorrent: 2.0.10.0
Boost: 1.86.0
OpenSSL: 3.3.2
zlib: 1.3.1

What is the problem?

Based on memory usage, there appears to be a slow leak introduced on my system by the 5.0.0 upgrade. I was running 4.6.7 beforehand (I think) without any issue.

The memory usage constantly increases and appears to be independent from torrent activity overall. There are spikes in memory when a torrent is actively downloading, but the overall trend slowly marches upwards as shown in the usage screenshot below. Restarting instantly fixes the issue in the immediate, but the usage will then start climbing from that point.

I understand that memory leaks SUCK to debug so I'm happy to run any diagnostic/troubleshooting steps you'd like!

Steps to reproduce

  1. Start qBittorrent
  2. Wait and see memory increasing

Additional context

Screenshot 2024-10-04 at 8 23 56 AM

Log(s) & preferences file(s)

watched_folders.json is empty (contents is {})
qBittorrent.conf.txt

@HanabishiRecca

This comment was marked as outdated.

@kieraneglin
Copy link
Author

kieraneglin commented Oct 4, 2024

Sure, but what would the explanation be for this only emerging with the upgrade from 4.6.7 to 5.0.0? Is it purely a reporting issue?

My logging history doesn't go back far enough for me to prove it, but I check my metrics dashboard a few times per week and this issue can be directly correlated with my upgrade to v5. That's not to say that you're wrong! But from where I'm standing this is a novel behaviour.

@TheBeastBHR
Copy link

I had the same experience, its using 22.5GB RAM very quickly before I ended the task from task manager (it wouldn't even exit normally and won't open a window to the app).
I used 4.6.6-4.6.7 extensively and didn't have this issue before.

@SanderBouwhuis
Copy link

This problem has been known for years now. Does v5 force the use of Libtorrent v2, or can I upgrade to v5 safely using Libtorrent v1.2?
I need to switch bittorrent clients every 5 to 10 years because projects somehow seem to screw up. This trend started with uTorrent.

@ArcticGems
Copy link

ArcticGems commented Oct 5, 2024

@SanderBouwhuis
there are 2 variation of each release, for example:
5.0.0
5.0.0 (qt6 lt20)

First one uses libtorrent 1.2 (both uses qt6)

In the next major release of qB (5.1 not 5.0.x), will have a workaround for the lt 2.0 issue.

@Dekumori
Copy link

Dekumori commented Oct 5, 2024

Can confirm. I was on 4.6.7 up until 10/01 and you can very clearly see where the upgrade took place. My Zabbix worker ended up crashing due to the memory usage spike and the memory usage has been erratic ever since.

image

@TGKx
Copy link

TGKx commented Oct 5, 2024

Can confirm same 5.0 docker container issue, RES size increased to 2GB of ram from initial usage of ~20MB for a single inactive torrent running over the period of 5-6 days. Going to stay on 4.6.x branch until this is properly fixed.

@ArcticGems
Copy link

ArcticGems commented Oct 5, 2024

@Dekumori @TGKx
What Libtorrent version is mentioned in your qB 5.0 installation: Help->About->Software Used

If Libtorrent 2.0.X, try qB 5.0 release variation with 1.2.X

@TGKx
Copy link

TGKx commented Oct 5, 2024

Qt: | 6.6.3
Libtorrent: | 1.2.19.0
Boost: | 1.84.0
OpenSSL: | 3.3.2
zlib: | 1.3.1

This is from docker image: https://github.com/qbittorrent/docker-qbittorrent-nox/pkgs/container/docker-qbittorrent-nox/281473771?tag=5.0.0-1

@Dekumori
Copy link

Dekumori commented Oct 5, 2024

@ArcticGems I am on Lib 2.0.X. I'll switch releases. I did mitigate the issue by limiting memory usage in my docker compose yml file. My system is stable, though the container is still unstable. Below is the memory usage since the change. I'll update after a few hours running the alternate release.

image

@maxim-mityutko
Copy link

The problem is definitely there. 4.6.7 was working without an issue. 5.0.0 gets OOM killed on my cluster every 8-9 hours, like a clockwork since the update
image

@ArcticGems
Copy link

The problem is definitely there. 4.6.7 was working without an issue. 5.0.0 gets OOM killed on my cluster every 8-9 hours, like a clockwork since the update

@maxim-mityutko
What Libtorrent version is mentioned in your qB 5.0 installation: Help->About->Software Used

If Libtorrent 2.0.X, try qB 5.0 release variation with 1.2.X

@Dekumori
Copy link

Dekumori commented Oct 6, 2024

Confirmed still occurring after switching to Libtorrent 1.2.19.0 release. It's not maxing out because I put a limit on the docker container.

image

@HanabishiRecca
Copy link
Contributor

Confirmed still occurring after switching to Libtorrent 1.2.19.0 release.

Finally a useful comment. There is another known issue:

Does it resembles your use pattern?

@Dekumori
Copy link

Dekumori commented Oct 6, 2024

It looks like both come down to issues with the webui being open. I closed all webui tabs and will monitor for a few hours. If the issue persists, I will restart the container and monitor again. If the issue still persists, I'll restart the server and monitor again. I'll return back with results once the issue is fixed or all of my above steps have been exhausted. I'm happy to try any alternative steps that are suggested as well.

@maxim-mityutko
Copy link

@ArcticGems


qBittorrent was built with the following libraries:

Qt:	6.7.3
Libtorrent:	1.2.19.0
Boost:	1.86.0
OpenSSL:	3.3.2
zlib:	1.3.1.zlib-ng

@maxim-mityutko
Copy link

It looks like both come down to issues with the webui being open. I closed all webui tabs and will monitor for a few hours. If the issue persists, I will restart the container and monitor again. If the issue still persists, I'll restart the server and monitor again. I'll return back with results once the issue is fixed or all of my above steps have been exhausted. I'm happy to try any alternative steps that are suggested as well.

  • The machine from which I usually manage the torrents is turned off for the most part of the day.
  • My instance is running in the pod in Kubernetes, pretty much isolated from anything else
  • I'm using VueTorrent WebUI instead of the native one

@IRainman
Copy link

IRainman commented Oct 7, 2024

Same behaviour in Windows 10 too.
Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”
image

@HanabishiRecca
Copy link
Contributor

Please post full technical details about your system and qBittorent build, as requested by the bug report template.
"Me too" comments without details are not helpful.

@HanabishiRecca
Copy link
Contributor

Thank you.

Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”

Try the regular one instead and see if the problem persists.
https://www.fosshub.com/qBittorrent.html?dwl=qbittorrent_5.0.0_x64_setup.exe

@IRainman
Copy link

IRainman commented Oct 7, 2024

Thank you.

Problem started after I'm updating from version installed from file “qbittorrent_4.6.7_lt20_qt6_x64_setup.exe” to the version “qbittorrent_5.0.0_lt20_qt6_x64_setup.exe”

Try the regular one instead and see if the problem persists. https://www.fosshub.com/qBittorrent.html?dwl=qbittorrent_5.0.0_x64_setup.exe

Thank you, but I don't have any reason to use it. I especially use the version with qt6, lib torrent 2.0, enabled memory mapped files, and SQLite for storing data. Currently, I am downgrading to version 4.6.7 and absolutely all goings fine.
image

@kieraneglin
Copy link
Author

Reverted to 4.6.7 with Libtorrent 2 and the issue does NOT persist. Usage in the graph correlates perfectly with new torrents being downloaded.

Qt: 6.7.2
Libtorrent: 2.0.10.0
Boost: 1.86.0
OpenSSL: 3.3.2
zlib: 1.3.1

Screenshot 2024-10-07 at 9 33 22 AM

@starobots
Copy link

I use the version 1.2 libtorrent of 5.0 qbt, but it cost about 20gb physics memory and all the virtual memory about 30 gb,50 gb in total.
And I installed 4.6.6 back, and it works much better than 5.0.0ver.

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Oct 7, 2024

Another possible test: try to reset the client to default settings.
I.e. delete existing client config (make a backup if you want). The config location per platform is described in the wiki.

Also we need:

  1. Reproduction steps. Exactly what you do with the client, maybe some patterns you've noticed.
  2. Logs. Any info as much as possible.
  3. Crash reports, dumps and backtraces, if any.

P.S. If everyone going to flip the table and roll back to older versions, the problem would never be fixed.

@kieraneglin
Copy link
Author

P.S. If everyone going to flip the table and roll back to older versions, the problem would never be fixed.

I'm not sure if this is your intent, but many of your replies are coming off as antagonistic. I've rolled back as a troubleshooting step to gather more details and not because I'm throwing in the towel. As said in my original post I'm more than happy to put in some work to help identify the root cause.

In any case, I'll spin up a new v5.0.0 Docker container which will provide me with a completely blank slate and let you know what I see!

@HanabishiRecca
Copy link
Contributor

HanabishiRecca commented Oct 7, 2024

I've rolled back as a troubleshooting step to gather more details and not because I'm throwing in the towel. As said in my original post I'm more than happy to put in some work to help identify the root cause.

Glad to hear. Due to the comment section being kinda spammy, it's easier to lose track.
That's because 3 comments in a row above my message looked like a throw:

I am downgrading to version 4.6.7
Reverted to 4.6.7
I installed 4.6.6 back


I'll spin up a new v5.0.0 Docker container which will provide me with a completely blank slate and let you know what I see!

Sure. It's a crucial step now to pinpoint the issue and make it reproducible.


And yes, my initial assumption (#21502 (comment)) seems to be wrong now. We just so used to such reports at this point, that anything about "memory leak" triggers that association by reflex.

@kieraneglin
Copy link
Author

Update: A completely fresh qbit 5.0.0 install with no torrents has been slowly increasing in memory usage. It's important to note the scale here - this is showing 62 MB to ~85MB over the course of 2 hours. Far, far slower than the prior ~500 MB per hour I was seeing before. These numbers may be too small as to be useful but I figured I'd update.

I'll keep it running and see what I find

Screenshot 2024-10-07 at 3 27 46 PM

@glassez
Copy link
Member

glassez commented Oct 8, 2024

@stalkerok
I know you maintain big amount of torrents. Have you already switched to qBittorrent v5.0?

@vincejv
Copy link

vincejv commented Oct 8, 2024

hello all, i'm here to confirm this bug, i'm fully aware of the mmap issue w/ LT20, actually i'm one of the people who pleaded to have the LT1.2 option with QBT 5.0 so I'm still using QBT 5.0 with LT1.2 ...

if anyone can recommend a tool to memory profile the client while it's running so I can provide more details? one that works on docker containers/qbitorrent nox.. it's so severe that the machine actually uses up the entire swap memory, this happened on two separate linux machines, one hasn't crashed yet thankfully, but will downgrade to 4.6.7 with lt1.2 for the meantime...

Machine 1:
image

Machine 2:
image

@vincejv
Copy link

vincejv commented Oct 8, 2024

@HanabishiRecca but ideally i would like to use my old profile so i can reproduce under the same environment and just figure out what's causing an issue with the old profile..

is there nothing like visual vm like in java for C?
image

@stalkerok
Copy link
Contributor

@stalkerok
I know you maintain big amount of torrents. Have you already switched to qBittorrent v5.0?

Yes, I am using libtorrent 1.2. I don't have any issues with the update, settings have not changed. The system and machine are completely stable.
2024-10-08_210200
2024-10-08_210249

@glassez
Copy link
Member

glassez commented Oct 8, 2024

@stalkerok
As I can see, you're seeding all these torrents. Perhaps the problem is with the download after all. Could you watch it at your place if you're going to download something soon?

@stalkerok
Copy link
Contributor

When downloading, everything is also fine. I didn't notice anything unusual.
2024-10-08_223405

@claabs
Copy link

claabs commented Oct 8, 2024

I'm seeing the issue on my instance with 2300 seeding torrents. It was upgraded from a 4.6.6 instance.

I noticed the memory increase rate is correlated with the upload rate. I used my Grafana dashboards to confirm over a 5 minute period with various upload limits:

  • avg upload rate of 4.42MiB/s; avg memory usage increase of 4.51 MiB/s
  • avg upload rate of 1.82MiB/s; avg memory usage increase of 2.02 MiB/s
  • avg upload rate of 28.9KiB/s; avg memory usage increase of 279 KiB/s
  • no memory increase with no network activity
  • I haven't tested download

I see the memory increase on both libtorrent v1 and v2.
Currently just limiting the container to a 10G limit, which causes this sawtooth memory chart (6MiB/s limit here):
image

System info:
Ubuntu 24.04
Docker: lscr.io/linuxserver/qbittorrent:latest & lscr.io/linuxserver/qbittorrent:libtorrentv1
qBittorrent: 5.0.0
Libtorrent: 1.2 and 2.0.10.0

@TGKx
Copy link

TGKx commented Oct 18, 2024

Just to add; on my qbt 5.0 libtorrent 1.2 container, I have a single torrent seeding and webui is running but I have not even opened the interface in a browser. It starts slowly the first two days seem relatively stable, then it leaks to a few hundred MB, then within 4-5 days it's normally into the 1-2GB memory usage range. So it doesn't seem to be strictly related to utilizing the WebUI or downloading. Hope this helps a little bit.

@stalkerok
Copy link
Contributor

Try reducing the file pool size and disk cache to 20 and 50MiB.
2024-10-18_111423
If it helps, you can increase these values to the limit you want.

@Estebiu
Copy link

Estebiu commented Oct 18, 2024

I'm using qbittorrent-nox on arch linux, and I'm having the same problem.
I just updated my system, and did a reboot only to be met with.. a completely unresponsive system. After several forced restarts, I found out the issue was qbittorent hogging al the ram.. after creating a 36gb swapfile, it seems like qbittorrent manages to eat up to 38-39gb of ram with only 100 torrents seeding.
Dark mode sure is cool, but I'm gonna stick to 4.6.7 for the time being.

2024-10-18_22:20:50:805652081

@wary-hermit
Copy link

I'm running two docker instances, lscr.io/linuxserver/qbittorrent:latest, Linuxserver.io version: 5.0.0-r2-ls359.
First instance with 99 torrents (3 active) takes up 5GB RAM.

image image image

Second instance with 2545 torrents (all inactive) takes up 100MB RAM.

image image image

5GB RAM container has 3 "active" torrents, but speed is 0, they're not even seeding right now. Just because I've seeded (or downloaded) a bit before it caused the problem. Container with 2k torrents and no memory leaks hasn't seeded anything, so no memory problem there.

@vincejv
Copy link

vincejv commented Oct 19, 2024

@wary-hermit which linux kernel? is it lts? 6.1 or 6.6 or 6.11?

@wary-hermit
Copy link

@wary-hermit which linux kernel? is it lts? 6.1 or 6.6 or 6.11?

Debian SID, 6.11.2-amd64

@wary-hermit
Copy link

Just updated from Linuxserver.io version: 5.0.0-r2-ls359 to Linuxserver.io version: 5.0.0-r2-ls360and watched as RAM usage grew.

After update - nothing has been seeded yet, just under 70MB of RAM (it was using 5GB earlier).
image
SCR-20241020-nied

After downloading a thing and actively seeding it RAM usage went over 1GB.
image
SCR-20241020-rcic

After few hours, right now, no longer actively seeding, but I've already seeded over 6GB - and RAM usage went over 2GB.
SCR-20241020-umwf
SCR-20241020-umum

So yes, thing that causes RAM usage to ballon up - is seeded amount. More GB seeded (even if no longer actively seeding) - more RAM will be used.

@HanabishiRecca
Copy link
Contributor

So yes, thing that causes RAM usage to ballon up - is seeded amount.

Are you sure it's not arvidn/libtorrent#6667?

@vincejv
Copy link

vincejv commented Oct 20, 2024

@wary-hermit what he's trying to say is if you're not using Libtorrent 2.0 as it's a known bug..

Are you on libtorrent v1 or v2? this can be confirmed on the about menu or docker tag used

@wary-hermit
Copy link

I think I'm on Libretorrent 2.0...
image

@HanabishiRecca
Copy link
Contributor

I think I'm on Libretorrent 2.0...

Then I guess you just see the intended behavior. As described in the very first comment here. #21502 (comment)

There is a neat trick to figure that out. Set File pool size to 0. If all the excessive memory drops immediately - that's it.
Don't forget to return the value back.

@wary-hermit
Copy link

Then I guess you just see the intended behavior. As described in the very first comment here. #21502 (comment)

There is a neat trick to figure that out. Set File pool size to 0. If all the excessive memory drops immediately - that's it.
Don't forget to return the value back.

Yep, that did the trick. And thanks for explaining! 👍

@Dekumori
Copy link

Dekumori commented Oct 24, 2024

I closed all webui tabs and will monitor for a few hours.

No change, still sawtooth.

If the issue persists, I will restart the container and monitor again.

Same behavior.

If the issue still persists, I'll restart the server and monitor again.

Same behavior.

I'm happy to try any alternative steps that are suggested as well.

I also tried:

  • setting the file pool size to 0.
  • stopping all torrents.

System Info:
Linux
6.5.0-28-generic
29-Ubuntu
x86_64

qBittorrent Info:
qBittorrent: 5.0.0
Qt: 6.8.0
Libtorrent: 1.2.19.0
Boost: 1.86.0
OpenSSL: 3.3.2
zlib: 1.3.1.zlib-ng

image
Red - Current Behavior
Yellow - Restarted Container, memory usage dipped and spike right back up
Blue - Stopped Container to confirm system memory was stable with all other applications running
Purple - Set File Pool Size to 0 with no change.
Orange - Stopped all torrents, saw momentary relief, but memory started climbing again shortly after. Did NOT resume torrents.

@segln
Copy link

segln commented Oct 24, 2024

I was experiencing OOM kills, but after reducing total trackers to under 10 the issue disappeared. Try it out.

@vincejv
Copy link

vincejv commented Nov 2, 2024

This seems to be highly correlated with the Web UI/API, so I had one instance of QBT running thru Web UI the entire day and the RAM Usage grew until OOM... while the other instance I didn't access the Web UI, just monitored through qbittorrent exporter tool (a Prometheus exporter that polls on QBT API)

The previous memory leak issues that were raised were the browser leaking, but this time the server/the qbittorrent client itself is leaking RAM on the server, I hope you can figure out what's causing this as pretty much this RCE Bug https://news.ycombinator.com/item?id=42004219 caused QBT < 5.0.1 to be outright banned in several popular trackers forcing the usage of QBittorrent 5.0

I would guess @stalkerok isn't affected since he's not using the Web UI.

@vincejv
Copy link

vincejv commented Nov 2, 2024

So to repro this memory leak, here's my checklist

  • Many torrents loaded in the client >400
  • Many trackers on those torrents, mine is around an estimated 200 tracker URLs overall
  • Have an active Web UI open all day long (refresh rate 2000ms)

@stalkerok
Copy link
Contributor

I hope you can figure out what's causing this as pretty much this RCE Bug https://news.ycombinator.com/item?id=42004219 caused QBT < 5.0.1 to be outright banned in several popular trackers

lol 😂
People don't seem to realize that this is not a critical vulnerability at all. I wouldn't call it a vulnerability at all. Somebody's just trying to make a big deal out of it.

I would guess @stalkerok isn't affected since he's not using the Web UI.

Yeah, I don't use the Web UI.

@HanabishiRecca
Copy link
Contributor

The previous memory leak issues that were raised were the browser leaking, but this time the server/the qbittorrent client itself is leaking RAM on the server

The issue mentioned in #21502 (comment) is about memory leaking on the server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests