F-Droid builds rarely compact database

[ouchadam]

Steps to reproduce

1. With the F-Droid variant, join many large rooms (1000+ members, such as matrix-hq)
2. Notice the app storage steadily increases (this is expected due to realm's extra storage allocation)

Outcome

What did you expect?

For the session store database to be compacted

What happened instead?

The app is currently set up to only compact the database on cold launch, however, the f-droid builds have a noop GuardService which forces the Application instance to be kept alive which means we end up skipping the compacting.

Compaction can only happen when there are no connections to the database.

Your phone model

No response

Operating system version

No response

Application version and app store

1.4.0

Homeserver

No response

Will you send logs?

No

[ouchadam]

Marked as major as this can cause devices to run out of storage space

[T-bond]

Does this the cause of the #2620? Or at least partially (not the high RAM useage).

[ouchadam]

@T-bond #2620 (the storage only part) is actually a feature of the realm database we're using, the extra space allocations allow for better performance and internal consistency, I might close the issue...

however in the F-Droid build there may be times where we don't compact as often as we should leading to the device running out of storage.

[bmarty]

That's a good point.

We could try to use Realm.compactRealm(RealmConfiguration), but as you said, all the Realm instances as to be release, else this method does nothing (and returns false)

[opusforlife2]

Is it possible to programmatically react to high RAM usage? Maybe if the app is idle, but it's using something like 300-400 MB of RAM, it could trigger an app restart?

[ouchadam]

@opusforlife2 this sounds very reasonable, I'm imagining an option to force the f-droid app to quit and relaunch every X days (when idle)

[opusforlife2]

@ouchadam More like relaunch every X hours, when idle. X ~ 6, going by my own anecdotal experience. Let's see if 1.4.4 improves the situation any.

EDIT: I just noticed that having an active conversation quickly bloats the RAM usage. Like, 100 MB -> 1 GB within 5-10 minutes. Still on 1.4.2.

Edit 2: Yup. It's very consistent. 100% reproducible.

[hafiz013]

Guys for those who want to why this elemens keep growing the data user storage because matrix itself keep old session without deleting event though u already in new session. Hai deleveloper matrix please enchanced your code to improve memory storage without storage so many session. Below here is the proof why the apps data is increase when refersh token / synch background. Please improve the code

Question: How to make sure if new session then no need to store old session like before under files? How to do that?

[hafiz013]

@ouchadam how do u manage to solve issue data user storage/ram in the code? because if u said to download whole code of vector so difficult to find it. Could you explain how you do it please?

[opusforlife2]

Adding to #5244 (comment), @ouchadam, I have now had some long conversations on 1.4.4 as well.

GOOD NEWS: The insane 1 GB RAM hogging doesn't happen anymore.

Bad news: Element still hovers between 300-400 MB when actively having a conversation (with 1 person in a DM). Often, it goes up to the 400-500 MB range. This is still more than Fenix, a full-fledged web browser, so I hope it can be reduced further. For a closer comparison, Telegram FOSS hovers between 100-200 MB.

[opusforlife2]

Update 3: @ouchadam I just saw another Element restart happen. It ejected my browser (which I was actively using), media app, and everything else that was open from the RAM. I didn't get the opportunity to see how much RAM it occupied before restarting, but based on the fact that it removed all other apps from memory, it must have been a lot.

[opusforlife2]

Update 4: I just had a long conversation and was monitoring RAM side-by-side. It went up to 700 MB. :(

It was the only app I was using, so there was nothing to eject from RAM, but for a device with 3GB RAM, that is still too much.

[ouchadam]

thanks for the updates 👍

I just had a long conversation and was monitoring RAM side-by-side. It went up to 700 MB. :(

does the memory usage eventually return to lower amounts or do you have to kill the app?

[opusforlife2]

@ouchadam There is a sort of range in between which the memory hovers. Like if it's around 300MB, then it would stay between, say, 250 and 350, so for 700MB, it wouldn't have gone below 600MB in any case, even after a long time.

And Fenix is my main browser, which has its own RAM issues, so I need to kill Element pretty much instantly after having a long, RAM-heavy chat so I can use Fenix without problems.

Additional info I forgot to mention in Update 4: When I received an image and opened it, RAM usage jumped up by ~100MB. Don't know if that is an expected value, but it seemed like a lot for just one image.

[opusforlife2]

Until this problem sees a proper resolution, @ouchadam, is it possible to include a button to manually compact the database in Advanced settings? Something similar to the Minimize memory usage button in about:memory in Firefox? Maybe even make the button hidden behind Developer mode?

Compaction can only happen when there are no connections to the database.

Would a manual button be able to handle this problem?

---

BTW, out of curiosity:

The app is currently set up to only compact the database on cold launch

How often would a cold launch happen on a Play Store build, anyway? Wouldn't the app keep accumulating RAM until some app or the other, not necessarily Element itself, is ejected from memory?

[opusforlife2]

however, the f-droid builds have a noop GuardService which forces the Application instance to be kept alive which means we end up skipping the compacting.

@ouchadam After switching to ntfy for push notifications, the mega RAM-hungry Element service is gone completely, replaced by a 10x smaller ntfy service. Seems like a great way to bypass this problem.