Handling large amounts of data using redux-persist

[dublin-anondson-movista]

Platform

• Android

Issue Description

I'm trying to vet this storage engine as possible alternative to using AsyncStorage. But I can't seem to get android to behave correctly when I add a lot of data to state. In our android app at around 35-40Mb the app crashes.

It's closed source so I can't provide much in terms of code snippets but here's the logs, based on what I can tell MMKV falls on its face and stops persisting data after it can't allocate any more memory. I find that strange considering it's a small amount of data.

Is there some way to circumvent this from happening? Surely mmkv can scale better than this?

2021-04-07 17:27:23.019 9940-9940/com.movista.*** D/ApkSoSource: not allowing consideration of lib/x86/libmmkvnative.so: deferring to libdir
2021-04-07 17:27:26.916 9940-10024/com.movista.*** I/MMKV: <MMKV.cpp:156::initialize> version v1.2.7 page size:4096
2021-04-07 17:27:26.916 9940-10024/com.movista.*** I/MMKV: <MMKV.cpp:198::initializeMMKV> root dir: /data/user/0/com.movista.***/files/mmkv
2021-04-07 17:27:29.125 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:81::loadFromFile> loading [mmkv.default] with 0 actual size, file size 4096, InterProcess 0, meta info version:0
2021-04-07 17:27:29.126 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:130::loadFromFile> loaded [mmkv.default] with 0 key-values
2021-04-07 17:27:30.257 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:367::ensureMemorySize> extending [mmkv.default] file size from 4096 to 16384, incoming size:989, future usage:7976
2021-04-07 17:28:33.686 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:367::ensureMemorySize> extending [mmkv.default] file size from 16384 to 262144, incoming size:14211, future usage:132136
2021-04-07 17:28:41.561 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:367::ensureMemorySize> extending [mmkv.default] file size from 262144 to 134217728, incoming size:11615236, future usage:93043816
2021-04-07 17:29:16.524 9940-10032/com.movista.*** I/MMKV: <MMKV_IO.cpp:367::ensureMemorySize> extending [mmkv.default] file size from 134217728 to 1073741824, incoming size:46415236, future usage:649843840
2021-04-07 17:29:20.822 9940-10032/com.movista.*** E/MMKV: <MemoryFile.cpp:117::mmap> fail to mmap [/data/user/0/com.movista.***/files/mmkv/mmkv.default], Out of memory

As a general question, is this library designed to handle large amounts of data? The upper limit I'm trying to reach currently is at about 300Mb.

Let me know if I need to provide any more info! Thanks! 😄

[mrousavy]

Hi! I personally haven't tested the limits of MMKV, but keep in mind that it heavily relies on RAM (in-memory database which writes back to disk). So if your phone has a RAM of 500MB and you're already using 300MB of that just for database, it's likely your app will get killed (out of memory).

I don't know if you can change the maximum memory it allows before running out of memory or if that's determined by the OS, but you could try digging around in the native code and patch it yourself. Also, you'll probably get more information on the MMKV repo; Tencent/MMKV.

My advice; If you really need to store those huge amounts of data and don't need to access that data really often use a file-based database.