Compared to 2.5.0, 2.8.0 uses 60%+ memory in a resting state

[zackshapiro]

Issue Description

We recently upgraded to Parse Server 2.8.4 from 2.5.0 for the improvements around Live Query between the two versions. In doing so, we noticed rather troubling increases in memory usage from Parse.

Here's a graph from CloudWatch of their respective idle MemoryUtilization

After downgrading our production instances back to 2.5.0, the MemoryUtilization goes back to near zero:

Steps to reproduce

Running two identical boxes on AWS aside from the Parse version, and having me be the only person querying each box in the last 24 hours. We observed these memory stats coming from ssh'ing in and running top.

Both running Node.js 8.11.1

Server Name: Release
Mem: 7867492k total.
5,816,296k used
2,051,196k free
181928k buffers
Every ~10 seconds, top showed the process node running at 62.5%

Server Name: Release-Downgraded
Mem: 7867492k total
688,704k used
7,178,788k free
46600k buffers
Every ~10 seconds, top showed the process node running at 1.4%

Expected Results

I expected far less memory consumption from the upgraded version of Parse.

I've looked through the release notes in the parse-server Release section for anything that could relate to this, indicating a major memory issue had been solved. I've combed through issues and pull requests that have been closed. We use enableSingleSchemaCache: true already, so that's not an issue here.

Environment Setup

• Server

  • parse-server version (Be specific! Don't say 'latest'.) : 2.8.4 vs. 2.5.0
  • Operating System: 64 bit Amazon Linux/4.5.0
  • Hardware: M5.Large instance
  • Localhost or remote server? (AWS, Heroku, Azure, Digital Ocean, etc): AWS

• Database

  • MongoDB version: 3.2.16 WiredTiger (M7 Standard)
  • Storage engine: Not sure
  • Hardware: 700 gb ssd block storage (general purpose abs encrypted)
  • Localhost or remote server? (AWS, mLab, ObjectRocket, Digital Ocean, etc): mlab on AWS

[flovilmart]

Please Upgrade to 3.1.0. It is possible 2.8.4 has a bug that was resolved with susequent releases

[zackshapiro]

Thanks @flovilmart. Before I invest the time upgrading our codebase, do you know of specific bugs that were fixed between 2.8.4 and 3.0 that would cause the memory use reduction? I specifically didn’t upgrade past 2.8.4 because of the breaking changes so I’d love just a little more info, even if just bullet points about what specific bug fixes caused that great of a memory reduction.

Have a great weekend

[flovilmart]

I don’t have any specific on the top of my head. Enabling Single Schema cache can reduce the memory pressure.

Lastly, you should upgrade beyond 2.8.4. The upgrade path should be straightforward as we published guides both for server and the JS SDK.

[zackshapiro]

Enable single scheme cache is already true. I understand there are guides and I’ve reviewed them. My app is tens of thousands of lines of code and I need to better understand what memory improvements were made beyond 2.8.4 to justify the investment of time and resources.

Surely improvements that drop memory 40+% are named somewhere?

[flovilmart]

Can you provide a profile / heap dump of before / after so you’ll be able to identify the objects and categories that are created in excess?

Most of the memory the server will use is related to the live caches of users and roles, a big chunk was refacored in the latest release and leads to dramatic performance improvements in 3.1.0 that adds support for CLP in live query as well.

Also you are comparing with versions like 2.5.0 which was released more than a year ago and is 472 commits behind master, to 2.8.4 which is already about 150 commits behind. I cannot stress enough the importance to keep your dependencies up to date.

[flovilmart]

See: #4387 (comment)