Description
Issue Description
I want to preface my submission with the observation that there have been several reports in this repo about memory leaks / usage issues but all of them have been closed without a specific resolution. One was closed with "fixed with redis" which I didn't find very satisfying.
We've been struggling with ParseServer memory increases for the past month on a system that is quite simple.
We started with an existing Cloud Code implementation of a route which had one query and one save and was running very stable for years on Parse.com and then on NodeChef.
Recently, we added two more queries and an additional save to this route. When these extra queries and save are activated with a load of less than 1 request per second, we see the memory of our ParseServer instances on NodeChef grow by about 40KB per second. The processing load is not significant and the garbage collection runs relatively often but does not reclaim the memory which will continue to grow until we run out of memory in the containers. We've had to use memory limit rules to restart the containers to get around this.
Our code is pretty straightforward. We use the results of the previous queries when processing the subsequent queries so we use _.partial to pass the values along. We've also implemented this code as promises and as one big function previously in an attempt to reduce the memory consumption but it has remained relatively constant.
We've read other reports of similar behavior. One person reflected that its just an endemic problem and you need to rewrite from Parse to XYZ. Another said that the nature of V8 GC is that it falls behind and needs to catch up. Or that we have hidden globals, etc, etc. Or just restart your containers every so often (which is what we're doing).
Well our system is just not being stressed enough to realistically cause these problems and we've reviewed and reimplemented the code multiple times. And we started with a code base that did not grow memory like this.
The data sets being queried are also very small - on the order of a few hundred items in each collection. The JSON backup files for the data is under 100KB. (Before anyone suggests it, we could probably (and still might) rewrite the whole thing without Parse queries and instead store the data in JS arrays and objects but it would be a lot less maintainable).
Folks have also mentioned caching but again how do you eat up 140MB of memory caching 100KB of data?
Steps to reproduce
Essentially string together a series of queries which make use of the results of previous queries.
app.post('/points', auth, function(req, res) {
var busSerial = req.body.serial_number;
// FIRST QUERY
var query = new Parse.Query("BusData");
query.equalTo("serial_number", busSerial);
query.find({
success: _.partial(queryHandler, req, res),
error: function(error) {
res.send();
}
});
});
function queryHandler(req, res, results) {
// normal processing using results of first query...
// FIRST SAVE
bus.save(null, {
useMasterKey: true,
success: function(newBus) {
res.send();
},
error: function(newBus, error) {
res.send();
}
});
// SECOND QUERY
var query2 = new Parse.Query("RouteStops");
query2.equalTo('routeCode', busRoute);
query2.ascending('stopOrder');
query2.find({
success: _.partial(query2Handler, bus, point, busRoute),
error: function(error) {}
});
}
function query2Handler(bus, point, busRoute, results) {
// more processing using results of first and second queries...
// THIRD QUERY
var query3 = new Parse.Query("StopData");
query3.near("loc", point);
query3.containedIn("stop", stopNames);
query3.limit(5);
//console.log('POINTS: processArrivalStop geoquery.find');
query3.find({
success: _.partial(query3Handler, bus, point, busRoute, stopInfo),
error: function(error) {}
});
}
function query3Handler(bus, point, busRoute, stopInfo, results) {
// more processing using results of first, second and third queries...
// SECOND SAVE
bus.save(null, {
useMasterKey: true,
success: function(newBus) {},
error: function(newBus, error) {}
});
}
Expected Results
Flat memory usage
Actual Outcome
The memory of each container increases until the system can no longer process requests. The memory increase is directly proportional to the number of times this particular route is invoked.
We've narrowed the source of the problem down to the code in the route from the second query onwards. When we activate that part of the code, the memory increases and when we deactivate it, the memory usage becomes flat.
Environment Setup
-
Server
- parse-server version: observed with 2.2.7, 2.3.2, and 2.4.2
- Operating System: NodeChef
- Hardware: 3 512MB NodeChef containers
- Localhost or remote server? : NodeChef hosted
-
Database
- MongoDB version:
- Storage engine:
- Hardware:
- Localhost or remote server? NodeChef MongoDB
Logs/Trace
We've been using memwatch-next to monitor garbage collection. Occasionally there will be a big GC of 40+MB - but that might happen just once per day. Usually we just see a couple of megabytes at most and the growth outpaces what the GC is collecting.
We've had no abnormal reports in the logs in verbose mode.