Massive storage usage

[dignifiedquire]

We have been running this for a few weeks and there seems to be a problem in storage usage:

rs0:PRIMARY> db.sharejs.count()
17595
rs0:PRIMARY> db.sharejs_ops.count()
1501573

We know there is a lot of data going through sharejs but we need some kind of way of cleaning that up as otherwise the storage just grows and grows.
Any ideas or suggestions on what to do?

[josephg]

Well, the ops are only needed if a client reconnects at an old version. For example, if I have a browser tab open, then go offline for 2 weeks on holiday then reconnect, I'll want all the ops that have happened in the last 2 weeks. Pick a time (say, 2 weeks) and take all the ops generated prior to those two weeks offline.

If you're staying in mongo, you could use the timestamp (.ts) property in the database to find the ops you want to remove and cull them. (I don't think mongo has a TTL property, but I could be wrong - that would be better still). There's no tooling around this, but it shouldn't be hard to make something yourself.

Or you can put the ops somewhere else (your choice where). You only have to implement 3 functions for your operation storage:

MyDb.prototype.writeOp = function(cName, docName, opData, callback) {...};
MyDb.prototype.getVersion = function(cName, docName, callback) {...};
MyDb.prototype.getOps = function(cName, docName, start, end, callback) {...};

Documentation for the methods & tests are in livedb.

[dignifiedquire]

Okay thanks. I just wanted to be sure that there is no other harm in removing them.

[josephg]

Yeah it should be ok. You might want to check the error paths if ops don't exist and make sure the client responds accordingly.

[chaporgin]

Mongodb has TTL indexed collections starting from some 2.x version. It might be a good idea to call _ops.ensureIndex( { "ts": 1 }, { expireAfterSeconds: 100500 } ) from livedb-mongo on op collection on application start.

[josephg]

Not everyone wants that (we certainly don't want to delete any of our ops -
but we have far fewer of them than you do).

Just call it on the collection outside of livedb-mongo.

[dignifiedquire]

Okay so the mongodb side of things seems clear to me. But what about the redis part? Should I just clean that at the same point? What will happen if I don't/things are not entirely in sync?

[josephg]

Everything in redis has a TTL set. It should get cleaned up automatically.
I think the default TTL is 1 week. You can clear it yourself if you want to

• its sort of a cache. You may get obscure errors if you manually delete
  data in redis while writes are happening. - I haven't tested that
  thoroughly.