Tidy up src/seq fields and use a sparse index?

[alecgibson]

The o_XXX collection for holding ops has an index on {src: 1, seq: 1, v: 1}. Since this index points to every single op, it's pretty big. However, from what I can tell, its only use is to differentiate between slightly different errors when getting a document creation error, and it won't be needed for most of the time under "happy" conditions.

It seems pretty mad to have an entire index just for this corner case, but there we are. I was wondering if we could at least contain the index size by doing something like:

• Making the index sparse
• Adding a mechanism for clients to "tidy up", like a disconnection hook or similar. Given that src is a transient value meant to be unique per client, once a client disconnects we could potentially null all of the src fields on the ops they've touched in their session (could be expensive?), which would then release them from the index (I think? Not 100% sure this is how Mongo handles sparse indexes). Should note that clients now have a stable src between reconnects, so handling a "true" disconnect may be tricky?

I don't know if there's anything else we can do about this? Like change how ShareDB acts upstream?

[ericyhwang]

Things that we discussed:

• Adding a partial index on {d: 1, create: 1, v: 1} with {create: {$exists: true}} and using that to find all create ops, which in most cases is one per doc. Could be O(n) in the common case though
• Or {d:1, src: 1, create: 1} with a sparse index?
• Removing getCommittedOpVersion and instead scanning with d and v to find the create op. May need to scan the entire op history since create ops usually don't contain a v when submitted, since creates are usually done "blind" without fetching the doc by id first

We're going to punt on this for now, to let us stew it over a bit more

[alecgibson]

Marking this is a goal for v2, because we'd need a breaking change in the sharedb API.