Snapshot API

[juliangruber]

What about this API for snapshots

db#snapshot()

Create a new snapshot.

snapshot#get(key, value[, opts])

Read a single value from a snapshot.

snapshot#create{Read,Key,Value}Stream

Read multiple values from a snapshot.

snapshot#dispose()

Delete the snapshot.

Maybe also:

snapshot#{del,createWriteStream,...}

Just forwarded to db#*

Usage

For a consistend read stream this would be:

var snapshot() = db.snapshot();
snapshot.createReadStream()
  .pipe(...)
  .on('close', shapshot.dispose.bind(snapshot));

[rvagg]

nice creative thinking here, I was thinking more in terms of a token to be applied to the root db for read operations but this looks quite nice.

[juliangruber]

This makes it easy to do multiple operations on a snapshot but for just one operations it introduces some overhead, comparing the example from above to

db.createReadStream({ snapshot : true }).pipe(...)

[rvagg]

Yeah, but { snapshot : true } is unnecessary, every time you create a LevelDB iterator you create a snapshot that gets discarded when you finish, so you're using snapshots even though you don't see them. Snapshots in the sense we're talking about here are for longer-lived operations, where multiple read types (get & iterator) can reference the same point-in-time.

I was thinking more:

var snapshot = db.createSnapshot()
db.createReadStream({ snapshot : snapshot }).pipe(...)
snapshot.dispose()

[juliangruber]

Oh, true.

What I like about the syntax I proposed is that it

• extends no options objects (which should stay as small as possible)
• just swaps db with snapshot and all the APIs stay the same

[juliangruber]

Bad about my syntax:

Plugins that overwrite methods need to do it in 2 places, or just use hooks.

But, I think we need more hooks libraries anyways, as only they should do db.get = function() {...}. With them in place there's no problem here.

[rvagg]

ahh, perhaps we could do both approaches - i.e. hide the fact that it uses an options object on createReadStream() and get() to present your API. That way, plugins mostly won't need to worry since they will be routed back through the standard calls:

snapshot.createReadStream() === db.createReadStream({ snapshot: snapshot })
snapshot.get('foo', cb) === db.get('foo', { snapshot: snapshot }, cb)

We wouldn't even necessarily need to document the option.

[juliangruber]

Nice! Let's do it like this.

Does LevelDOWN expose snapshots already?

[rvagg]

there's a tiny bit of snapshot code in there, not much and it's not tested.

[mcollina]

Are you sure we should expose the del, put, batch and createWriteStreammethods in the snapshots? The snapshot will totally ignore their changes
The following example shows the problem:

var snap = db.snapshot();
snap.put("a", "b", function() {
  snap.get("a", function(err, result) {
    // unfortunately, the key will not exists on the snapshot
  });
  db.get("a", function(err, result) {
    // but it will exists here!!
  });
});

[juliangruber]

Yeah, that kinda sucks.

I was thinking to store new state in memory, so snap.get('a') would return a value from memory...But, no, that's going to cause problems.

On the other hand, it could be nice to do plugin(snapshot) instead of plugin(db) and it just works...Can't think of a real use case though...

The let's just do the reading methods for now.

[mcollina]

If you implement it with the {snapshot: snap } option, any plugin will work by attaching itself to the main db.

[juliangruber]

Hm true, any plugin that will need snapshots will just have to use them. I was thinking about a plugin that would read 2 values and store 1. If it didn't have to be consistent you could call it with plugin(db) but if it had do, it would be plugin(snap).

Anyways, there's no real use case for this. So,

• expose snapshots in LevelDOWN
• add snapshot option to all methods that read something
• add snapshot sugar with fluent api

[kesla]

I've been lagging a bit behind descussion about snapshots, so I'd like to know what the real use case for snapshots are to begin with? I'd say that we're in the risk of bloating levelup if we add snapshots only because we can...

[dominictarr]

I can't think of any uses for snapshots either, hmm, except providing a consistent view of data while you perform some computation -- but what is the point of consistent, if it's not the current data?

Anyway, on aesthetic grounds, I agree with @mcollina that snapshots shouldn't have any write methods,
also, if the db emitted a snapshot event, then plugins could listen on this, and apply themselves to the snapshot also.

But most importantly - what do we need snapshots for?

Oh, I remember one thing: I had some problems with createReadStream and then syncly putting data, and having it turn up in the read stream... i.e. a normal createReadStream should not return data that was inserted after it was created...

[rvagg]

ref #128 for that last problem, there's some discussion there that covers some relevant ground