Current functionality and the future (hoping on support for all use cases)

[ralphtheninja]

Just wanted to jot down how it actually works but with words. Mostly for my own processing but also to explain to others that might be interested.

What I like the most with this module, is that it allows you to create sub levels (duh) but also with their specific encodings and it does this quite cleverly by peeling off the levelup layer, adding one down layer (SubDb) on top of the current down (leveldown, memdown etc wrapped by levelup) and then finishes off by adding back a levelup layer.

This trickery happens in ._open() (with some added comments):

SubDown.prototype._open = function (opts, cb) {
  var self = this

  if (this.db.isOpen()) {
    if (this.db.db.type === 'subdown' && this.db.db.prefix) {
      // This happens when we do a nested sub level
      // this.db is a levelup and this.db.db is the SubDown and
      // this.db.db.leveldown is the original down (see else case below)
      this.prefix = this.db.db.prefix + this.prefix
      this.leveldown = this.db.db.leveldown
    } else {
      // this.db is a levelup and this.db.db is the *down it's wrapping
      this.leveldown = this.db.db
    }
    return done()
  }

  this.db.on('open', this.open.bind(this, opts, done))

  function done (err) {
    if (err || !self._beforeOpen) return cb(err)
    self._beforeOpen(cb)
  }
}

The reason this works is because older versions of levelup takes care of the encodings and just applies them to a *down (SubDown in this case).

From index.js:

module.exports = function (db, prefix, opts) {
  if (typeof prefix === 'object' && !opts) return module.exports(db, null, prefix)
  if (!opts) opts = {}

  opts.db = function () {
    return subdown(db, prefix, opts)
  }

  return levelup(opts)
}

The problem we face now is that levelup was rewritten and encodings moved out into encoding-down so if we want to support encodings in the same way we can no longer rely on levelup alone.

So what if we in index.js finish off with:

module.exports = function (db, prefix, opts) {
  // ..
  return levelup(encoding(subdown(db, prefix, opts), opts), opts)
}

That should take care of the current functionality of levelup. But it's not enough. What if we want to create a sub level out of a level? This will not work (as @emilbayes pointed out here #7 (comment)) since it would mean double decodings (and encodings I guess).

So what I propose is that we continue with this trickery a bit and tweak ._open() to peel off two layers if there's an encoding-down:

SubDown.prototype._open = function (opts, cb) {
  var self = this

  if (this.db.isOpen()) {
    if (this.db.db.type === 'subdown' && this.db.db.prefix) {
      this.prefix = this.db.db.prefix + this.prefix
      this.leveldown = this.db.db.leveldown
    } else if (encDown.isEncodingDown(this.db.db)) { // <-- HERE!
      this.leveldown = this.db.db.db
    } else {
      this.leveldown = this.db.db
    }
    return done()
  }

  this.db.on('open', this.open.bind(this, opts, done))

  function done (err) {
    if (err || !self._beforeOpen) return cb(err)
    self._beforeOpen(cb)
  }
}

It's a bit hacky, but I think it should work well. We need to implement a proper static encDown.isEncodingDown() function though which should support cross realms (using Symbol.for() etc).

[vweevers]

Amazing write-up!

We need to implement a proper static encDown.isEncodingDown() function though which should support cross realms (using Symbol.for() etc).

I think it'd suffice to do it the same as the browserified Buffer.isBuffer():

encDown.prototype._isEncodingDown = true
encDown.isEncodingDown = function (db) {
  return db != null && db._isEncodingDown === true &&
    db !== encDown.prototype // so encDown.isEncodingDown(encDown.prototype) will be false
}

However, I wonder if there's isn't a more generic solution. There are similar use cases for wanting to get the "inner db", like getting access to .iterator() which is not exposed by levelup. If you want to get to leveldown's iterator for its iterator.seek() method, you possibly have to bypass deferred-leveldown and encoding-down.

So what if both levelup and abstract-leveldown had a innerDown method, that punches through the layers to get the innermost db?

levelup would return this.db, abstract-leveldown would default to returning this, deferred-leveldown would return this._db, subleveldown would return this.leveldown.

[vweevers]

a innerDown method

Or down() :)

[vweevers]

Hmm down() would also need an optional type argument that tells it where to stop traversing:

AbstractLevelDOWN.prototype.down = function (type) {
   var db = this._down(type)
   return db == null ? null : db === this ? db : db.down(type)
}

AbstractLevelDOWN.prototype._down = function (type) {
  return type ? null : this
}

SubDown.prototype._down = function (type) {
  return type === 'subleveldown' ? this : this.leveldown
}

SubDown.prototype._open = function (opts, cb) {
  if (this.db.isOpen()) {
    var subdb = this.db.down('subleveldown')

    if (subdb && subdb.prefix) {
      this.prefix = subdb.prefix + this.prefix
      this.leveldown = subdb.down()
    } else {
      this.leveldown = this.db.down()
    }
  }

  ..
}

[ralphtheninja]

However, I wonder if there's isn't a more generic solution. There are similar use cases for wanting to get the "inner db", like getting access to .iterator() which is not exposed by levelup. If you want to get to leveldown's iterator for its iterator.seek() method, you possibly have to bypass deferred-leveldown and encoding-down.

So what if both levelup and abstract-leveldown had a innerDown method, that punches through the layers to get the innermost db?

levelup would return this.db, abstract-leveldown would default to returning this, deferred-leveldown would return this._db, subleveldown would return this.leveldown.

Great idea! This is also why I wanted to write this to get feedback on smart solutions. I just skimmed it right now, but will give it some more thought the coming day(s).

[ralphtheninja]

I'm wondering what we can do as minimal change to get this working, without having to rewrite the whole world, i.e. make a quick hack, then replace the hack underneath with a proper .down() solution.

[ralphtheninja]

@vweevers

var subdb = this.db.down('subleveldown')

That line calls levelup#down(). What would that implementation look like?

[vweevers]

That line calls levelup#down(). What would that implementation look like?

return this.db.down(type). It doesn't matter if this.db is deferred-leveldown or the open db, the result would be the same.

I'm wondering what we can do as minimal change to get this working, without having to rewrite the whole world

I was thinking about that too. We might be able to get by with while (db.db) db = db.db with some additional check that db.db is an abstract-leveldown (and not an underlying IndexedDB, for example).

[ralphtheninja]

I was thinking about that too. We might be able to get by with while (db.db) db = db.db with some additional check that db.db is an abstract-leveldown (and not an underlying IndexedDB, for example).

Sounds good. I have something to play with now at least!

[vweevers]

More complete plan for a temporary solution:

1. Rename subdown#leveldown to subdown#db. #db is already levelup, that works
2. Set the value of subdown#type to subleveldown
3. Write a recursive utility that we can use as fallback in subleveldown, levelup ánd abstract-leveldown:

function down (db, type) {
  if (typeof db.down === 'function') {
    return db.down(type)
  }

  if (type && db.type === type) return db
  if (isAbstract(db.db)) return down(db.db, type) // encdown, subleveldown, levelup
  if (isAbstract(db._db)) return down(db._db, type) // deferred-leveldown

  return type ? null : db
}

For isAbstract, perhaps we can use the old is-leveldown.js code.

[ralphtheninja]

So basically do

SubDown.prototype._open = function (opts, cb) {
  if (this.db.isOpen()) {
    var subdb = down(this.db, 'subleveldown')

    if (subdb && subdb.prefix) {
      this.prefix = subdb.prefix + this.prefix
      this.leveldown = down(subdb)
    } else {
      this.leveldown = down(this.db)
    }
  }

  ..
}

[vweevers]

Exactly. And later:

LevelUP.prototype.down = function (type) {
  return down(this.db, type)
}

AbstractLevelDOWN.prototype.down = function (type) {
   var db = this._down(type)
   return db == null ? null : db === this ? db : down(db, type)
}

[ralphtheninja]

@vweevers Had to tweak the first subdb check to this.leveldown = down(subdb.db) since calling this.leveldown = down(subdb) will result back in subdb.

[vweevers]

I now think it's better to not add down() to levelup and/or abstract-leveldown, partly because of the recursive issue (#34 (comment)), partly because we don't need it in that many places, but mostly because we'd end up with multiple versions of the code.

Closing this, continuing at Level/community#82.