Discussion: More efficient use of localstorage options

[yoshuawuyts]

Right now all entries into a Backbone collection get concatenated into a single string. A screenshot of how a collection (with models) is represented is attached here:

Although this might be efficient (you're writing a single string!) it's not very friendly to debug. In a server database it would be very common to split the models up so that they each have their own key.

Now without rushing things (this would take a while to implement), what are your thoughts on splitting collections and models up?

[magalhas]

There's something wrong with your ids in that screenshot but I guess you're aware of that.

I see some constraints on splitting things up, mainly ids must be unique even among different collections, else we'll need to namespace those keys (for example per collection storage id).

Not sure how other drivers besides localStorage work but I guess those already work with normalized tables.

CC @tofumatt

[tofumatt]

When I wrote the adapter it did The Simplest Thing Possible™, so there's probably room for improvement. Currently it just runs JSON.stringify() on the entire collection and saves that.

If there's a smart way we can split off models on their own I'm fine with that, but I'll admit my Backbone sync knowledge is rudimentary. Namespacing keys for collection + id is something I'm fine with, for example.

So basically: I'm for this idea. I think it will help with race conditions too (obviously our current code is prone to multiple .sync calls stepping on each other.

[magalhas]

Though there's also the chance that you want to use the same model among different collections. So probably a solution could be something like this (to support polymorphic collections):

var Collection = Backbone.Collection.extend({
  sync: Backbone.localforage.sync('collectionStorageId'),
  model: Backbone.Model.extend({
    sync: Backbone.localforage.sync('modelNamespaceId')
  })
});

Collections should be stored with just an array of modelNamespaceIds + modelIds conjunction.

So given the following example:

var Collection = Backbone.Collection.extend({
  sync: Backbone.localforage.sync('NewsCollection'),
  model: Backbone.Model.extend({
    sync: Backbone.localforage.sync('News')
  })
});
var collection = new Collection([{
  title: 'Foo'
}, {
  title: 'Bar'
}]);

The stored key/values would be something like:
NewsCollection / ['News1', 'News2']
News1 / {title: 'Foo'}
News2 / {title: 'Bar'}

[yoshuawuyts]

Wouldn't it make more sense to have a specific bucket for each type? I'll go bonkers on this one, but here's how I envision my localstorage looking:

Web SQL > localforage > newsCollections, newsModels
IndexedDB > localforage > newsCollections, newsModels
Local Storage > newsCollections, newsModels
Session Storage > newsCollections, newsModels

In more complex applications its quite common to have a collection > models > collections > models type of structure, so placing collections into their own bucket would make sense.

p.s. Yeah, my id's are totally broken. Running some experiments to see what's possible 8-)

[tofumatt]

That's the simplistic version I think, but putting each model in it's own key, then having the collections simply be a list of ids, which is what I think @magalhas is getting at, seems the most intuitive to me. It's also less likely to run up against race conditions, though I'm not sure how easy it makes filtering through collections based on things other than ID.

[magalhas]

Well I was thinking from an API approach. @yoshuawuyts can you elaborate a little bit further your suggestion? I think I'm not grasping what you mean.

[yoshuawuyts]

What I'm proposing is to split each model and collection into its own bucket. So if I got a newsCollection and a bunch of newsModels we'd create a bucket for newsCollections and one for newsModels. This provides a sane solution for more complex structures in which multiple instances of a collection can be stored.

I do get that it would be harder to implement since you no longer have the collection + models in it nicely stored together, making it way harder to "just fetch" things from localstore.

[magalhas]

What does it differ from what I've suggested?
NewsCollection key has [ 'News1', 'News2' ] -> collection bucket
News1 key has {title: 'Foo'} -> isolated model
News2 key has {title: 'Bar'} -> isolated model

[magalhas]

What do you mean by having a models bucket? Collection bucket = array of keys, models bucket = object map by id?

[yoshuawuyts]

Woops, you're right. Misread it yesterday, it was a bit late. 100% for your solution @magalhas.

[magalhas]

I'm with a good bucket of spare time so I guess I'm going to tackle this unless you're intending to @yoshuawuyts. Let me know.

[yoshuawuyts]

Hah, awesome! By all means, please do.

[magalhas]

I'm just wondering if "namespacing" the models is really needed. Maybe we could make it optional.

[tofumatt]

Namespacing them with keys like ‘ModelName/{id}’ makes sense to me, if that’s what you mean? I don’t like optional things; I say we make configuration unneeded and let the library do the work for us.