Migration path

[retorquere]

[x] Documentation issue or request

Would it be possible to offer in-place upgrade from lokijs 1 serialization format to 2 in loadDatabase? I don't mind doing it myself as long as there would be a clear transformation I could do inside my own load handler.

[Viatorus]

Yes, this could be possible.

Maybe a hook on the function loadJSONObject https://github.com/LokiJS-Forge/LokiJS2/blob/13a3d4756eb773f6c0d3bd25c6310022fbe45712/packages/loki/src/loki.ts#L727 could do it.

But at the moment, the API is not final so there can be still changes.

[Viatorus]

@obeliskos
I refactoring the persistence logic of LokiDB to support LokiJS databases and also prepare for future database changes with an easy but typesafe mechanism (more about this after my vacation [4 weeks]).

LokiJS' loadDatabase currently supports Loki objects and Loki serialized JSON objects
Is it really necessary to support Loki objects in the future? Dropping this feature would it make simpler.

Note:

• In LokiJS: Loki object and loki serialized object share the same properties.
• In LokiDB: This isn't the fact.

[obeliskos]

Interesting, looking forward to hearing your ideas on that... after you take what i'm sure is a much needed vacation.

So, just to clarify, is this :

• only a problem for supporting legacy lokijs file formats?
• only for transparently supporting legacy lokijs format via same load codepath as new database format
• an issue even not considering legacy lokijs file format?

I believe in LokiJS loadDatabase we allow inflating from objects for reference adapters vs basic adapters. This (loadJSONObject) method is also being used elsewhere for cloning. If we need to rework that interface I'm sure we can come up with something or if we want a different 'import' code path which applied upconverting format that might be an option as well.

On a similar but somewhat unrelated note, I have been considering taking the iterator/generator approach used in LokiFsStructuredAdapter and bringing similar interface internal to LokiDB to allow 'streaming' serialization. That may take the form of a (third?) adapter interface if it can be done cleanly.

[Viatorus]

Hey @obeliskos,
My initial issue is fixed - thank you for your clarification. ;)

So here is the current design concept/implementation:

An user can now migrate his database from LokiJS to LokiDB and of course, we can also support migration from different LokiDB versions to the current one, in a type safe way.

This works the following way:
The file v1_5.ts contains the current type structure of LokiJS serialization.
The file v2_0.ts contains the current type structure of LokiDB serialization.

With the help of a fancy conditional type we can use the typescript compiler to show us unresolved differences (type errors). These can be solved with the mergeRightBiasedWithProxy function (see migration.ts and migration.spec.ts).

// used internal
return mergeRightBiasedWithProxy(obj,
{
    databaseVersion: 2.0 as 2.0,
    collections: obj.collections.map(coll => mergeRightBiasedWithProxy(coll, {
    dynamicViews: coll.DynamicViews.map(dv => mergeRightBiasedWithProxy(dv, {
        persistent: dv.options.persistent,
        sortPriority: dv.options.sortPriority,
        minRebuildInterval: dv.options.minRebuildInterval,
        resultSet: mergeRightBiasedWithProxy(dv.resultset, {
        filteredRows: dv.resultset.filteredrows,
        scoring: null
        }),
        sortByScoring: false,
        sortCriteriaSimple: {
        field: dv.sortCriteriaSimple.propname
        },
    })),
    cloneMethod: migrateCloneMethod(coll.cloneMethod),
    transforms: coll.transforms as any as Dict<V2_0.Transform[]>, // TODO not accurate
    nestedProperties: [],
    ttl: undefined,
    ttlInterval: undefined,
    fullTextSearch: null,
    }))
});
}

Because of existing feature differences between LokiJS and LokiDB (nestedProperties, FullTextSearch...) and possible other differences in future versions of LokiDB an user has now the possibility to add a callback function. This functions gets called for each collection just before deserialization. The function has three parameters: original database version, migrated database object and the collection options (like the options used by addCollection(...)). So the user can either alter the database object before deserialization and return with false or adjust the collection options and return with true. In the second case, a brand new collection with the defined options will be created and the collection data will be inserted using standard Collection.insert.

db.loadJSONObject(legacyDB as Serialization.Serialized, {
    migrate: (_1, coll) => {
        coll.nestedProperties = [{name: "d.msg", path: ["d", "msg"]}];
        return false;
    }
});

db.loadJSONObject(legacyDB as Serialization.Serialized, {
    migrate: (_1, _2, options) => {
        options.nestedProperties = ["d.msg"];
        return true;
    }
});

What do you think? Is it understandable for you?

---

Still todos:

• tidy up type structure of v1.5
• finalize type structure/naming of v2.0
• add better type safety for transform

[obeliskos]

Wow, interesting... that's more type safety than I expected for serialization. Might take me longer than a cursory glance to fully understand. So as we make changes to lokidb serialized structure we will need to update the v2_0.ts interfaces and possibly the migration.ts (or is that migrate function handled within loki.ts)?

[Viatorus]

If we change the lokidb serialized structure we have to copy and rename v2_0.ts to v2_1.ts (e.g) and adjust the type structure. After that, migration.ts must be extended with a migration function v2_0 to v2_1 (possibly smaller and simpler than v1_5 to v2_0).

[retorquere]

Is there example code to use this migration? I have a lokijs DB that saves/loads using a custom reference adapter.

[Viatorus]

Hey @retorquere,
there exists a migration path but only as WIP branch. Please be patient.
LokiDBs serialization is not final and should not be use in production yet.

[obeliskos]

I'll be adding a LokiCompatibilityOperatorPackage which I will try to make as close as possible to the LokiJS operator package. When migrating databases from LokiJS, we can assign that as the default operator package for the imported collections.

I'll probably also look into adding query docs and an optional parameter to find to allow overriding the operator package to use.

[Viatorus]

@obeliskos
Good point! Thank you.

What do you think about using a key/value store (object, map) instead of an array for collection data? ;)