New Data Layer

[maxnowack]

As already discussed some time ago on discord, Here are my ideas and plans for the improved data handling for SignalDB. Keep in mind that this is a draft and not final. I'm open for feedback and suggestions. Feel free to add your comments and to challenge the design.

Currently all data is stored in memory as well as in the storage of the persistence adapters. This is not optimal, as the data is stored twice and the memory usage is quite high.
Another issue with this is, that all operation take place in the main thread as the memory is accessed from here.

My plan is to move to add a separate layer that handles the data storage, querying and write operations. This layer can be moved to a separate thread or web worker to not block the main thread. This data layer will "talk" to the connected collection in the mean threads via events and also update the data in the collection.

The data in the collection itself is just a cached representation of the data in the data layer. The collection "registeres" a query, once it's used in a reactive context. The query result then will be empty in the first place, but will trigger the data layer to fetch the data from the storage and update the collection with the result. The data will be then cached in the collection and the query will rerun and present the result. There will be also a way to get the current loading state of a query, so that the UI can show a loading indicator. If the reactive context is terminated, the query will be “unregistered” and the cache will be cleared (maybe with a configurable grace period).

The collection class will be trimmed down to a minimum. The most logic will be moved to the data layer. The collection only acts as a proxy to the data layer. It doesn’t store any data and also does not contain any complex logic. The API of the collection stays the same as current. But calling methods will then communicate with the data layer. So the collection act more as a reactivity proxy for the data layer.

The memoryAdapter of the collection will be ditched as it's functionality will be obsolete or merged into the data layer.

The data layer will be collection agnostic and can act as the backend for many collection instances.

Usage

The API how collections are instanciated will slightly change. It's also thinkable to develop the whole thing backwards compatible, but I'm unsure if it's worth it.
The data layer will be created with a factory function that takes the storage adapter as an argument. The storage adapter will be created with a factory function that takes the storage name as an argument. This way the storage adapter can be used for multiple collection instances.
The data layer will be passed to the collection constructor. The collection will then be connected to the data layer and will be able to communicate with it.

import { Collection, createDataLayer } from '@signaldb/core';
import createIndexedDBAdapter from '@signaldb/indexeddb'
import maverickjsReactivityAdapter from '@signaldb/maverickjs'

const dataLayer = createDataLayer({
  storage: name => createIndexedDBAdapter(name),
});

const collection = new Collection('posts', dataLayer, {
  reactivity: maverickjsReactivityAdapter,
});

Implementation

Data Layer

interface DataLayerOptions {
  // factory function to create a persistence adapter for a collection
  storage: (name: string) => PersistenceAdapter,
}

interface DataLayer {
  createCollectionInterface(name: string): CollectionInterface,
}

interface CollectionInterfaceEvents {
  // events that will be emitted when the loading state of a query changes.
  'query-state-change': { selector: Selector, options: QueryOptions, state: 'active' | 'complete' | 'error' },
}

interface CollectionInterface extends EventEmitter<CollectionInterfaceEvents> {
  // CRUD operations will be proxied from the collection to the collection interface of the data layer. The CRUD logic itself will be done inside of the data layer.
  insert(data: any): void,
  updateOne(selector: Selector, modifier: Modifier): void,
  updateMany(selector: Selector, modifier: Modifier): void,
  removeOne(selector: Selector): void,
  removeMany(selector: Selector): void,

  // methods for registering and unregistering queries that will be called from the collection during find/findOne
  registerQuery(selector: Selector, options: QueryOptions): void,
  unregisterQuery(selector: Selector, options: QueryOptions): void,
  getQueryState(selector: Selector, options: QueryOptions): 'active' | 'complete' | 'error',
  getQueryResult(selector: Selector, options: QueryOptions): null | any[],
}

export default function createDataLayer(options: DataLayerOptions): DataLayer

The main benefits of this design are a reduced memory footprint as well as that are more operations not blocking the UI thread anymore (if a web worker is used)

Open questions:

• Indices are currently initialized with the collection. With the new approach, the data layer should be responsible for the indices. How should the API look like create and manage indices?

Connected issues/discussions: #397, #998

[DankTechnologies]

This layer can be moved to a separate thread or web worker to not block the main thread

Dumb question, but I'm trying to square this limitation with the web worker compat notes for Svelte which imply collections work on the main thread and in web workers. What are the current limitations today in 1.x w.r.t. querying and modifying the collections off the main thread?

The collection "registeres" a query, once it's used in a reactive context.
If the reactive context is terminated, the query will be “unregistered” and the cache will be cleared (maybe with a configurable grace period)

Is the idea that the reactive context would be set up on a per component basis? With Svelte, I've moved to a model with the reactive contexts consolidated in one place that initializes at startup, using $effect.root and exporting functions that return the reactive variables. Components then import and use $derived to further query the reactive variables.

db.svelte.ts

let feeds = $state.raw<Feed[]>([]);
let episodes = $state.raw<Episode[]>([]);

$effect.root(() => {
	$effect(() => {
		const feedsCursor = db.feeds.find();
		feeds = feedsCursor.fetch();

		return () => {
			feedsCursor.cleanup();
		};
	});

	$effect(() => {
		const episodesCursor = db.episodes.find();
		episodes = episodesCursor.fetch();

		return () => {
			episodesCursor.cleanup();
		};
	});

function getFeeds() {
	return feeds;
}

function getEpisodes() {
	return episodes;
}

export { getFeeds, getEpisodes };

component

	import { getEpisodes, getFeeds } from '$lib/stores/db.svelte';

	let episodes = $derived(
		getEpisodes()
			.filter((episode) => episode.feedId === feedId)
			.sort((a, b) => b.publishedAt.getTime() - a.publishedAt.getTime())
			.slice(0, limit)
	);

	let feed = $derived(getFeeds().find((feed) => feed.id === feedId));

So far so good, even with ~12K rows in the episodes collection. Having this global state management is nice w.r.t. keeping components simple, but curious for your thoughts on this approach in general and in context of the data layer idea.

Thanks!

[maxnowack]

Dumb question, but I'm trying to square this limitation with the web worker compat notes for Svelte which imply collections work on the main thread and in web workers.

The idea is to keep the Collection with all reactivity and stuff in the main thread, while outsourcing the data handling to a web worker. Reactivity will stay as it is with no additional limitations compared to today.

What are the current limitations today in 1.x w.r.t. querying and modifying the collections off the main thread?

If you run SignalDB in a worker context today, it creates entirely new collection instances that communicate with the same storage backend. This behavior will remain the same with the redesign.
In the current version, all data must be kept in memory, so using a worker effectively doubles the memory usage. In the redesigned approach, collections won't store any data themselves. Instead, the data layer will handle caching, specifically for reactive contexts.
Since there is no reactive context outside the main thread (or at least there shouldn't be), the data layer in the worker can directly query the storage without needing to cache anything.

Is the idea that the reactive context would be set up on a per component basis? With Svelte, I've moved to a model with the reactive contexts consolidated in one place that initializes at startup, using $effect.root and exporting functions that return the reactive variables. Components then import and use $derived to further query the reactive variables.

It depends. The idea is to cache data only while a reactive context is active, and to remove the cached data once that context is terminated. In component terms, you could think of it as caching on mount and clearing the cache on unmount.
If you move your queries into a global reactive context, the queried data would remain cached indefinitely. This might be a deliberate and valid design choice – especially if your application consistently needs access to that data. However, for all other data, keeping it cached globally could introduce unnecessary overhead, particularly in terms of memory usage. In such cases, it is better to use dedicated reactive contexts at the component level. This is also my recommendation for the current version of SignalDB.