In memory Log journal

[datasatanic]

Is there any way to implement in memory log journal with ASP Di?

[sakno]

Hi @datasatanic , it doesn't make sense. The log must be persistent to provide recovery after failure of instance. In-memory log will work if at least one instance remain alive. However, if all nodes crashed then the entire cluster will not be able to recover its data. There is ConsensusOnlyState WAL type but it's suitable for cases when you need consensus only. It's operating in-memory only. However, it doesn't store log entries (only their metadata).

If you care about performance, then use PersistentState.AppendAsync<TEntry>(TEntry entry, bool addToCache, CancellationToken token) overload and pass true to addToCache parameter. In this case all uncommitted log entries will be added to in-memory cache instead of disk.

Additionally, you can write your own WAL via implementation of DotNext.Net.Cluster.Consensus.Raft.IPersistentState interface.

[datasatanic]

Thanks for answer. We are creating a system with microservice architecture in docker containers with control via Nomad.
So there is no need to use file storage and it is significantly reduces the speed in my opinion. We needed only replication and log compression mechanisms, and i think the probability of whole set of nodes failed is very low.

1. I'm using PersistentState.AppendAndEnsureCommitAsync on each comming request, with AddToCache=true, but I don't see much performance gain.
2. If i implement IPersistentState how it bind with Hosting implementation. In services.UsePersistenceEngine or in inheritor of IClusterMemberLifetime?

[sakno]

Check #57 first, there is a good discussion about WAL performance. You need to enable caching with Options.UseCaching and tune I/O buffer size (Options.BufferSize and Options.SnapshotBufferSize). On my SSD, the performance is about 250 microseconds per write (when addToCache is enabled).

In case of your own implementation, you need to register WAL instance as a singleton service of type IPersistentState in DI. But if you really want to use in-memory WAL just setup ramfs for your container. In this case, your RAM will act as file system for WAL.