DistributedCache

Overview

DistributedCache is a database abstraction layer (DAL) that allows data (containers) to be cached, even when multiple instances of the application are running on multiple hosts. It uses another DAL as a source for populating the cache and making updates.

Based on Akka Clusters 2.5.x.

Warning: This component relies on the underlying data source to actually synchronize/replicate data between the database nodes/instances; the DistributedCache does NOT transfer any data.

More info on synchronizing/replication of the supported databases:

• CouchDB 1.6.x - replication
• Elasticsearch 5.3.x - clustering
• Redis 3.2.x - replication
• MariaDB 10.0 - replication

Basics

Initialization

1. Creates local cluster node
2. Creates the cache store
3. Creates the cluster messenger
4. Joins an existing cluster or creates a new one (depending on config)

Operation

Queries

All queries are satisfied either from the local cache directly (if no containers were evicted) or the missing containers are retrieved from the local source and returned along with the already cached containers.

Updates (create, update, delete)

The local cache and the source are updated with the new data, and the cluster is notified of the change. Each cluster node will then attempt to retrieve the changes from its own local source.

Note: It is up to the source databases on each cluster node to synchronize/replicate the data.

Warning: There is no guarantee that after a successful update on one node queries on other nodes will reflect the changes that have been made. There could be delays in the source's synchronization/replication and in the cluster message processing.

Cache Store

This is the component that does the actual local caching and ensures that the correct versions of the data are maintained in the cache. On cluster updates, several queries may be made (over time) to the source DAL while attempting to get back in sync.

Cluster Messenger

This component is responsible for sending and receiving cluster messages.

On each update, a message is sent to the cluster notifying all nodes that a change was made. This notification includes a message ID and the details of the object/container that has changed: containerType, objectID, revision and revisionNumber.

Periodically, each node broadcasts its last message ID to ensure that no messages were lost. Should a remote node find that it is behind that (message IDs are sequential), a full cache reload is triggered.

TLS support for cluster communication is enabled via Netty config. Actual settings can be found in Akka's Remoting docs.

Configuration

• A single node cluster can be created by assigning the same local and cluster ports.
• The first node to start up in a multi-node cluster must be configured to bind to itself (in the same way as a single-node cluster configuration).
• The initial message synchronization between cluster nodes is configured based on the specified syncInterval plus a random number (0-5) of seconds, to try to avoid multiple synchronizations happening at the same time.
• If preload is set to true, the actual preload will be started after a delay, based on actionTimeout.
• The Akka config option akka.remote.netty.tcp.port is always overridden by the parameter localPort.
• The Akka config option akka.actor.provider is always overridden by the parameter actorProvider.

server.static {

  ...

  database {
    requestTimeout = 5 //in seconds

    ...

    distributed-cache {
      clusterHost = "<some host>"
      clusterPort = 0
      localPort = 0
      preload = true
      actionTimeout = 5 //in seconds
      containerTypeMaxCacheSize = 1000
      syncInterval = 5 //in seconds
      maxLoadAttempts = 5
      actorProvider = "akka.cluster.ClusterActorRefProvider"
    }

    ...

  }

  ...

}