Advanced Data Architecture Patterns

HOME > [NFJS 2015](NFJS 2015) > [Advanced Data Architecture Patterns](Advanced Data Architecture Patterns)

Matt Stein

Just to reitterate: MICROSERVICES

###Losely Coupled Why can I deploy when ever I want?
Because I have CDC's that you are verifying you adhere to

###Bounded Contexts DB Per Service

• It creates a hard boundry between the services data. You physically enforce other services to use the APIs that were defined.
• Allows Polyglot Persistence (might not be truly possible...but it's supported)

###Case Study RedBox or FluffBox (his own clone)

Seems as though he always has an API that sits in front of all the services.

###Distributed Systems Failure and Latency make solving our problems harder

CAP Theorem

• Consistency - All nodes see the same data at the same time
• Availability - node failures do no prevent survivors from continuing to operate
• Partition tolerance - the system continues to operate despite message loss due to network and/or node failure

Only two can be satisified simultaneously

Three Options

#####CA:

Fallacy #1 of Distributed Systems: The Network is Reliable

Two Phase commits try to work this way.

This doesn't work

#####CP:

Strong Consistency. Nodes communicate and agree on every operation.

Partition Tolerant Consensus Algos: Paxos, Raft, Zab

If you think you need to build a CP system, You should read the papers referenced in his talk.

He doesn't think CP works either.

#####AP:

Eventual Consistency - It is important to know that you must ensure that the system WILL become consistent within some time.

These systems are easier to implement (not easy, but easier), so we will focusing here.

####Event-Driven Architectures You need a messaging arch that guarantees at least once delivery.

#####Solution #1: Raw Events

• Service does something, send event
• Other Services can consume that event and act on it.

These systems are:

• Easiest to understand
• Limited transaction support as long as you are using a DB that supports transactions.
• Spring Data rest can fire events

#####Solution 2: Event Sourcing/CQRS I can't type fast enough...look at slides.

Event Sourcing is like a write ahead log. But the problem with this is if you fail, you have to start from 0 and replay all the events.

CQRS: Uses an event store to store what happened and when. You can get subsets of those events to recreate a state.

What to use as an Event Store

• Kafka http://kafka.apache.org - his recommendation
• EventStore http://geteventstore.com
• Datomic http://datomic.com

App Frameworks:

• Axon - integrates with Spring
• Jdon

These don't seem very active...

He usually uses Kafka and Spring Stream Processing

The log allows you to fix failures/bugs. If you were writing buggy events, you don't fix history. You just append to it. Send update events that fix the problem.

#####Solution 3: Stream Processing Can adress the spanning query problem

Lambda Architecture - Nathan Marz

Tools:

• Storm
• Spark
• Samza
• Spring Cloud Streams and a Spring Data Flow

Summary

• Adopting Microservices introduces new data challenges
• Traditional transactional approaches don't work in distributed systems
• Understand CAP Thorem and its tradeoffs
• Understand and apply data architecture patters when they make sense for your context