Kafka Serialization Talk (WIP)

Table of Contents

• Chapter 01: Introduction
• Chapter 02: Model & Business Logic
• Chapter 03: JSON Serialization
• Chapter 04: Integrate JSON Serialization with Our Model
• Chapter 05: Integrate Business Logic with Kafka Streams
• Chapter 06: Our First main: A Complete Application
• Chapter 07: More Serialization Formats
• Chapter 08: Apache Avro
• Chapter 09: Confluent Variants
• Chapter 10: Benchmarks
• Chapter 11: Revised Serialization Implementations
• Chapter 12: Wrap up

Talk outline, work in progress.

Chapter 01: Introduction

• No code, only whiteboard

• Model the data

• API first (sort of)

Chapter 02: Model & Business Logic

• Implement model and business logic

• Independent of any other part of the program

bazel test //chapter02/...

Chapter 03: JSON Serialization

• Implement JSON API

• Independent of our model

• Can run integration tests with external sources

• Can produce sample data for external clients

bazel test //chapter03/...

Chapter 04: Integrate JSON Serialization with Our Model

• Glue code between external API and model (ports and adapters)

• Must be adapated for every change in API or model

• Is usually pretty simple

• Concrete serialization format does not show up in our test

bazel test //chapter04/...

Chapter 05: Integrate Business Logic with Kafka Streams

• Use our own embedded Kafka 3.0 Server, because we can

• Define a typesafe Kafka Streams topology

bazel test //chapter05/...

Chapter 06: Our First main: A Complete Application

• So far, we have been driven by tests

• Run a local Kafka server

bazel run //:kafka-sensors

Chapter 07: More Serialization Formats

• Protocol Buffers

• Apache Thrift

• Alternate JSON implementation: GSON

• Serializing the model with mix-ins, useful for prototyping

• Because we can: Amazon Ion

• Same pattern as in Chapters 03 & 04

bazel test //chapter07/...

Chapter 08: Apache Avro

• History: ApacheCon09: Avro

• Apache Hadoop, Apache Pig, Apache Hive

• Dynamic format

• Writer and reader schema required

bazel test //chapter08/...

Chapter 09: Confluent Variants

• History: AVRO-1124: RESTful service for holding schemas

• Incompatible with standard serializers

bazel test //chapter09/...

Chapter 10: Benchmarks

• Maybe not the most important part, but interesting

• Some implementations are slow

• Investigate

bazel run //:bench10

bazel run //:bench10 -- -p "format=avro-specific,avro-generic" \
  -prof "async:output=flamegraph;direction=forward" "Bench\\.serialize"

bazel run //:bench10 -- -p "format=gson,json" \
  -prof "async:output=flamegraph;direction=forward" "Bench\\.deserialize"

Chapter 11: Revised Serialization Implementations

• Make serializers faster

bazel test //chapter11/...
bazel run //:benchmark

Chapter 12: Wrap up

• Comparison of formats, what to use when

• Used techniques

  • Value objects

  • Hexagonal Architecture

  • Development driven by tests

• Decoupled development enables separate testing and benchmarking

• Early integration tests possible

• Slow implementations are not hidden in the business logic

• Separate input and business rules validation