Event Driven Architecture Workshop

About OSO DevOps

We are a DevOps and Amazon Web Services (AWS) Professional Services Consultancy based in London. We build secure bespoke automated big data platforms on AWS using open source tooling, helping businesses to extract value from their data in the most cost effective and agile way. We provide self service tooling to enable business workflows to make data driven decisions.

Our Approach To The Workshop

To help you understand and harness the power of event driven architectures, we propose to deliver a 2 day workshop to cover the architecture, data model and roadmap for the secure deployment of Kafka. The workshop is not purely a technical exercise. It will cover people and process aspects of how Kafka can be consumed at scale throughout the organisation. Documentation is a large part of the workshop and will be delivered after the engagement.

Following the workshop, recommendations will be provided, adding depth to the findings where requested by senior stakeholders. By applying OSO DevOps proven agile methodology, the overall risk will be reduced to a clear set of deliverables, drawing on our high performing team of experts.

Agenda:

Day 1
Time	Topic	Output
09:30 - 10:00	Introduction to Apache Kafka and event-driven architectures.
10:00 - 12:00	Capture current architecture practices, standards: Stakeholders and their requirements FOSS v Commercial v Cloud Strawman architecture of client ecosystem (including: choreography / orchestration/workflows) Technology stack (frameworks, data formats) Development processes (source repo, build pipelines, automation, self-service) Regulations / compliance / reporting (PCI, GDPR, etc) Security standards and architecture / implementation (LDAP, OAuth, SSO, Vault etc) Tooling (alerting / monitoring / logging / orchestration) DevOps, operations, practices tooling WRT: run-books, No-ops, infra-as-code upgrades, 24x7, uptime, release processes & support processes. Infrastructure (cloud provider / DC / regions / OS) Future direction and timeline	Captured line items, weighted in terms of relevance to following activities. A list of assumptions together with priorities and timeframes.
12:00 - 12:45	Lunch break.
12:45 - 15:00	Enumerate Kafka ecosystems pros / cons and features required. Detailed explanation of schema-registry, replication options MM2 v replicator, data-balancing, over-partitioning, retention, transactionality, REST proxy, security options. SWOT Confluent enterprise, community & cloud features. SWOT AWS MSK, Kafka OSS & complementary tools (monitoring, alerting, management) Score SWOT outputs on factors including (L/M/H) Suitability for dev and/or prod Cost Security and encryption Support (i.e. skills and tooling for self-support vs cloud support etc) Operations Current business appetite Fit strategic direction (i.e. cloud, central-nervous-system, security, hybrid cloud, multi-cloud)	Capture scoring and the qualified recommended solution (and gaps) for production and pre-prod. Implement a high level plan proposal / steps / timelines with dependency tracking.
15:00 - 15:30	Break.
15:30 - 16:30	Identify high level requirements: Service availability, regions and SLA commitments. Current production runbooks / deployment pipelines. Development / testing environment requirements. CI / CD tooling requirements. Key management and certificate requirements. Data classification and schematic versioning	High level strawman deployment architecture for all none-production and production environments

Day 2
Time	Topic	Output
09:30 - 10:00	Review of Day 1, the outputs and assumptions captured.
10:00 - 10:30	Educational session on Kafka in production: Key architectural principles to adhere to. A blueprint of common architectural patterns. How to scale Kafka in production. Concepts of Kafka Connect. KSQL vs Kafka Streams. Self service tooling and Kafka ecosystem of connectors.
10:30 - 11:00	Education session on DevOps in distributed architecture: How to build and deploy high performance, fault tolerant systems. How to measuring DevOps success with four key metrics: Lead time for changes Deployment frequency Time to restore service Change failure rate DevOps tooling (automation / monitoring / alerting / reporting / metrics. DevOps tooling.
11:00 - 12:00	Capture operational infrastructure and processes: Monitoring tools and metrics (incl self hosted and others). Discuss monitoring / alerting options. How to handle dead-letter queues. Audit processes. Logging and log retention policy. Support and out of hours issue resolution Patching and vulnerability scanning.	Map onto the recommended Kafka ecosystem. Recommended operational tooling implementation plan proposal / steps / timeline.
12:00 - 12:45	Lunch break.
12:45 - 13:15	Identify and scoping development requirements: Development environment, developer onboarding and tooling requirements. Source control policy and tooling. CI / CD deployment and pipelines. Cost analysis on anything new. Skills matrix of existing development team.	Strawman development architecture and tooling. Recommended operational tooling & implementation plan proposal / steps / timeline.
13:15 - 15:45	Use case modeling: Capture the first use-case as a dataflow including key distribution, partition assignment, data flow points, cost of failure, choreography. Data model and evolution including serialization format / compatibility / evolution. Validate requirements (correctness, side effects). Upgrade paths and strategy. Future / envisaged use-case states. Workload characteristics (historic, test, live, etc) with respect to: Throughput. Latency Volume Retention Resilience	Capture and document findings. Provide high-level data-flow model. Recommended deployment and implementation plan proposal / steps / timeline for testing and productionisation.
15:45 - 16:00	Break.
16:00 - 17:00	Initial planning activity dependency mapping and feedback: High level statements of work. Roadmap of dependencies. Next steps. Session feedback.	Dependency flow of activities broken down as work package activities

Resourcing and Logistics

OSO DevOps can typically initiate a full workshop team within 1-2 weeks, but can often mobilise teams faster. OSO DevOps will select the following resources for this project:

Neil Avery has over 25 years of commercial software development. Recently he worked at the Office of the CTO within Confluent to help shape the future direction of Kafka and Streaming in general. He has been instrumental in developing KSQL and Kafka Streams as well as helping large organisations understand how Kafka is adopted at scale. His blog series Journey to event driven has helped shape mindset and understanding of what Kafka adoption means. Recently, authoring the Kafka Transport binding for CloudEvents 1.0 and also has a keen interest in Cloud, serverless and the next generation of technology (quarkus).

Sion Smith has over 18 years of commercial software development experience and twelve years experience with Java and the Spring Framework. His focus and experience are predominantly in the financial services sector, gaining expert knowledge of the Spring Boot and Kafka framework. Experienced in event-driven programming principles, using a mixture of open source and Confluent Kafka.

Next Steps

The 2 day workshop provides a strategic advantage and increases the successful adoption of Kafka at scale. We tailor each workshop to the needs of the client, in our experience, 90% of organisations participating in the workshop proceed with the recommendations.

Have any questions? Require a sample of the workshop outputs? Email us enquiries@osodevops.io or schedule a call