Kafka module for Puppet

Table of Contents

1. Overview
2. Module Description - What the module does and why it is useful
3. Setup - The basics of getting started with Kafka
   • What Kafka affects
   • Setup requirements
   • Beginning with Kafka
4. Usage - Configuration options and additional functionality
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module

Overview

The Kafka module for managing the installation and configuration of Apache Kafka.

Module Description

The Kafka module for managing the installation and configuration of Apache Kafka: it's brokers, producers and consumers.

Setup

What Kafka affects

Installs the Kafka package and creates a new service.

Setup requirements

This module has the following dependencies:

• deric/zookeeper
• camptocamp/systemd
• puppet/archive
• puppetlabs/java
• puppetlabs/stdlib

Beginning with Kafka

To successfully install Kafka using this module you need to have Apache ZooKeeper already running at localhost:2181. You can specify another ZooKeeper host:port configuration using the config hash of the kafka::broker class.

The default configuration installs Kafka 0.11.0.3 binaries with Scala 2.11:

  class { 'kafka': }

If you want a Kafka broker server that connects to ZooKeeper listening on port 2181:

  class { 'kafka::broker':
    config => {
      'broker.id'         => '0',
      'zookeeper.connect' => 'localhost:2181'
    }
  }

Usage

You can specify different Kafka binaries packages versions to install. Please take a look at the different Scala and Kafka versions combinations at the Apache Kafka Website

Installing Kafka version 1.1.0 with scala 2.12

We first install the binary package with:

  class { 'kafka':
    version => '1.1.0',
    scala_version => '2.12'
  }

Then we set a minimal Kafka broker configuration with:

  class { 'kafka::broker':
    config => {
      'broker.id'         => '0',
      'zookeeper.connect' => 'localhost:2181'
    }
  }

Reference

The reference documentation of this module is generated using puppetlabs/puppetlabs-strings.

Limitations

This module only supports Kafka >= 0.9.0.0.

This module is tested on the following platforms:

• Debian 8
• Debian 9
• Debian 10
• Ubuntu 16.04
• Ubuntu 18.04
• CentOS 7

It is tested with the OSS version of Puppet (>= 5.5) only.

Development

This module has grown over time based on a range of contributions from people using it. If you follow these contributing guidelines your patch will likely make it into a release a little more quickly.

Author

This module is maintained by Vox Pupuli. It was originally written and maintained by Liam Bennett.