Skip to content
This repository has been archived by the owner on Nov 8, 2022. It is now read-only.

intelsdi-x/snap-plugin-publisher-kafka

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

82 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DISCONTINUATION OF PROJECT.

This project will no longer be maintained by Intel.

This project has been identified as having known security escapes.

Intel has ceased development and contributions including, but not limited to, maintenance, bug fixes, new releases, or updates, to this project.

Intel no longer accepts patches to this project.

DISCONTINUATION OF PROJECT

This project will no longer be maintained by Intel. Intel will not provide or guarantee development of or support for this project, including but not limited to, maintenance, bug fixes, new releases or updates. Patches to this project are no longer accepted by Intel. If you have an ongoing need to use this project, are interested in independently developing it, or would like to maintain patches for the community, please create your own fork of the project.

Build Status

Snap publisher plugin - Kafka

Allows publishing of data to Apache Kafka

It's used in the snap framework.

  1. Getting Started
  1. Documentation
  1. Community Support
  2. Contributing
  3. License
  4. Acknowledgements

Getting Started

System Requirements

  • Uses sarama golang client for Kafka by Shopify

Installation

Download Kafka plugin binary:

You can get the pre-built binaries for your OS and architecture at plugin's GitHub Releases page.

To build the plugin binary:

Fork https://github.com/intelsdi-x/snap-plugin-publisher-kafka

Clone repo into $GOPATH/src/github.com/intelsdi-x/:

$ git clone https://github.com/<yourGithubID>/snap-plugin-publisher-kafka.git

Build the plugin by running make within the cloned repo:

$ make

This builds the plugin in ./build

Configuration and Usage

Task Manifest Config

In task manifest, in config section of Kafka publisher the following settings can be declared:

Key Type Default value Description
topic string "snap" The topic to send messages
brokers string "localhost:9092" Semicolon delimited list of "server:port" brokers
Kafka's standard broker communication port is 9092

Notice To publish to remote Kafka endpoint, please ensure that all listed brokers are available from the host where snap-plugin-publisher-kafka is running. You can simply check it by running telnet providing address and port where Kafka advertised broker is running, for example telnet 123.45.67.89 9092.

Documentation

Kafka Quickstart

This is a minimal-configuration needed to run the Kafka broker service on Docker

Run ZooKeeper server on docker

Kafka uses ZooKeeper so you need to first start a ZooKeeper server if you don't already have one.

 $ docker run -d --name zookeeper jplock/zookeeper:3.4.6

Check if ZooKeeper docker is running:

$ docker ps  

	CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS              			NAMES
	9b0ddbdd75cd        jplock/zookeeper:3.4.6     "/opt/zookeeper/bin/   38 seconds ago      Up 38 seconds       2181/tcp, 2888/tcp, 3888/tcp	zookeeper

Run Kafka server on docker

 docker run -d --name kafka --link zookeeper:zookeeper ches/kafka

Verify the running docker containers:

$ docker ps

	CONTAINER ID        IMAGE                        COMMAND                CREATED             STATUS              PORTS							NAMES
	dfb3cdfb3f87        ches/kafka:latest            "/start.sh"            7 seconds ago       Up 6 seconds        7203/tcp, 9092/tcp				kafka
	9b0ddbdd75cd        jplock/zookeeper:3.4.6       "/opt/zookeeper/bin/   3 minutes ago       Up 3 minutes        2181/tcp, 2888/tcp, 3888/tcp	zookeeper

Get Kafka advertised hostname:

$ docker inspect --format '{{ .NetworkSettings.IPAddress }}' kafka

  172.17.0.14

Read more about Kafka on http://kafka.apache.org

Published data

The plugin publishes all collected metrics serialized as JSON to Kafka. An example of published data is below:

[
  {
    "timestamp": "2016-07-25T11:27:59.795548989+02:00",
    "namespace": "/intel/mock/bar",
    "data": 82,
    "unit": "",
    "tags": {
      "plugin_running_on": "my-machine"
    },
    "version": 0,
    "last_advertised_time": "2016-07-25T11:27:21.852064032+02:00"
  },
  {
    "timestamp": "2016-07-25T11:27:59.795549268+02:00",
    "namespace": "/intel/mock/foo",
    "data": 72,
    "unit": "",
    "tags": {
      "plugin_running_on": "my-machine"
    },
    "version": 0,
    "last_advertised_time": "2016-07-25T11:27:21.852063228+02:00"
  }
]

Examples

Example of running psutil collector plugin and publishing data to Kafka.

Set up the Snap framework

Ensure Snap daemon is running:

  • initd: service snap-telemetry start
  • systemd: systemctl start snap-telemetry
  • command line: sudo snapteld -l 1 -t 0 &

Download and load Snap plugins (paths to binary files for Linux/amd64):

$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-publisher-kafka/latest/linux/x86_64/snap-plugin-publisher-kafka
$ wget http://snap.ci.snap-telemetry.io/plugins/snap-plugin-collector-psutil/latest/linux/x86_64/snap-plugin-collector-psutil
$ snaptel plugin load snap-plugin-publisher-kafka
$ snaptel plugin load snap-plugin-collector-psutil

Create a task manifest (see exemplary tasks), for example psutil-kafka.json with following content:

{
  "version": 1,
  "schedule": {
    "type": "simple",
    "interval": "1s"
  },
  "workflow": {
    "collect": {
      "metrics": {
        "/intel/psutil/load/load1": {},
        "/intel/psutil/load/load15": {}
      },
      "publish": [
        {
          "plugin_name": "kafka",
          "config": {
            "topic": "test",
            "brokers": "172.17.0.14:9092"
          }
        }
      ]
    }
  }
}

Create a task:

$ snaptel task create -t psutil-kafka.json

Watch created task:

$ snaptel task watch <task_id>

To stop previously created task:

$ snaptel task stop <task_id>

Roadmap

There isn't a current roadmap for this plugin, but it is in active development. As we launch this plugin, we do not have any outstanding requirements for the next release.

If you have a feature request, please add it as an issue and/or submit a pull request.

Community Support

This repository is one of many plugins in Snap, a powerful telemetry framework. See the full project at http://github.com/intelsdi-x/snap To reach out to other users, head to the main framework

Contributing

We love contributions!

There's more than one way to give back, from examples to blogs to code updates. See our recommended process in CONTRIBUTING.md.

License

Snap, along with this plugin, is an Open Source software released under the Apache 2.0 License.

Acknowledgements

And thank you! Your contribution, through code and participation, is incredibly important to us.