added prometheus in favor of graphite

use metricbeat instead of diamond and burrow
version upgrades

README.md

@@ -4,15 +4,14 @@
 
 In case you need a local cluster providing Kafka (**with SSL and ACL**) including a monitoring suite.
 
-* [Apache Kafka 2.0.0](http://kafka.apache.org/20/documentation.html)
-* [Elastic Search 6.5.4](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/index.html)
-* [Logstash 6.5.4](https://www.elastic.co/guide/en/logstash/6.5/index.html)
-* [Kibana 6.5.4](https://www.elastic.co/guide/en/kibana/6.5/index.html)
-* [Grafana 5.4.2](https://grafana.com)
-* [Graphite-Web & Python-Carbon 0.9.16](https://graphiteapp.org)
-* [Jmxtrans Agent 1.2.6](https://github.com/jmxtrans/jmxtrans-agent/)
-* [Burrow 1.1.0](https://github.com/linkedin/Burrow)
-* [Diamond 4.0.515](https://diamond.readthedocs.io/en/latest/)
+* [Apache Kafka 2.1.0](http://kafka.apache.org/21/documentation.html)
+* [Elastic Search 6.6.0](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index.html)
+* [Logstash 6.6.0](https://www.elastic.co/guide/en/logstash/6.6/index.html)
+* [Kibana 6.6.0](https://www.elastic.co/guide/en/kibana/6.6/index.html)
+* [Filebeat 6.6.0](https://www.elastic.co/guide/en/beats/filebeat/6.6/index.html)
+* [Metricbeat 6.6.0](https://www.elastic.co/guide/en/beats/metricbeat/6.6/index.html)
+* [Grafana 5.4.3](https://grafana.com)
+* [Prometheus 2.7.1](https://prometheus.io)
 
 ## Prerequisites
 
@@ -44,11 +43,11 @@ The result if everything wents fine should be
 
 | IP | Hostname | Description | Settings |
 |:--- |:-- |:-- |:-- |
-|192.168.10.2|mon-1|running elk and diamond| 4096 MB RAM |
-|192.168.10.3|mon-2|running grafana, graphite and diamond| 2048 MB RAM |
-|192.168.10.4|kafka-1|running a kafka broker and diamond| 2048 MB RAM |
-|192.168.10.5|kafka-2|running a kafka broker and diamond| 2048 MB RAM |
-|192.168.10.6|kafka-3|running a kafka broker and diamond| 2048 MB RAM |
+|192.168.10.2|mon-1|running elk and metricbeat | 4096 MB RAM |
+|192.168.10.3|mon-2|running grafana, prometheus and metricbeat | 2048 MB RAM |
+|192.168.10.4|kafka-1|running a kafka broker and metricbeat | 2048 MB RAM |
+|192.168.10.5|kafka-2|running a kafka broker and metricbeat | 2048 MB RAM |
+|192.168.10.6|kafka-3|running a kafka broker and metricbeat | 2048 MB RAM |
 
 
 ### Connections
@@ -60,25 +59,30 @@ The result if everything wents fine should be
 |Kibana|[http://mon-1:5601](http://mon-1:5601)|
 |Elasticsearch|[http://mon-1:9200](http://mon-1:9200)|
 |Grafana|[http://mon-2:3000](http://mon-2:3000)|
-|Graphite|[http://mon-2](http://mon-2)|
-|Burrow|[http://kafka-1:8000/burrow/admin](http://kafka-1:8000/burrow/admin)|
-
+|Prometheus|[http://mon-2:9090](http://mon-2:9090)|
 
 # Monitoring
 
 ## System Overview
 
-Gathered with Diamond on each host.
+#### Kibana Infrastructure
 
-![System Overview](doc/system_overview.png)
+![System Overview](doc/system.png)
+![System Host Detail](doc/system_detail.png)
 
 
 ## Kafka Overview
 
-Gathered with Jmxtrans Agent from Kafka Brokers.
+#### Gathered with Prometheus Agent from Kafka Brokers.
 
 ![Kafka Overview](doc/kafka_overview.png)
 
+#### Kibana Metricbeat
+
+![Metricbeat Kafka](doc/metricbeat_kafka.png)
+
+#### Kibana Logs
+
 ![Filebeat Kafka](doc/filbeat_kafka.png)
 
 # Usage

Vagrantfile

@@ -48,6 +48,16 @@ Vagrant.configure("2") do |config|
     end
     mon.vm.network :private_network, ip: "192.168.10.3", auto_config: true
 
+    mon.vm.provision :ansible do |ansible|
+      ansible.compatibility_mode = "2.0"
+      ansible.limit = "mon"
+      ansible.playbook = "ansible/network.yml"
+      ansible.inventory_path = "ansible/inventories/vbox"
+      ansible.raw_arguments  = [
+        "-vv"
+      ]
+    end
+
     mon.vm.provision :ansible do |ansible|
       ansible.compatibility_mode = "2.0"
       ansible.limit = "grafana"

ansible/cluster.yml

@@ -6,21 +6,27 @@
   - elasticsearch
   - logstash
   - kibana
-  - diamond
+  - metricbeat
+
+- hosts: prometheus
+  remote_user: vagrant
+  serial: 100%
+  roles:
+  - metricbeat
+  - prometheus
 
 - hosts: grafana
   remote_user: vagrant
   serial: 100%
   roles:
-  - graphite
   - grafana
-  - diamond
 
 - hosts: zookeeper
   remote_user: vagrant
   serial: 33%
   roles:
     - java8
+    - metricbeat
     - secure
     - zookeeper
 
@@ -29,12 +35,5 @@
   serial: 33%
   roles:
     - java8
-    - kafka
-    - diamond
     - filebeat
-
-- hosts: kafka-1
-  remote_user: vagrant
-  serial: 100%
-  roles:
-    - burrow
+    - kafka

ansible/inventories/vbox/group_vars/all

@@ -5,7 +5,6 @@ ssl_client_dir: "{{ exchange }}/ssl-client"
 usr_local: /usr/local
 etc_profiles: /etc/profile.d
 system_units: /etc/systemd/system
-jmxtrans_dest: /usr/local/jmxtrans
 keystore_pwd: keystore-secret
 key_pwd: key-secret
 truststore_pwd: truststore-secret

ansible/inventories/vbox/hosts

@@ -8,6 +8,9 @@ mon-1
 [grafana]
 mon-2
 
+[prometheus]
+mon-2
+
 [zookeeper]
 kafka-1
 kafka-2