Kafka Installation & Monitoring

Kakfa Installtion

Host Details : -

10.10.72.108 kuber1.tuxhub.com kuber1
10.10.72.109 kuber2.tuxhub.com kuber2
10.10.72.114 kuber4.tuxhub.com kuber4


 Host :- kuber1.tuxhub.com

[root@kuber1 ]#  mkdir /usr/local/kafka;cd /usr/local/kafka;wget http://apache.mirror.digitalpacific.com.au/kafka/0.10.2.1/kafka_2.12-0.10.2.1.tgz;tar -zxf kafka_2.12-0.10.2.1.tgz ;mv kafka_2.12-0.10.2.1 kafka-2.12

[root@kuber1 config]#
[root@kuber1 config]# vim server.properties
broker.id=1                               # Need to change this on every node
delete.topic.enable=true
advertised.listeners=PLAINTEXT://kuber1.tuxhub.com:9092    # Need to change this on every node
num.network.threads=3
num.io.threads=8
default.replication.factor=3
min.insync.replicas=2
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka
num.partitions=8
num.recovery.threads.per.data.dir=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=10.10.72.109:5181/kafka     # Zookeeper host
zookeeper.connection.timeout.ms=6000
auto.create.topic.enable=true


On All nodes

Make sure to chnage "java.rmi.server.hostname"

[root@kuber1 ]# vim /usr/local/kafka/kafka-2.12/bin/kafka-server-start.sh
export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=kuber1.tuxhub.com  -Djava.net.preferIPv4Stack=true"
export JMX_PORT=9999
export KAFKA_OPTS='-javaagent:/usr/local/prometheus/jmx-export/lib/jmx_prometheus_javaagent-0.3.1.jar=7071:/usr/local/prometheus/jmx-export/conf/kafka.yml'

For prometheus monitoring:

[mapr@kuber1 ~]$  wget https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
[mapr@kuber1 ~]$ mkdir -p /usr/local/prometheus/jmx-export/{conf,lib}
[mapr@kuber1 ~]$  wget https://github.com/prometheus/prometheus/releases/download/v2.4.3/prometheus-2.4.3.linux-amd64.tar.gz
[mapr@kuber1 ~]$  cd /usr/local/prometheus/
[mapr@kuber1 ~]$  gunzip prometheus-2.4.3.linux-amd64.tar.gz
 [mapr@kuber1 ~]$  tar -xvf prometheus-2.4.3.linux-amd64.tar
 [mapr@kuber1 ~]$  mv  prometheus-2.4.3.linux-amd6/ prometheus-2.4.3

[mapr@kuber1 ~]$ cp -pav jmx_prometheus_javaagent-0.3.1.jar /usr/local/prometheus/jmx-export/lib/

[mapr@kuber1 ~]$ cd  /usr/local/prometheus/jmx-export/conf
[mapr@kuber1 ~]$ curl -O https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml
[mapr@kuber1 ~]$ vim /usr/local/prometheus/prometheus-2.4.3/prometheus.yml

 Add config to prometheus.yml

 scrape_configs:
 - job_name: 'kafka'
   static_configs:
    - targets:
      - 10.10.72.108:7071
      - 10.10.72.109:7071
      - 10.10.72.114:7071

Start Kafka broker on all nodes

[mapr@kuber1 ~]$ /usr/local/kafka/kafka-2.12/bin/kafka-server-start.sh -daemon /usr/local/kafka/kafka-2.12/config/server.properties

Start Prometheus :

[mapr@kuber1 ~]$  /usr/local/prometheus/prometheus-2.4.3/prometheus --web.listen-address=10.10.72.108:9080

Start topic/producer/consumer :

A) Topic create
[mapr@kuber1 ~]$  /usr/local/kafka/kafka-2.12/bin/kafka-topics.sh --zookeeper kuber2.tuxhub.com:5181/kafka --partitions 3 --replication-factor 3  --create --topic t2

B) Producer launch

[mapr@kuber2 ~]$ /usr/local/kafka/kafka-2.12/bin/kafka-console-producer.sh --broker-list kuber1.tuxhub.com:9092,kuber2.tuxhub.com:9092,kuber4.tuxhub.com:9092 --topic t2

= > Send Msg1

C) Consumer Launch

[mapr@kuber4 ~]$ /usr/local/kafka/kafka-2.12/bin/kafka-console-consumer.sh --bootstrap-server  kuber1.tuxhub.com:9092,kuber2.tuxhub.com:9092,kuber4.tuxhub.com:9092 --topic t2  --from-beginning
= > Recive Msg1


Monitoring

A) prometheus

http://10.10.72.108:9080/

B) Grafana

Add a data source as prometheus

Import Kafka Dashboard

https://gist.githubusercontent.com/njadhav1/3f7fa2c7199f6952773d9a150eeebaf1/raw/473f45e916d8fc273fa6df70a38a6946f317ab4a/Kafka%2520Grafana%2520Dashboard

http://kuber1.tuxhub.com:3000/


C) Kafka Manager

Kafka Manager WebUI

[root@kuber1 docker]# cat kafka-manager.yml
version: '2'
services:
    kafka-manager:
        image: intropro/kafka-manager
        container_name: kafka-manager
        ports:
            - "9009:9000"
        environment:
            ZK_HOSTS: 10.10.72.109:5181


[root@kuber1 docker]# docker-compose -f kfka-manager.yml up -d

WebUI :

http://10.10.72.108:9009/clusters/

Dashboard :

http://kuber1.tuxhub.com:3000/