Kafka Metrics Monitoring with Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels.

Features : -

Prometheus’s main features are:

• a multi-dimensional data model with time series data identified by metric name and key/value pairs
• PromQL, a flexible query language to leverage this dimensionality
• no reliance on distributed storage; single server nodes are autonomous
• time series collection happens via a pull model over HTTP
• pushing time series is supported via an intermediary gateway
• targets are discovered via service discovery or static configuration
• multiple modes of graphing and dashboarding support

Architecture

Kafka and Prometheus JMX exporter

Kafka is an open-source stream-processing software platform written in Scala and Java. The general aim is to provide a unified, high-throughput, low-latency platform for real-time handling of data feeds. The storage layer of the software platform makes it extremely beneficial for businesses in terms of processing the streaming data. Moreover, Kafka is capable to connect to the external systems via Kafka Connect. Apache Kafka provides you with opportunities:

• to subscribe to streams of records
• to publish data to any numbers of systems
• to store the streams of records
• to process the streams of records.

Prometheus JMX exporter is a collector, designed for scraping and exposing mBeans of a JMX target. It runs as a Java agent as well as an independent HTTP server. The JMX exporter can export from various applications and efficiently work with your matrix.

Installation

Kafka JMX Metrics Setup in Kafka Machine

sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

Start Kafka Broker :

sudo KAFKA_HEAP_OPTS=”-Xmx1000M -Xms1000M” KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml

Start Kafka Consumer :

KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml” /opt/kafka/bin/kafka-console-consumer.sh

Start Kafka Procedure :

KAFKA_OPTS=”-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml” /opt/kafka/bin/kafka-console-producer.sh

To check the JMX metrics

curl localhost:7071

Prometheus Server Installation :

Create user and Directory.

sudo useradd --no-create-home prometheus
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
wget  https://github.com/prometheus/prometheus/releases/download/v2.23.0/prometheus-2.23.0.linux-amd64.tar.gz
tar -xvf prometheus-2.23.0.linux-amd64.tar.gz
sudo cp prometheus-2.23.0.linux-amd64/prometheus /usr/local/bin
sudo cp prometheus-2.23.0.linux-amd64/promtool /usr/local/bin
sudo cp -r prometheus-2.23.0.linux-amd64/consoles /etc/prometheus/
sudo cp -r prometheus-2.23.0.linux-amd64/console_libraries /etc/prometheus
sudo cp prometheus-2.23.0.linux-amd64/promtool /usr/local/bin/
rm -rf prometheus-2.23.0.linux-amd64.tar.gz prometheus-2.19.0.linux-amd64

Now we will configure Prometheus to monitor itself using yaml file. Create a prometheus.yml file at /etc/prometheus/prometheus.yml

global:
scrape_interval: 15s
external_labels:
monitor: ‘prometheus’
scrape_configs:
— job_name: ‘prometheus’
static_configs:
— targets: [‘localhost:9090’]

Now we want to run the Prometheus as a Service so that in case of server restart service will come automatically.

Let’s create a file /etc/systemd/system/prometheus.service with the below content

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
— config.file /etc/prometheus/prometheus.yml \
— storage.tsdb.path /var/lib/prometheus/ \
— web.console.templates=/etc/prometheus/consoles \
— web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target

Change the ownership of all folders and files which we have created to the user which we have created

sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus

Now we will configure the service and start it

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Now open it on the browser using below url:

http://Prometheus-Machine_IP:9090/

To Check Kafka Metrics :

Login with http://Prometheus-Machine -IP:9090/

In enable query section. We can able to see Kafka enabled Metrics.