This content originally appeared on Level Up Coding - Medium and was authored by Rohit Singh
The post focus on how to set up Kafka in the docker environment, the common mistakes we do while connecting to Kafka from the Docker host or remote machines, and how to monitor Kafka metrics via Prometheus and Grafana.
Overview
Kafka is a highly available distributed messaging system, designed as an event streaming platform. Docker is an application containerization platform used to run containers. Kafka can be run as a docker service and exposed via listeners which allows clients to communicate with Kafka.
Kafka Listeners
There are two types of listeners, which we need to understand from Docker's perspective. First, LISTENERS, are what interfaces Kafka binds to. This is for clients connecting on docker. Secondly, ADVERTISED_LISTENERS, are how external clients can connect to Kafka and this is for clients connecting outside of the Docker machine. In the below diagram, Listener A acts as an internal listener and exposes Kafka to clients running on a Docker host. And, Listener B acts as an external listener and allows connections to Kafka from outside the Docker host to remote clients.
Kafka as Docker Container
Below is a standard example for setting Kafka up as a Docker container. We have to declare two services, one is kafka and another one is zookeeper as Kafka relies on Zookeeper for metadata and leader election. We are using confluent images for Kafka. Let's look at each attribute.
ports —for exposing Kafka service to the outside world.
image — docker image urls
broker id — symbolic id representing Kafka broker instance.
zookeeper connect — zookeeper connection
listeners — set of listeners with host/IP and port to which Kafka brokers binds to for listening.
advertised listeners — set of listeners with their host/IP and port and is what clients will use to connect to the brokers whether inside or from outside.
listener security protocol map — defines key/value pairs for the security protocol to use per listener name. (SSL or no SSL)
inter broker listener name — listener used by brokers internally to connect.
services:
kafka:
ports:
- 29094:29092
image: confluentinc/cp-kafka:latest
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
networks:
- kafka
depends_on:
- zookeeper
zookeeper:
ports:
- 32181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
image: confluentinc/cp-zookeeper:latest
networks:
- kafka
networks:
kafka:
external: false
With the above configuration, we can run Kafka as a Docker service on Docker-enabled machines. Save the above file as kafka.yml and run the below command:
---> run
docker-compose -f kafka.yml up -d
---> status
docker-compose -f kafka.yml ps
Client connection to Kafka
Kafka clients may be local to the broker’s network or maybe not, and this is where the listeners play an important role as described earlier. Each listener will, when connected, report back the address on which it can be reached. The address on which you reach a broker depends on the network used. If you’re connecting to the broker from an internal network it’s going to be a different host/IP than when connecting externally. There are basically three scenarios that arise:
- Clients within Docker network — Clients within the Docker network connect using listener PLAINTEXT, with port 9092 and hostname as kafka. Each Docker container will resolve kafka using Docker’s internal network and be able to reach the broker. The Bootstrap server used by clients to connect will be [kafka:9092]
- Clients external to the Docker network but on the same Docker host machine — clients connect using listener PLAINTEXT_HOST, with port 29092 and hostname localhost. Port 29092 is exposed by the Docker container and therefore becomes available to connect to. The Bootstrap server used by clients to connect will be [localhost:29092]
- Clients external to the Docker network and Docker host machine — clients connect using listener PLAINTEXT_HOST, with port 29094 and hostname localhost. Port 29094 is exposed by the Docker container for external clients and ports are mapped. Kindly note that in order to access this, you also need to do port mapping in putty on the client machine which needs to resolve to this docker host. The Bootstrap server used by clients to connect will be [localhost:29094].
Port Mapping (External remote client)
As mentioned here, most of the time we do mistake here, if you see a message like the one described below. First, please make sure Kafka brokers are running and check the logs to ensure that the Kafka server is started successfully. Second, if you are connecting from a remote client machine, then ensure to have port mapping updated in the connection tunnel where you have mapped bootstrap server port to docker-machine running Kafka service. For instance, let’s say your Docker machine IP is 62.96.231.219 and Kafka service is exposed on port 29094, then on the remote client machine, map port 29094 to 62.96.239.219:29094, and then the client can connect to Kafka via bootstrap server localhost:29094.
Conneciton could not be established. Broker may not be available.
Metrics and monitoring
Kafka brokers can be monitored using prometheus and grafana. We will use the JMX exporter to expose Kafka’s metrics for Prometheus to scrape and then visualize the metrics on a Grafana dashboard. JMX exporter acts as a collector which exposes Kafka metrics over an HTTP endpoint and then can be consumed by any system such as Prometheus.
We have added EXTRA_ARGS to use JMX exporter for exposing Kafka metrics and port mapping for exposing the same for Prometheus target to scrape. JMX exporter jar is required for the same. We also need to provide Kafka metrics configuration (kafka-broker.yaml) in order to expose the required metrics.
Updated docker-compose: (with metrics exposed)
services:
kafka:
logging:
driver: local
ports:
- 29094:29092
- 29103:29101
image: confluentinc/cp-kafka:latest
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENERS: PLAINTEXT://:9092,PLAINTEXT_HOST://:29092
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://127.0.0.1:29094
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
EXTRA_ARGS: -javaagent:/usr/share/jmx_exporter/jmx_prometheus_javaagent-0.17.0.jar=29101:/usr/share/jmx_exporter/kafka-broker.yaml
networks:
- kafka
volumes:
- ./jmx-exporter:/usr/share/jmx_exporter/
- kafka-jmx-volume:/jmx-exporter
depends_on:
- zookeeper
zookeeper:
ports:
- 32181:2181
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
image: confluentinc/cp-zookeeper:latest
networks:
- kafka
volumes:
kafka-jmx-volume:
networks:
kafka:
external: false
Update and save the file as kafka.yml and run the below command:
---> run
docker-compose -f kafka.yml up -d
---> status
docker-compose -f kafka.yml ps
Kafka metrics can be reached at : http://localhost:29103/metrics
Prometheus can be reached at : http://localhost:9090/graph
Grafana can be reached at : http://localhost:3000/login (kafka/kafka)
If Kafka metrics endpoint is successfully started and Prometheus target can scrape it, it should look like as below:
Grafana dashboard for Kafka cluster:
The detailed configuration on setting up Prometheus and Grafana can be found here:
https://github.com/rohsin47/kafka-docker-kubernetes/tree/main/docker
References:
- Kafka Listeners - Explained | Confluent
- jmx-monitoring-stacks/jmxexporter-prometheus-grafana at 6.1.0-post · confluentinc/jmx-monitoring-stacks
Level Up Coding
Thanks for being a part of our community! More content in the Level Up Coding publication.
Follow: Twitter, LinkedIn, Newsletter
Level Up is transforming tech recruiting ➡️ Join our talent collective
Kafka Primer for Docker: How to setup Kafka, start messaging and monitor broker metrics in Docker was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.
This content originally appeared on Level Up Coding - Medium and was authored by Rohit Singh
Rohit Singh | Sciencx (2022-07-11T00:12:39+00:00) Kafka Primer for Docker: How to setup Kafka, start messaging and monitor broker metrics in Docker. Retrieved from https://www.scien.cx/2022/07/11/kafka-primer-for-docker-how-to-setup-kafka-start-messaging-and-monitor-broker-metrics-in-docker/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.