This content originally appeared on DEV Community and was authored by Marcos Maia
In a recent post I showed a setup to run Kafka on Kubernetes locally using Persistent Volumes(PV) and Persistent Volume Claims(PVC), this post covers the setup using Kubernetes Storage Class(SC) and leveraging the default one provisioned automatically by Kind which is the Rancher local-path-provisioner.
This setup is simpler and uses less code than the previous one with the trade off of having a bit less control over the path of data externalized to the host machine while requiring some internal knowledge of Kind to set it up. In general I favor this approach for local development over the one from my previous post.
Strimzi is an awesome simpler alternative to achieve the same, check it out. My goal with this setup here is for learning and to have a more "realistic" Kubernetes setup on local development machine so I opted to not use Strimzi or Helm charts.
I have created and tested these approaches on a Linux Development machine. It should work for Mac and Windows with some minimal adjustments also but I have never tried it.
You can get the full source from Github repo where you will find the files and Quick Start for both aforementioned approaches. To clone the repo git clone git@github.com:mmaia/kafka-local-kubernetes.git.
The setup using Storage class
If you checked out the repo described above the setup presented here is under storage-class-setup folder. You will find multiple Kubernetes declarative files in this folder, please notice that you could also combine all files in a single one separating them with a line containing triple dashes ---, if combining them is your preference you can open a terminal and from the storage-class-setup folder run for each in ./kafka-k8s/*; do cat $each; echo "---"; done > local-kafka-combined.yaml this will concatenate all files in a single one called local-kafka-combined.yaml.
The main thing to notice in this setup below compared to the previous one is that you don't have any PV or PVC configuration files this time because we're leveraging the default Rancher local-path-provisioner provided by Kind automatically through it's default Storage class.
I keep them separate to explicitly separate each type in this case and because it's convenient as you can just run kubectl pointing to the directory as described below in the "Running it" section.
kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes.
kind-config.yaml - This file configures Kind to expose the kafka and schema-registry ports to the local machine host so you can connect your application while developing from your IDE or command line and connect with Kafka running on Kubernetes it also maps the default path of the Rancher storage provisioner from Kind container to your local host machine.
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 30092 # internal kafka nodeport
hostPort: 9092 # port exposed on "host" machine for kafka
- containerPort: 30081 # internal schema-registry nodeport
hostPort: 8081 # port exposed on "host" machine for schema-registry
extraMounts:
- hostPath: ./tmp
containerPath: /var/local-path-provisioner
readOnly: false
selinuxRelabel: false
propagation: Bidirectional
kafka-network-np.yaml - Sets up the internal Kubernetes network used by the setup.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: kafka-network
spec:
ingress:
- from:
- podSelector:
matchLabels:
network/kafka-network: "true"
podSelector:
matchLabels:
network/kafka-network: "true"
kafka-service.yaml - This file defines the mappings between the internal containers and ports that are exposed, called NodePorts by defaul in Kubernetes nodeports can be used in the range 30000 to 32767.
apiVersion: v1
kind: Service
metadata:
labels:
service: kafka
name: kafka
spec:
selector:
service: kafka
ports:
- name: internal
port: 29092
targetPort: 29092
- name: external
port: 30092
targetPort: 9092
nodePort: 30092
type: NodePort
kafka-ss.yaml - This is the definition of Kafka in this setup, this time we use a Stateful Set.
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
service: kafka
name: kafka
spec:
serviceName: kafka
replicas: 1
selector:
matchLabels:
service: kafka
template:
metadata:
labels:
network/kafka-network: "true"
service: kafka
spec:
enableServiceLinks: false
containers:
- name: kafka
imagePullPolicy: IfNotPresent
image: confluentinc/cp-kafka:7.0.1
ports:
- containerPort: 29092
- containerPort: 9092
env:
- name: CONFLUENT_SUPPORT_CUSTOMER_ID
value: "anonymous"
- name: KAFKA_ADVERTISED_LISTENERS
value: "INTERNAL://kafka:29092,LISTENER_EXTERNAL://kafka:9092"
- name: KAFKA_AUTO_CREATE_TOPICS_ENABLE
value: "true"
- name: KAFKA_BROKER_ID
value: "1"
- name: KAFKA_DEFAULT_REPLICATION_FACTOR
value: "1"
- name: KAFKA_INTER_BROKER_LISTENER_NAME
value: "INTERNAL"
- name: KAFKA_LISTENERS
value: "INTERNAL://:29092,LISTENER_EXTERNAL://:9092"
- name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
value: "INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT"
- name: KAFKA_NUM_PARTITIONS
value: "1"
- name: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR
value: "1"
- name: KAFKA_LOG_CLEANUP_POLICY
value: "compact"
- name: KAFKA_ZOOKEEPER_CONNECT
value: "zookeeper:2181"
resources: {}
volumeMounts:
- mountPath: /var/lib/kafka/data
name: kafka-data
hostname: kafka
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: kafka-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
The remaining files are declarative Kubernetes configurations files to schema-registry and zookeeper.
schema-registry-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
service: schema-registry
name: schema-registry
spec:
replicas: 1
selector:
matchLabels:
service: schema-registry
strategy: {}
template:
metadata:
labels:
network/kafka-network: "true"
service: schema-registry
spec:
enableServiceLinks: false
containers:
- env:
- name: SCHEMA_REGISTRY_HOST_NAME
value: "schema-registry"
- name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
value: "kafka:29092"
- name: SCHEMA_REGISTRY_LISTENERS
value: "http://0.0.0.0:30081"
image: confluentinc/cp-schema-registry:7.0.1
name: schema-registry
ports:
- containerPort: 30081
resources: {}
hostname: schema-registry
restartPolicy: Always
schema-registry-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: schema-registry
name: schema-registry
spec:
ports:
- port: 30081
name: outport
targetPort: 30081
nodePort: 30081
type: NodePort
selector:
service: schema-registry
zookeeper-service.yaml
apiVersion: v1
kind: Service
metadata:
labels:
service: zookeeper
name: zookeeper
spec:
ports:
- name: "2181"
port: 2181
targetPort: 2181
selector:
service: zookeeper
zookeeper-ss.yaml - Again the main difference this time is the usage of Stateful Set.
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
service: zookeeper
name: zookeeper
spec:
serviceName: zookeeper
replicas: 1
selector:
matchLabels:
service: zookeeper
template:
metadata:
labels:
network/kafka-network: "true"
service: zookeeper
spec:
enableServiceLinks: false
containers:
- name: zookeeper
imagePullPolicy: IfNotPresent
image: confluentinc/cp-zookeeper:7.0.1
ports:
- containerPort: 2181
env:
- name: ZOOKEEPER_CLIENT_PORT
value: "2181"
- name: ZOOKEEPER_DATA_DIR
value: "/var/lib/zookeeper/data"
- name: ZOOKEEPER_LOG_DIR
value: "/var/lib/zookeeper/log"
- name: ZOOKEEPER_SERVER_ID
value: "1"
resources: {}
volumeMounts:
- mountPath: /var/lib/zookeeper/data
name: zookeeper-data
- mountPath: /var/lib/zookeeper/log
name: zookeeper-log
hostname: zookeeper
restartPolicy: Always
volumeClaimTemplates:
- metadata:
name: zookeeper-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
- metadata:
name: zookeeper-log
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 250Mi
Running it
- Open a terminal and cd to the storage-class-setup folder.
- Create a folder called "tmp", this is where the storage will be automatically provisioned by the default Kind storage class.
- Run kind specifying configuration:
kind create cluster --config=kind-config.yml. This will start a Kind kubernetes control plane + worker. - Run kubernetes configuration for kafka
kubectl apply -f kafka-k8s - When done stop kubernetes objects:
kubectl delete -f kafka-k8sand then if you want also stop the kind cluster which will also delete the storage on the host machine:kind delete cluster.
After running the
kubectl applycommand(step 4 above) check your local tmp folder where you will find the automated storage mapped to your local host disk, notice that those folders will be deleted when you shutdown the Kind cluster but they will persist over pod restarts of Kafka and zookeeper.
That's it, it's done, you have a functional local Kafka +
Schema Registry running on Kubernetes that you can reach from your application running on your developer machine or IDE.
Photo by Fotis Fotopoulos on Unsplash
This content originally appeared on DEV Community and was authored by Marcos Maia
Marcos Maia | Sciencx (2022-01-17T14:13:08+00:00) Running Kafka on Kubernetes for local development with Storage class. Retrieved from https://www.scien.cx/2022/01/17/running-kafka-on-kubernetes-for-local-development-with-storage-class/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.