This content originally appeared on DEV Community and was authored by eidher
1) Source connectors: Start our kafka cluster. This is the docker-compose.yml file:
version: '2'
services:
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
environment:
ADV_HOST: 127.0.0.1
RUNTESTS: 0
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
Create and start the container:
docker-compose up kafka-cluster
2) Create the topic we're going to write to:
docker run --rm -it --net=host landoop/fast-data-dev:cp3.3.0 bash
kafka-topics --create --topic demo-2-distributed --partitions 3 --replication-factor 1 --zookeeper 127.0.0.1:2181
3) In a browser go to 127.0.0.1:3030 -> Connect UI
Create a new connector -> File Source
Paste the next configuration and click the Create button:
name=file-stream-demo-distributed
connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector
tasks.max=1
file=demo-file.txt
topic=demo-distributed
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=true
4) Create the source file:
Execute docker ps to see the containerId.
docker exec -it <containerId> bash
touch demo-file.txt
echo "add any content" >> demo-file.txt
3) In a browser go to 127.0.0.1:3030 -> Kafka Topics UI
Click the topic defined previously (demo-distributed) and validate the content:
This content originally appeared on DEV Community and was authored by eidher
eidher | Sciencx (2024-08-23T22:31:57+00:00) Kafka Connect: FileStreamSourceConnector in distributed mode. Retrieved from https://www.scien.cx/2024/08/23/kafka-connect-filestreamsourceconnector-in-distributed-mode/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.