This content originally appeared on DEV Community and was authored by Sagar Lakshmipathy
In this article, we will walk through the steps to set up Apache Kafka on an Amazon EC2 instance (Amazon Linux distribution). We'll start with updating the system, installing Java, and then proceed with the installation and configuration of Kafka. By the end, you'll have a running Kafka instance, ready for producing and consuming messages.
Prerequisites
An Amazon EC2 instance (running Amazon Linux 2 or a similar distribution)
Basic knowledge of using the terminal
Steps
Step 1: Update the System
First, ensure your system is up to date by running:
sudo yum update -y
Step 2: Install Java
Apache Kafka requires Java to run. Install Java 11 using the following command:
sudo yum install java-11-amazon-corretto-devel -y
Step 3: Download and Extract Kafka
Next, download Kafka from the official Apache archive and extract it:
wget https://archive.apache.org/dist/kafka/3.3.1/kafka_2.12-3.3.1.tgz
tar -xzf kafka_2.12-3.3.1.tgz
cd kafka_2.12-3.3.1
Step 4: Start Zookeeper
Kafka requires Zookeeper to manage its cluster. Start Zookeeper with:
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > zookeeper.log 2>&1 &
Step 5: Configure Kafka
Edit the Kafka configuration file to set the advertised listeners. Replace ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com with your EC2 instance's public DNS:
vi config/server.properties
Add the following line:
advertised.listeners=PLAINTEXT://ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092
Step 6: Start Kafka Server
Now, start the Kafka server:
nohup bin/kafka-server-start.sh config/server.properties > kafka-server.log 2>&1 &
Step 7: Create a Topic
Create a new Kafka topic named test:
bin/kafka-topics.sh --create --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092 --partitions 1 --replication-factor 1
Step 8: Produce Messages
Start a Kafka producer to send messages to the test topic:
bin/kafka-console-producer.sh --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092
You can now type messages into the console, and they will be sent to the Kafka topic.
Step 9: Consume Messages
Start a Kafka consumer to read messages from the test topic:
bin/kafka-console-consumer.sh --topic test --bootstrap-server ec2-xx-xxx-xx-xxx.us-west-2.compute.amazonaws.com:9092 --from-beginning
You should see the messages you produced earlier displayed in the console.
Conclusion
By following these steps, you've successfully set up Apache Kafka on an Amazon EC2 instance. You can now produce and consume messages, enabling you to build real-time data streaming applications. This setup forms the foundation for more advanced Kafka configurations and use cases.
Feel free to explore Kafka's extensive features and tailor the configuration to suit your specific requirements. Happy streaming!
This content originally appeared on DEV Community and was authored by Sagar Lakshmipathy
Sagar Lakshmipathy | Sciencx (2024-07-31T03:49:05+00:00) Apache Kafka on Amazon Linux EC2. Retrieved from https://www.scien.cx/2024/07/31/apache-kafka-on-amazon-linux-ec2/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.