This content originally appeared on DEV Community and was authored by Richard Zhang
Agenda
- Kafka Architecture
- Basic concepts(topic/partition/consumer group/commit log/offset)
- Delver into the process of sending a message to the broker
- Technical Highlights
Architecture - Overview
Architecture - Detail
Architecture - Basic Concepts
Topics
A Kafka topic (https://www.instaclustr.com/support/documentation/kafka/using-kafka/topic-management/) defines a channel through which data is streamed.Producers publish messages to topics,and consumers read messages from the topic they subscribe to.
Topics organize and structure messages,with particular types of messages published to particular topics.Topics are identified by unique names within a Kafka cluster, and there is no limit on the number of topics that can be created.
Partitions
Within the Kafka cluster, topics are divided into partitions (https://www.instaclustr.com/blog/the-power-of-kafka-partitions-how-to-get-the-most-out-of-your-kafka-cluster/), and the partitions are replicated across
brokers. From each partition, multiple consumers can read from a topic in parallel. It's also
possible to have producers add a key to a message-all messages with the same key will go to
the same partition.enter image description here
While messages are added and stored within
partitions in sequence,messages without keys are
written to partitions in a round robin fashion.By
leveraging keys,you can guarantee the order of
processing for messages in Kafka that share the
same key.This is a particularly useful feature for
applications that require total control over records.
There is no limit on the number of Kafka
partitions that can be created(subject to the
processing capacity of a cluster).
This content originally appeared on DEV Community and was authored by Richard Zhang
Richard Zhang | Sciencx (2024-11-06T03:05:41+00:00) Kafka – Overview. Retrieved from https://www.scien.cx/2024/11/06/kafka-overview/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.