In this section, we will delve into the core concepts of Kafka: Topics and Partitions. Understanding these concepts is crucial for effectively using Kafka in your applications.

A Topic in Kafka is a category or feed name to which records are stored and published. Topics are fundamental to Kafka's publish-subscribe model.

• Topics are logical channels to which producers send records and from which consumers read records.
• Each topic is identified by its name.
• Topics are multi-subscriber; a topic can have zero, one, or many consumers that subscribe to the data written to it.

Imagine a topic named user-signups where all user signup events are published. Multiple services can subscribe to this topic to process these events.

A Partition is a division of a topic. Each topic can have multiple partitions, which allows Kafka to scale horizontally.

• Partitions are the basic unit of parallelism in Kafka.
• Each partition is an ordered, immutable sequence of records.
• Partitions are distributed across multiple brokers in a Kafka cluster.
• Each record within a partition has a unique offset, which is an integer value that uniquely identifies each record within the partition.

If the user-signups topic has 3 partitions, the records will be distributed across these partitions. This distribution allows for parallel processing and better performance.

• When a producer sends a record to a topic, Kafka decides which partition to place the record in.
• The partitioning can be done based on a key (if provided) or in a round-robin fashion if no key is provided.

• By having multiple partitions, Kafka can handle more data and more consumers.
• Each partition can be consumed by a different consumer in a consumer group, allowing for parallel processing.

• Partitions are replicated across multiple brokers to ensure fault tolerance.
• If a broker fails, another broker with the replica can take over.

Let's look at a simple example of creating a topic with partitions using Kafka's command-line tools.

• --create: Command to create a new topic.
• --topic user-signups: Name of the topic.
• --partitions 3: Number of partitions for the topic.
• --replication-factor 1: Number of replicas for each partition.
• --bootstrap-server localhost:9092: Address of the Kafka broker.

1. Create a topic named order-events with 4 partitions and a replication factor of 2.
2. Produce some messages to the order-events topic.
3. Consume the messages from the order-events topic.

1. Create the topic:

   kafka-topics.sh --create --topic order-events --partitions 4 --replication-factor 2 --bootstrap-server localhost:9092
   

2. Produce messages:

   kafka-console-producer.sh --topic order-events --bootstrap-server localhost:9092
   

3. Consume messages:

   kafka-console-consumer.sh --topic order-events --from-beginning --bootstrap-server localhost:9092
   

1. Create a topic named product-updates with 3 partitions.
2. Produce messages with keys to the product-updates topic.
3. Observe how messages with the same key go to the same partition.

1. Create the topic:

   kafka-topics.sh --create --topic product-updates --partitions 3 --replication-factor 1 --bootstrap-server localhost:9092
   

2. Produce messages with keys:

   kafka-console-producer.sh --topic product-updates --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092
   

   Example input:

   key1:message1
   key2:message2
   key1:message3
   

3. Consume messages:

   kafka-console-consumer.sh --topic product-updates --from-beginning --bootstrap-server localhost:9092 --property "print.key=true" --property "key.separator=:"
   

In this section, we covered the fundamental concepts of Kafka topics and partitions. We learned how topics serve as logical channels for data, and how partitions enable parallelism and scalability. We also explored practical examples and exercises to reinforce these concepts. Understanding topics and partitions is essential for designing efficient and scalable Kafka-based applications.