The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Managing Kafka Topics

Next ►
►

In this section, we will delve into the management of Kafka topics, which are fundamental to organizing and categorizing messages in Kafka. We will cover the following key areas:

  1. Creating Topics
  2. Listing Topics
  3. Describing Topics
  4. Modifying Topics
  5. Deleting Topics

  1. Creating Topics

Kafka topics can be created using the Kafka command-line tools or programmatically using the Kafka Admin API.

Using Command-Line Tools

To create a topic using the command-line tool, you can use the kafka-topics.sh script. Here is an example:

bin/kafka-topics.sh --create --topic my-topic --bootstrap-server localhost:9092 --partitions 3 --replication-factor 2

Explanation:

  • --create: Indicates that we want to create a new topic.
  • --topic my-topic: The name of the topic to be created.
  • --bootstrap-server localhost:9092: The Kafka broker address.
  • --partitions 3: The number of partitions for the topic.
  • --replication-factor 2: The replication factor for the topic.

Using Kafka Admin API

You can also create topics programmatically using the Kafka Admin API. Here is an example in Java:

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.NewTopic;

import java.util.Collections;
import java.util.Properties;

public class CreateTopicExample {
    public static void main(String[] args) {
        Properties config = new Properties();
        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (AdminClient adminClient = AdminClient.create(config)) {
            NewTopic newTopic = new NewTopic("my-topic", 3, (short) 2);
            adminClient.createTopics(Collections.singleton(newTopic)).all().get();
            System.out.println("Topic created successfully");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Explanation:

  • AdminClient.create(config): Creates an AdminClient with the specified configuration.
  • NewTopic("my-topic", 3, (short) 2): Creates a new topic with the specified name, number of partitions, and replication factor.
  • adminClient.createTopics(Collections.singleton(newTopic)).all().get(): Creates the topic and waits for the operation to complete.

  1. Listing Topics

To list all the topics in a Kafka cluster, you can use the kafka-topics.sh script or the Kafka Admin API.

Using Command-Line Tools

bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Using Kafka Admin API

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;

import java.util.Properties;

public class ListTopicsExample {
    public static void main(String[] args) {
        Properties config = new Properties();
        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (AdminClient adminClient = AdminClient.create(config)) {
            adminClient.listTopics().names().get().forEach(System.out::println);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

  1. Describing Topics

To get detailed information about a specific topic, you can use the kafka-topics.sh script or the Kafka Admin API.

Using Command-Line Tools

bin/kafka-topics.sh --describe --topic my-topic --bootstrap-server localhost:9092

Using Kafka Admin API

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;
import org.apache.kafka.clients.admin.TopicDescription;

import java.util.Properties;

public class DescribeTopicExample {
    public static void main(String[] args) {
        Properties config = new Properties();
        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (AdminClient adminClient = AdminClient.create(config)) {
            TopicDescription description = adminClient.describeTopics(Collections.singleton("my-topic")).all().get().get("my-topic");
            System.out.println(description);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

  1. Modifying Topics

Modifying topics typically involves changing the number of partitions or updating configurations. Note that reducing the number of partitions is not supported.

Changing the Number of Partitions

bin/kafka-topics.sh --alter --topic my-topic --partitions 5 --bootstrap-server localhost:9092

Updating Topic Configurations

bin/kafka-configs.sh --alter --entity-type topics --entity-name my-topic --add-config retention.ms=604800000 --bootstrap-server localhost:9092

  1. Deleting Topics

To delete a topic, you can use the kafka-topics.sh script or the Kafka Admin API.

Using Command-Line Tools

bin/kafka-topics.sh --delete --topic my-topic --bootstrap-server localhost:9092

Using Kafka Admin API

import org.apache.kafka.clients.admin.AdminClient;
import org.apache.kafka.clients.admin.AdminClientConfig;

import java.util.Properties;

public class DeleteTopicExample {
    public static void main(String[] args) {
        Properties config = new Properties();
        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

        try (AdminClient adminClient = AdminClient.create(config)) {
            adminClient.deleteTopics(Collections.singleton("my-topic")).all().get();
            System.out.println("Topic deleted successfully");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Conclusion

In this section, we covered the essential operations for managing Kafka topics, including creating, listing, describing, modifying, and deleting topics. Understanding these operations is crucial for effectively managing your Kafka infrastructure and ensuring that your data is organized and accessible. In the next section, we will explore monitoring Kafka to ensure its health and performance.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved