In this section, we will delve into the various configurations available in Kafka, how to set them up, and best practices for managing these configurations. Proper configuration is crucial for ensuring that Kafka runs efficiently and reliably.

1. Broker Configuration: Settings that apply to individual Kafka brokers.
2. Topic Configuration: Settings that apply to specific Kafka topics.
3. Producer Configuration: Settings that apply to Kafka producers.
4. Consumer Configuration: Settings that apply to Kafka consumers.
5. ZooKeeper Configuration: Settings for the ZooKeeper ensemble that Kafka uses for coordination.

1. broker.id: A unique identifier for each broker in the cluster.
2. log.dirs: Directories where Kafka will store log data.
3. zookeeper.connect: The ZooKeeper connection string.
4. listeners: The address the broker will bind to and listen on.
5. num.network.threads: Number of threads handling network requests.
6. num.io.threads: Number of threads handling I/O operations.

• broker.id=1: This sets the unique ID for the broker.
• log.dirs=/var/lib/kafka/logs: Specifies the directory for log data.
• zookeeper.connect=localhost:2181: Connects to the ZooKeeper instance running on localhost.
• listeners=PLAINTEXT://:9092: Configures the broker to listen on port 9092.
• num.network.threads=3: Allocates three threads for network requests.
• num.io.threads=8: Allocates eight threads for I/O operations.

1. retention.ms: Time to retain a log before deletion.
2. segment.bytes: Size of a single log segment file.
3. cleanup.policy: Policy for log cleanup (e.g., delete, compact).
4. min.insync.replicas: Minimum number of replicas that must acknowledge a write.

• retention.ms=604800000: Retains logs for one week (in milliseconds).
• segment.bytes=1073741824: Sets the segment size to 1 GB.
• cleanup.policy=delete: Configures the log cleanup policy to delete old logs.
• min.insync.replicas=2: Requires at least two replicas to acknowledge a write.

1. bootstrap.servers: List of Kafka brokers to connect to.
2. acks: Number of acknowledgments the producer requires.
3. retries: Number of retries on failed send attempts.
4. batch.size: Size of the batch for sending messages.
5. linger.ms: Time to wait before sending a batch.

• bootstrap.servers=localhost:9092: Connects to the Kafka broker on localhost.
• acks=all: Requires all replicas to acknowledge the write.
• retries=3: Retries up to three times on failure.
• batch.size=16384: Sets the batch size to 16 KB.
• linger.ms=1: Waits 1 ms before sending a batch.

1. bootstrap.servers: List of Kafka brokers to connect to.
2. group.id: Consumer group ID.
3. auto.offset.reset: What to do when there is no initial offset.
4. enable.auto.commit: Whether to enable auto-commit of offsets.
5. session.timeout.ms: Timeout for consumer group session.

• bootstrap.servers=localhost:9092: Connects to the Kafka broker on localhost.
• group.id=my-consumer-group: Sets the consumer group ID.
• auto.offset.reset=earliest: Starts reading from the earliest offset.
• enable.auto.commit=true: Enables auto-commit of offsets.
• session.timeout.ms=10000: Sets the session timeout to 10 seconds.

1. dataDir: Directory where ZooKeeper will store its data.
2. clientPort: Port on which ZooKeeper will listen for client connections.
3. tickTime: Basic time unit in milliseconds used by ZooKeeper.
4. initLimit: Time ZooKeeper servers in quorum have to connect and sync.
5. syncLimit: Time to allow followers to sync with the leader.

• dataDir=/var/lib/zookeeper: Specifies the directory for ZooKeeper data.
• clientPort=2181: Configures ZooKeeper to listen on port 2181.
• tickTime=2000: Sets the tick time to 2000 ms.
• initLimit=10: Allows 10 ticks for initial synchronization.
• syncLimit=5: Allows 5 ticks for followers to sync with the leader.

1. Objective: Configure a Kafka broker with the following settings:

   • Broker ID: 2
   • Log directory: /data/kafka/logs
   • ZooKeeper connection: zookeeper1:2181,zookeeper2:2181
   • Listener: PLAINTEXT://:9093
   • Network threads: 4
   • I/O threads: 10

2. Steps:

   • Open the Kafka broker configuration file (server.properties).
   • Set the required parameters as specified.
   • Save the configuration file.
   • Restart the Kafka broker to apply the changes.

• broker.id=2: Sets the broker ID to 2.
• log.dirs=/data/kafka/logs: Specifies the log directory.
• zookeeper.connect=zookeeper1:2181,zookeeper2:2181: Connects to the specified ZooKeeper instances.
• listeners=PLAINTEXT://:9093: Configures the broker to listen on port 9093.
• num.network.threads=4: Allocates four threads for network requests.
• num.io.threads=10: Allocates ten threads for I/O operations.

In this section, we covered the essential configurations for Kafka brokers, topics, producers, consumers, and ZooKeeper. Proper configuration is vital for the efficient and reliable operation of a Kafka cluster. We also provided a practical exercise to reinforce the concepts learned. In the next section, we will explore how to manage Kafka topics effectively.