Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It is designed to handle large volumes of data and can be used to stream data from various sources to centralized data stores such as HDFS (Hadoop Distributed File System).

• Reliability: Ensures data is reliably transferred from source to destination.
• Scalability: Can handle large volumes of data and scale horizontally.
• Flexibility: Supports various data sources and destinations.
• Extensibility: Allows custom implementations for sources, sinks, and channels.

Apache Flume's architecture is based on a simple and flexible model of data flows. The main components of Flume are:

1. Source: The component that receives data from an external source.
2. Channel: The conduit between the source and the sink, acting as a buffer.
3. Sink: The component that delivers data to the final destination.

1. Event: A unit of data with a byte payload and optional headers.
2. Agent: A JVM process that hosts the source, channel, and sink.
3. Flow: The path data takes from the source to the sink through the channel.

• Java Development Kit (JDK) installed.
• Apache Flume binary downloaded from the official website.

1. Download and Extract Flume:

   wget https://downloads.apache.org/flume/1.9.0/apache-flume-1.9.0-bin.tar.gz
   tar -xzf apache-flume-1.9.0-bin.tar.gz
   cd apache-flume-1.9.0-bin
   

2. Set Environment Variables:

   export FLUME_HOME=/path/to/apache-flume-1.9.0-bin
   export PATH=$FLUME_HOME/bin:$PATH
   

3. Verify Installation:

   flume-ng version
   

A Flume configuration file defines the sources, channels, and sinks. Below is an example configuration file (flume.conf):

To start the Flume agent with the above configuration:

1. Configuration File (flume-hdfs.conf):

   agent1.sources = source1
   agent1.channels = channel1
   agent1.sinks = sink1
   
   agent1.sources.source1.type = exec
   agent1.sources.source1.command = tail -F /var/log/syslog
   
   agent1.channels.channel1.type = memory
   agent1.channels.channel1.capacity = 1000
   agent1.channels.channel1.transactionCapacity = 100
   
   agent1.sinks.sink1.type = hdfs
   agent1.sinks.sink1.hdfs.path = hdfs://localhost:9000/user/flume/logs/
   agent1.sinks.sink1.hdfs.fileType = DataStream
   agent1.sinks.sink1.hdfs.writeFormat = Text
   agent1.sinks.sink1.hdfs.batchSize = 1000
   agent1.sinks.sink1.hdfs.rollSize = 0
   agent1.sinks.sink1.hdfs.rollCount = 10000
   

2. Start the Flume Agent:

   flume-ng agent --conf ./conf --conf-file flume-hdfs.conf --name agent1 -Dflume.root.logger=INFO,console
   

1. Task: Set up a Flume agent to read data from a local file and print it to the console.
2. Steps:
   • Create a configuration file to define the source as a file source.
   • Define the sink as a logger.
   • Start the Flume agent and verify the data flow.

1. Task: Configure a Flume agent to stream data from a network source to HDFS.
2. Steps:
   • Create a configuration file to define the source as a netcat source.
   • Define the sink as HDFS.
   • Start the Flume agent and verify the data is written to HDFS.

In this section, we covered the basics of Apache Flume, including its architecture, setup, and configuration. We also provided practical examples and exercises to help you get hands-on experience with Flume. Understanding Flume is crucial for efficiently collecting and transporting large volumes of data in a Hadoop ecosystem. In the next module, we will explore another powerful tool in the Hadoop ecosystem: Apache Oozie.