The Hadoop ecosystem is a suite of tools and technologies designed to work together to process and analyze large datasets. This ecosystem extends the capabilities of the core Hadoop components (HDFS, MapReduce, and YARN) by providing additional functionalities such as data storage, data processing, data analysis, and data management.

• Purpose: Provides scalable and reliable data storage.
• Functionality: Stores large files across multiple machines, ensuring fault tolerance and high availability.

• Purpose: A programming model for processing large datasets.
• Functionality: Breaks down tasks into smaller sub-tasks (Map) and then combines the results (Reduce).

• Purpose: Manages resources in a Hadoop cluster.
• Functionality: Allocates resources to various applications running in the cluster.

• Purpose: High-level platform for creating MapReduce programs.
• Functionality: Uses a scripting language called Pig Latin to simplify the coding process.

• Purpose: Data warehousing and SQL-like query language.
• Functionality: Allows users to query and manage large datasets using a SQL-like interface.

• Purpose: NoSQL database for real-time read/write access to large datasets.
• Functionality: Provides random, real-time access to data stored in HDFS.

• Purpose: Data transfer between Hadoop and relational databases.
• Functionality: Facilitates the import and export of data between Hadoop and structured data stores.

• Purpose: Data ingestion tool.
• Functionality: Collects, aggregates, and moves large amounts of log data from various sources to HDFS.

• Purpose: Workflow scheduler for Hadoop jobs.
• Functionality: Manages the execution of complex data processing workflows.

1. Start the Hive Shell:

   hive
   

2. Create a Database:

   CREATE DATABASE example_db;
   USE example_db;
   

3. Create a Table:

   CREATE TABLE employees (
       id INT,
       name STRING,
       age INT,
       department STRING
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   STORED AS TEXTFILE;
   

4. Load Data into the Table:

   LOAD DATA LOCAL INPATH '/path/to/employees.csv' INTO TABLE employees;
   

5. Query the Data:

   SELECT * FROM employees WHERE age > 30;
   

• Step 1: Start the Hive shell to interact with Hive.
• Step 2: Create a new database and switch to it.
• Step 3: Create a table named employees with columns for id, name, age, and department.
• Step 4: Load data from a local CSV file into the employees table.
• Step 5: Run a query to select all employees older than 30.

1. Task: Create a Hive table named sales with columns for transaction_id, product, quantity, and price. Load data from a CSV file and query the total sales amount for each product.
2. Solution:

   CREATE TABLE sales (
       transaction_id INT,
       product STRING,
       quantity INT,
       price FLOAT
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   STORED AS TEXTFILE;
   
   LOAD DATA LOCAL INPATH '/path/to/sales.csv' INTO TABLE sales;
   
   SELECT product, SUM(quantity * price) AS total_sales
   FROM sales
   GROUP BY product;
   

1. Task: Write a Pig script to load a dataset, filter records where the age is greater than 25, and store the results.
2. Solution:

   -- Load the dataset
   data = LOAD '/path/to/data.csv' USING PigStorage(',') AS (id:int, name:chararray, age:int, department:chararray);
   
   -- Filter records where age > 25
   filtered_data = FILTER data BY age > 25;
   
   -- Store the results
   STORE filtered_data INTO '/path/to/output' USING PigStorage(',');
   

In this section, we explored the Hadoop ecosystem and its key components. We learned about the purpose and functionality of each tool and saw practical examples of using Apache Hive and Apache Pig. Understanding the Hadoop ecosystem is crucial for effectively managing and processing large datasets. In the next module, we will delve deeper into the architecture of Hadoop and its core components.