Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis. Hive allows users to read, write, and manage large datasets residing in distributed storage using SQL. It abstracts the complexity of Hadoop's MapReduce framework and provides a simple SQL-like interface called HiveQL.

1. HiveQL (Hive Query Language): A SQL-like language for querying data stored in Hadoop.
2. Tables: Structured data storage in Hive, similar to tables in a relational database.
3. Partitions: A way to divide tables into parts based on the values of a particular column.
4. Buckets: Further division of data in a table into more manageable parts.
5. Metastore: A central repository that stores metadata about the tables, partitions, and other data structures.

Hive architecture consists of the following main components:

1. User Interface (UI): Allows users to submit queries and other operations to the system.
2. Driver: Manages the lifecycle of a HiveQL statement, including query compilation, optimization, and execution.
3. Compiler: Converts HiveQL statements into a directed acyclic graph (DAG) of MapReduce jobs.
4. Metastore: Stores metadata about the tables, columns, partitions, and data types.
5. Execution Engine: Executes the compiled query using Hadoop's MapReduce framework.

• Hadoop cluster (single-node or multi-node)
• Java Development Kit (JDK)
• Apache Hive binaries

1. Download Apache Hive:

   wget https://downloads.apache.org/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
   tar -xzvf apache-hive-3.1.2-bin.tar.gz
   

2. Set Environment Variables:

   export HIVE_HOME=/path/to/apache-hive-3.1.2-bin
   export PATH=$PATH:$HIVE_HOME/bin
   

3. Configure Hive: Create a hive-site.xml file in the $HIVE_HOME/conf directory with the following basic configuration:

   <configuration>
       <property>
           <name>javax.jdo.option.ConnectionURL</name>
           <value>jdbc:derby:;databaseName=metastore_db;create=true</value>
           <description>JDBC connect string for a JDBC metastore</description>
       </property>
       <property>
           <name>javax.jdo.option.ConnectionDriverName</name>
           <value>org.apache.derby.jdbc.EmbeddedDriver</value>
           <description>Driver class name for a JDBC metastore</description>
       </property>
       <property>
           <name>hive.metastore.warehouse.dir</name>
           <value>/user/hive/warehouse</value>
           <description>location of default database for the warehouse</description>
       </property>
   </configuration>
   

4. Initialize the Metastore:

   schematool -initSchema -dbType derby
   

5. Start Hive:

   hive
   

Exercise: Create a Hive table to store information about books and perform some basic queries.

1. Create a Database:

   CREATE DATABASE library;
   

2. Use the Database:

   USE library;
   

3. Create a Table:

   CREATE TABLE books (
       id INT,
       title STRING,
       author STRING,
       year INT,
       genre STRING
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   STORED AS TEXTFILE;
   

4. Load Data into the Table:

   LOAD DATA LOCAL INPATH '/path/to/books.csv' INTO TABLE books;
   

5. Query the Data:

   SELECT * FROM books WHERE year > 2000;
   

1. Create a Database:

   CREATE DATABASE library;
   

2. Use the Database:

   USE library;
   

3. Create a Table:

   CREATE TABLE books (
       id INT,
       title STRING,
       author STRING,
       year INT,
       genre STRING
   )
   ROW FORMAT DELIMITED
   FIELDS TERMINATED BY ','
   STORED AS TEXTFILE;
   

4. Load Data into the Table:

   LOAD DATA LOCAL INPATH '/path/to/books.csv' INTO TABLE books;
   

5. Query the Data:

   SELECT * FROM books WHERE year > 2000;
   

In this section, we covered the basics of Apache Hive, including its architecture, setup, and basic HiveQL commands. We also provided a practical exercise to reinforce the learned concepts. In the next module, we will explore Apache HBase, another important tool in the Hadoop ecosystem.