The Project | About Us | Contribute | Donations | License
HOME My courses Completed
◄ Previous
◄

Hadoop vs Traditional Databases

Next ►
►

In this section, we will explore the key differences between Hadoop and traditional databases. Understanding these differences is crucial for determining when to use Hadoop over traditional database systems.

Key Concepts

  1. Data Storage and Processing
  2. Scalability
  3. Data Types and Schema
  4. Fault Tolerance
  5. Cost Efficiency
  6. Use Cases

  1. Data Storage and Processing

Traditional Databases

  • Storage: Typically use structured data stored in tables with rows and columns.
  • Processing: Relational databases use SQL (Structured Query Language) for data manipulation and querying.
  • Architecture: Centralized architecture where data is stored on a single server or a small cluster of servers.

Hadoop

  • Storage: Uses HDFS (Hadoop Distributed File System) to store large volumes of unstructured, semi-structured, and structured data across multiple nodes.
  • Processing: Utilizes the MapReduce programming model for distributed data processing.
  • Architecture: Distributed architecture where data is stored and processed across a large cluster of commodity hardware.

  1. Scalability

Traditional Databases

  • Vertical Scalability: Scaling up by adding more resources (CPU, RAM) to a single server.
  • Limitations: Limited by the capacity of the server hardware.

Hadoop

  • Horizontal Scalability: Scaling out by adding more nodes to the cluster.
  • Advantages: Can handle petabytes of data by simply adding more nodes to the cluster.

  1. Data Types and Schema

Traditional Databases

  • Schema-on-Write: Requires a predefined schema before data can be inserted.
  • Data Types: Best suited for structured data with fixed schema.

Hadoop

  • Schema-on-Read: Allows data to be stored without a predefined schema. The schema is applied when the data is read.
  • Data Types: Can handle a variety of data types including structured, semi-structured, and unstructured data.

  1. Fault Tolerance

Traditional Databases

  • Fault Tolerance: Typically rely on RAID (Redundant Array of Independent Disks) and database replication for fault tolerance.
  • Recovery: Manual intervention is often required for recovery.

Hadoop

  • Fault Tolerance: Built-in fault tolerance through data replication across multiple nodes.
  • Recovery: Automatic recovery from node failures without manual intervention.

  1. Cost Efficiency

Traditional Databases

  • Cost: Often require expensive, high-end hardware and proprietary software licenses.
  • Maintenance: Higher maintenance costs due to specialized hardware and software.

Hadoop

  • Cost: Designed to run on commodity hardware, reducing hardware costs.
  • Open Source: Hadoop is open-source, which eliminates software licensing costs.

  1. Use Cases

Traditional Databases

  • Use Cases: Best suited for OLTP (Online Transaction Processing) systems, where quick read and write operations are required.
  • Examples: Banking systems, e-commerce websites, and inventory management systems.

Hadoop

  • Use Cases: Ideal for OLAP (Online Analytical Processing) systems, big data analytics, and batch processing.
  • Examples: Log analysis, recommendation systems, and data warehousing.

Comparison Table

Feature Traditional Databases Hadoop
Data Storage Structured data in tables Unstructured, semi-structured, and structured data in HDFS
Processing SQL MapReduce
Scalability Vertical Horizontal
Schema Schema-on-Write Schema-on-Read
Fault Tolerance RAID, replication Data replication across nodes
Cost High (expensive hardware and licenses) Low (commodity hardware, open-source)
Use Cases OLTP OLAP, big data analytics

Practical Example

Traditional Database Query (SQL)

SELECT customer_id, customer_name, total_purchases
FROM customers
WHERE total_purchases > 1000;
  • This query retrieves customer details from a relational database where the total purchases exceed 1000.

Hadoop MapReduce Example

// Mapper Class
public class PurchaseMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private Text customerId = new Text();
    private IntWritable purchaseAmount = new IntWritable();

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String[] fields = value.toString().split(",");
        customerId.set(fields[0]);
        purchaseAmount.set(Integer.parseInt(fields[2]));
        context.write(customerId, purchaseAmount);
    }
}

// Reducer Class
public class PurchaseReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int totalPurchases = 0;
        for (IntWritable val : values) {
            totalPurchases += val.get();
        }
        if (totalPurchases > 1000) {
            context.write(key, new IntWritable(totalPurchases));
        }
    }
}
  • This MapReduce program processes a large dataset of customer purchases, summing up the total purchases for each customer and filtering those with total purchases greater than 1000.

Practical Exercise

Exercise

  1. Task: Write a SQL query to find all customers who have made purchases in the last month.
  2. Task: Write a MapReduce program to count the number of purchases made by each customer in the last month.

Solution

SQL Query

SELECT customer_id, COUNT(*) as purchase_count
FROM purchases
WHERE purchase_date >= DATE_SUB(CURDATE(), INTERVAL 1 MONTH)
GROUP BY customer_id;

MapReduce Program

// Mapper Class
public class LastMonthPurchaseMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
    private Text customerId = new Text();
    private final static IntWritable one = new IntWritable(1);
    private static final SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy-MM-dd");

    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String[] fields = value.toString().split(",");
        String purchaseDate = fields[1];
        try {
            Date date = dateFormat.parse(purchaseDate);
            Calendar cal = Calendar.getInstance();
            cal.add(Calendar.MONTH, -1);
            if (date.after(cal.getTime())) {
                customerId.set(fields[0]);
                context.write(customerId, one);
            }
        } catch (ParseException e) {
            e.printStackTrace();
        }
    }
}

// Reducer Class
public class LastMonthPurchaseReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
    public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
        int purchaseCount = 0;
        for (IntWritable val : values) {
            purchaseCount += val.get();
        }
        context.write(key, new IntWritable(purchaseCount));
    }
}

Conclusion

In this section, we have compared Hadoop with traditional databases across various dimensions such as data storage, processing, scalability, schema, fault tolerance, cost efficiency, and use cases. We also provided practical examples and exercises to illustrate the differences. Understanding these differences will help you make informed decisions about when to use Hadoop over traditional database systems. In the next module, we will delve deeper into the Hadoop architecture and its core components.

◄ Previous
◄
Next ►
►
The Project | About Us | Contribute | Donations | License
© Copyright 2024. All rights reserved