Hadoop is an open-source framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from a single server to thousands of machines, each offering local computation and storage.

1. Big Data: Refers to extremely large data sets that may be analyzed computationally to reveal patterns, trends, and associations, especially relating to human behavior and interactions.
2. Distributed Computing: A field of computer science that studies distributed systems. A distributed system is a system whose components are located on different networked computers, which communicate and coordinate their actions by passing messages to one another.
3. Scalability: The capability of a system, network, or process to handle a growing amount of work, or its potential to accommodate growth.

Hadoop consists of four main modules:

1. Hadoop Common: The common utilities that support the other Hadoop modules.
2. Hadoop Distributed File System (HDFS): A distributed file system that provides high-throughput access to application data.
3. Hadoop YARN: A framework for job scheduling and cluster resource management.
4. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

• Scalability: Hadoop can easily scale from a single server to thousands of machines.
• Cost-Effective: It uses commodity hardware, which is cheaper than specialized hardware.
• Flexibility: Hadoop can process structured, semi-structured, and unstructured data.
• Fault Tolerance: Data is automatically replicated across multiple nodes, ensuring data availability even if some nodes fail.

• Complexity: Setting up and managing a Hadoop cluster can be complex.
• Latency: Hadoop is designed for batch processing and may not be suitable for real-time data processing.
• Resource Intensive: Requires significant computational resources and storage.

Let's look at a simple example to understand how Hadoop works. Suppose you have a large text file containing millions of lines, and you want to count the number of occurrences of each word.

In a traditional approach, you might write a program that reads the file line by line, splits each line into words, and then counts the occurrences of each word. This approach works fine for small files but becomes impractical for very large files due to memory and processing constraints.

With Hadoop, you can distribute the file across multiple nodes in a cluster and process each part of the file in parallel. Here's a simplified version of how this would work using the MapReduce framework:

1. Map Phase: Each node processes a portion of the file and produces a list of key-value pairs (word, 1).
2. Shuffle and Sort Phase: The framework sorts and groups the key-value pairs by key (word).
3. Reduce Phase: Each node processes the grouped key-value pairs and sums the values to get the total count for each word.

Here is a simple MapReduce program in Java to count word occurrences:

• TokenizerMapper: This class extends the Mapper class and overrides the map method. It splits each line into words and writes each word with a count of 1 to the context.
• IntSumReducer: This class extends the Reducer class and overrides the reduce method. It sums the counts for each word and writes the result to the context.
• Main Method: Configures and runs the MapReduce job.

In this section, we introduced Hadoop, its core components, and its advantages and disadvantages. We also provided a practical example of how Hadoop can be used to process large data sets efficiently. Understanding these basics will prepare you for more advanced topics in the subsequent modules.