In this project, we will explore how to leverage Hadoop for machine learning tasks. We will use the Hadoop ecosystem tools to preprocess data, train machine learning models, and evaluate their performance. This project will provide hands-on experience with integrating Hadoop with machine learning frameworks.

• Understand the integration of Hadoop with machine learning frameworks.
• Preprocess large datasets using Hadoop tools.
• Train and evaluate machine learning models on Hadoop.
• Implement a machine learning pipeline using Hadoop ecosystem tools.

• Basic understanding of Hadoop and its ecosystem.
• Familiarity with machine learning concepts.
• Knowledge of programming in Java or Python.

• Hadoop: For distributed storage and processing.
• Apache Mahout: A scalable machine learning library.
• Apache Spark: For distributed data processing and machine learning.
• HDFS: For storing large datasets.
• YARN: For resource management.

1. Install Hadoop: Ensure Hadoop is installed and configured on your system. Refer to Module 1 for setup instructions.
2. Install Apache Mahout: Follow the official Mahout installation guide.
3. Install Apache Spark: Follow the official Spark installation guide.

1. Load Data into HDFS:

   • Download a sample dataset (e.g., the Iris dataset).
   • Upload the dataset to HDFS using the following command:

     hdfs dfs -put iris.csv /user/hadoop/iris.csv
     

2. Data Cleaning with Apache Pig:

   • Write a Pig script to clean the data:

     -- Load the data
     raw_data = LOAD '/user/hadoop/iris.csv' USING PigStorage(',') 
                 AS (sepal_length:float, sepal_width:float, petal_length:float, petal_width:float, class:chararray);
     
     -- Filter out any invalid records
     clean_data = FILTER raw_data BY (sepal_length IS NOT NULL AND sepal_width IS NOT NULL 
                                      AND petal_length IS NOT NULL AND petal_width IS NOT NULL);
     
     -- Store the cleaned data
     STORE clean_data INTO '/user/hadoop/clean_iris' USING PigStorage(',');
     

   • Run the Pig script:

     pig -x mapreduce clean_data.pig
     

1. Using Apache Mahout:

   • Convert the cleaned data to Mahout's format:

     mahout seqdirectory -i /user/hadoop/clean_iris -o /user/hadoop/seq_iris
     mahout seq2sparse -i /user/hadoop/seq_iris -o /user/hadoop/sparse_iris
     

   • Train a k-means clustering model:

     mahout kmeans -i /user/hadoop/sparse_iris/tfidf-vectors -c /user/hadoop/clusters -o /user/hadoop/kmeans_output -k 3 -ow -cl
     

2. Using Apache Spark:

   • Write a Spark script to train a logistic regression model:

     from pyspark.sql import SparkSession
     from pyspark.ml.feature import VectorAssembler
     from pyspark.ml.classification import LogisticRegression
     from pyspark.ml.evaluation import MulticlassClassificationEvaluator
     
     # Initialize Spark session
     spark = SparkSession.builder.appName("IrisClassification").getOrCreate()
     
     # Load data
     data = spark.read.csv("/user/hadoop/clean_iris", header=False, inferSchema=True)
     data = data.withColumnRenamed("_c4", "label")
     
     # Assemble features
     assembler = VectorAssembler(inputCols=["_c0", "_c1", "_c2", "_c3"], outputCol="features")
     data = assembler.transform(data)
     
     # Split data into training and test sets
     train_data, test_data = data.randomSplit([0.7, 0.3])
     
     # Train logistic regression model
     lr = LogisticRegression(featuresCol="features", labelCol="label")
     model = lr.fit(train_data)
     
     # Make predictions
     predictions = model.transform(test_data)
     
     # Evaluate model
     evaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction", metricName="accuracy")
     accuracy = evaluator.evaluate(predictions)
     print(f"Test Accuracy: {accuracy}")
     
     # Stop Spark session
     spark.stop()
     

   • Run the Spark script:

     spark-submit iris_classification.py
     

• Evaluate the performance of the trained models using appropriate metrics (e.g., accuracy, precision, recall).
• Compare the results of different models and discuss their performance.

• Summarize the steps taken to preprocess data, train machine learning models, and evaluate their performance.
• Highlight the benefits of using Hadoop for machine learning tasks, such as scalability and efficiency.
• Discuss potential improvements and future work, such as experimenting with different algorithms or tuning hyperparameters.

In this project, we successfully integrated Hadoop with machine learning frameworks to preprocess data, train models, and evaluate their performance. We used Apache Mahout and Apache Spark to demonstrate different approaches to machine learning with Hadoop. This hands-on experience provided valuable insights into the capabilities and advantages of using Hadoop for large-scale machine learning tasks.