Cloud Dataproc is a fast, easy-to-use, fully managed cloud service for running Apache Spark and Apache Hadoop clusters. It allows you to process large datasets quickly and efficiently by leveraging the power of Google Cloud Platform (GCP).

1. Clusters: A collection of virtual machines (VMs) that run Hadoop and Spark jobs.
2. Jobs: Tasks that you submit to the cluster for processing.
3. Workflow Templates: Predefined workflows that automate the process of running jobs on clusters.
4. Autoscaling: Automatically adjusts the number of worker nodes in a cluster based on the workload.

• Speed: Quickly create and manage clusters.
• Cost-Effective: Pay only for what you use.
• Scalability: Easily scale your clusters up or down.
• Integration: Seamlessly integrates with other GCP services like BigQuery, Cloud Storage, and more.

1. Go to the GCP Console.
2. Click on the project drop-down and select "New Project".
3. Enter a project name and click "Create".

1. In the GCP Console, navigate to the "APIs & Services" > "Library".
2. Search for "Cloud Dataproc API".
3. Click "Enable".

1. In the GCP Console, navigate to "Dataproc" > "Clusters".
2. Click "Create Cluster".
3. Configure the cluster settings:
   • Cluster Name: Enter a name for your cluster.
   • Region: Select a region close to your data.
   • Zone: Select a zone within the region.
   • Cluster Mode: Choose "Standard" for a multi-node cluster or "Single Node" for a single-node cluster.
4. Click "Create".

1. Upload Your Data: Upload your data to a GCP Cloud Storage bucket.
2. Submit a Job:

   gcloud dataproc jobs submit spark 
          --cluster=<CLUSTER_NAME> 
          --region=<REGION> 
          --class=org.apache.spark.examples.SparkPi 
          --jars=file:///usr/lib/spark/examples/jars/spark-examples.jar 
          -- 1000
   

   • Replace <CLUSTER_NAME> with the name of your cluster.
   • Replace <REGION> with the region of your cluster.

1. In the GCP Console, navigate to "Dataproc" > "Jobs".
2. Click on the job you submitted to view its details and logs.

1. Create a Cluster:

   • Follow the steps in "Setting Up Cloud Dataproc" to create a new cluster.

2. Upload Data:

   • Upload a sample dataset to a Cloud Storage bucket.

3. Submit a Hadoop Job:

   gcloud dataproc jobs submit hadoop 
          --cluster=<CLUSTER_NAME> 
          --region=<REGION> 
          --jar=file:///usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar 
          -- wordcount gs://<BUCKET_NAME>/<INPUT_FILE> gs://<BUCKET_NAME>/<OUTPUT_DIR>
   

   • Replace <CLUSTER_NAME> with the name of your cluster.
   • Replace <REGION> with the region of your cluster.
   • Replace <BUCKET_NAME> with the name of your Cloud Storage bucket.
   • Replace <INPUT_FILE> with the path to your input file.
   • Replace <OUTPUT_DIR> with the path to your output directory.

1. Create a Cluster:

   • Follow the steps provided in the "Setting Up Cloud Dataproc" section.

2. Upload Data:

   • Use the GCP Console or gsutil command to upload your data:

     gsutil cp local-file.txt gs://your-bucket-name/input-file.txt
     

3. Submit a Hadoop Job:

   gcloud dataproc jobs submit hadoop 
          --cluster=my-cluster 
          --region=us-central1 
          --jar=file:///usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar 
          -- wordcount gs://my-bucket/input-file.txt gs://my-bucket/output-dir
   

• Cluster Configuration: Ensure that your cluster is properly configured with the necessary resources (CPU, memory) to handle your job.
• Data Location: Make sure your data is in the same region as your cluster to minimize latency and costs.
• Job Submission: Double-check the syntax and parameters when submitting jobs to avoid errors.

In this section, you learned about Cloud Dataproc, its key concepts, and how to set up and run jobs on a Dataproc cluster. You also completed a practical exercise to reinforce your understanding. In the next module, we will explore other data and analytics services offered by GCP, such as BigQuery and Cloud Dataflow.