In this section, we will explore the various commands used to interact with the Hadoop Distributed File System (HDFS). These commands are essential for managing files and directories within HDFS, and they are similar to Unix file system commands. Understanding these commands is crucial for effectively working with HDFS.

1. HDFS Shell: The HDFS shell is a command-line interface that allows users to interact with HDFS.
2. Basic Commands: Commands for basic file operations such as creating directories, copying files, and listing directory contents.
3. Advanced Commands: Commands for more complex operations like changing file permissions, checking file integrity, and managing file replication.

• Command: hdfs dfs -ls [path]
• Description: Lists the contents of a directory.
• Example:

  hdfs dfs -ls /user/hadoop
  

  This command lists all files and directories under /user/hadoop.

• Command: hdfs dfs -mkdir [path]
• Description: Creates a new directory in HDFS.
• Example:

  hdfs dfs -mkdir /user/hadoop/newdir
  

  This command creates a new directory named newdir under /user/hadoop.

• Command: hdfs dfs -put [local_path] [hdfs_path]
• Description: Copies a file from the local file system to HDFS.
• Example:

  hdfs dfs -put /local/path/to/file.txt /user/hadoop/
  

  This command copies file.txt from the local file system to /user/hadoop/ in HDFS.

• Command: hdfs dfs -get [hdfs_path] [local_path]
• Description: Copies a file from HDFS to the local file system.
• Example:

  hdfs dfs -get /user/hadoop/file.txt /local/path/
  

  This command copies file.txt from HDFS to the local file system.

• Command: hdfs dfs -rm [path]

• Description: Removes a file from HDFS.

• Example:

  hdfs dfs -rm /user/hadoop/file.txt
  

  This command removes file.txt from /user/hadoop/.

• Command: hdfs dfs -rm -r [path]

• Description: Removes a directory and its contents from HDFS.

• Example:

  hdfs dfs -rm -r /user/hadoop/newdir
  

  This command removes the directory newdir and all its contents from /user/hadoop/.

• Command: hdfs dfs -chmod [permissions] [path]
• Description: Changes the permissions of a file or directory.
• Example:

  hdfs dfs -chmod 755 /user/hadoop/file.txt
  

  This command sets the permissions of file.txt to 755.

• Command: hdfs dfs -chown [owner][:group] [path]
• Description: Changes the owner and group of a file or directory.
• Example:

  hdfs dfs -chown hadoop:hadoop /user/hadoop/file.txt
  

  This command changes the owner and group of file.txt to hadoop.

• Command: hdfs dfs -checksum [path]
• Description: Displays the checksum of a file.
• Example:

  hdfs dfs -checksum /user/hadoop/file.txt
  

  This command displays the checksum of file.txt.

• Command: hdfs dfs -setrep [replication_factor] [path]
• Description: Sets the replication factor of a file.
• Example:

  hdfs dfs -setrep 3 /user/hadoop/file.txt
  

  This command sets the replication factor of file.txt to 3.

1. Create a directory: Create a directory named testdir under /user/hadoop/.

   hdfs dfs -mkdir /user/hadoop/testdir
   

2. Copy a file to HDFS: Copy a local file example.txt to the newly created directory.

   hdfs dfs -put /local/path/example.txt /user/hadoop/testdir/
   

3. List the contents: List the contents of testdir.

   hdfs dfs -ls /user/hadoop/testdir
   

4. Remove the file: Remove example.txt from testdir.

   hdfs dfs -rm /user/hadoop/testdir/example.txt
   

1. Change permissions: Change the permissions of a file example.txt to 644.

   hdfs dfs -chmod 644 /user/hadoop/example.txt
   

2. Change ownership: Change the owner of example.txt to hadoopuser.

   hdfs dfs -chown hadoopuser /user/hadoop/example.txt
   

3. Set replication factor: Set the replication factor of example.txt to 2.

   hdfs dfs -setrep 2 /user/hadoop/example.txt
   

• Incorrect Path: Ensure that the paths specified in the commands are correct. A common mistake is to use an incorrect path, leading to errors.
• Permissions: Be mindful of file and directory permissions. Lack of proper permissions can prevent you from performing certain operations.
• Replication Factor: Setting a very high replication factor can lead to unnecessary use of storage space. Use an appropriate replication factor based on your requirements.

In this section, we covered the essential HDFS commands for managing files and directories within the Hadoop Distributed File System. These commands are fundamental for interacting with HDFS and performing various file operations. By practicing these commands, you will gain confidence in managing data within HDFS, which is a critical skill for working with Hadoop. In the next section, we will delve into the architecture of HDFS to understand how it stores and manages data.