Search This Blog

HDFS Commands examples

This article cover simple and useful HDFS commands examples. Apache Hadoop is data processing tool at web scale. Hadoop contains three modules :

HDFS                          ---   Data storage system

MAPREDUCE           ---    Data processing framework

YARN                         ---    Resource management system for Hadoop


We will discuss some commands to learn how to interact with Hadoop distributed File System (HDFS). All hdfs file system commands start with hdfs dfs.  Most of the hadoop distributions  (CDH, HDP) come with standard hdfs user.

1)

Change the current user  root to HDFS user. Mostly hdfs user will be password less user.









2).



Create a file on the local file system using cat command.




3).


Create a directory in HDFS using mkdir command.





4).


Upload local file helloworld to HDFS directory helloworld using put command.



5).


Check file is loaded into hdfs directory helloworld using ls command.







6).


Read hdfs file content using cat command.







7).


Rename HDFS file helloworld to helloworldfile using mv command.









8)

Copy helloworldfile to another hdfs directory using cp command.












9)

Check the size of a file in HDFS using du command.





10).


Check replication factor of a file in HDFS using ls command. Rounded number is the replication factor of the file.



11).

Change the replication factor of a file in HDFS using setrep command.The example below changes replication factor from 3 to 2.







1 comment:

  1. Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. big data projects for students But it’s not the amount of data that’s important.Project Center in Chennai

    Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Corporate TRaining Spring Framework the authors explore the idea of using Java in Big Data platforms.

    Spring Training in Chennai

    The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

    ReplyDelete