Starting HDFS,MAPREDUCE2 and YARN processes manually

We start HDFS,MAPREDUCE2 and YARN services using either Cloudera manager or Apache Ambari if we use CDH or HDP.Many a times Cloudera manager and Apache Ambari do not display complete error message if services fail to start.We can go to log directories and search for startup errors. Searching startup errors in log files is also not easy as log files are huge.

One easy way to find startup errors is starting processes manually from command line to locate errors easily on the console.

In this article we will learn how to start HDFS, MAPREDUCE2 and YARN services manually.

HDFS (Hadoop Distributed File System)

HDFS has processes data node, name node, Zookeeper failover controller and Journal nodes.
We will learn how to start them manually.


Starting datanode manually

We can start data node manually using HDFS datanode command. This needs to be run as hdfs user.

Command:

hdfs datanode &

Starting Namenode manually

We can start name node manually using HDFS namenode command. This also needs to be run as hdfs user.

Command :

HDFS namenode &
















Starting Zookeeper failover controller manually

We can start zookeeper failover controller manually using hdfs zkfc command.

Command :

hdfs zkfc &

Starting Journal node manually

We can start journal node manually using hdfs journalnode command. This also needs to be run as hdfs user.

Command :

hdfs journalnode &

MAPREDUCE2

Mapreduce2 has history server process, We can start history server process using mapred historyserver command. It is better to run as mapred user.

Command :

mapred historyserver &

YARN (Yet Another Resource Negotiator)

Yarn has node manager, resource manger and app timeline server processes.

Starting resource manager manually

We can start resource manager using yarn resource manager command. It is recommened to run as yarn user.

Command :

yarn resourcemanager &

Staring nodemanager manually

We can start node manager manually using yarn nodemanager command. This command also needs to be run as yarn user.

Command :

yarn nodemanager &

Staring app timeline server manually

We can start app timeline server manually using yarn timelineserver command. This command also needs to be run as yarn user.

Command :

yarn timelineserver &

These commands are useful only for trouble shooting startup errors, Apache ambari and Cloudera manager may not recognize your services if you start manually using above commands.

Happy Hadooping !!!!