Search This Blog

Enabling debug logs in Apache Hadoop and Hortonworks Data Platform


To trouble shoot hadoop issues we need to have debug logs to get more low level errors.by default debug logs are not enabled in Hortonworks data platform and in plain hadoop also.
In this post,we will discuss how to enable debug logs in HDP and plain hadoop.

1. Modify /var/lib/ambari-server/resources/stacks/HDP/2.0.6/hooks/after-INSTALL/templates/hadoop-env.sh.j2


In HDP,we need to add below line to hadoop-env.sh.j2 to enable debug logs on HDFS services.


export HADOOP_ROOT_LOGGER=DEBUG,console


If you are using Plain hadoop you can directly add above line to /etc/hadoop/conf/hadoop-env.sh file in all nodes and restart HDFS deamons on all nodes.
If you are using Hortonworks Data Platform ,You need to follow below steps.

2. Restart Ambari agents on all nodes


We need to restart ambari agent on all master nodes,data nodes and edge nodes if any. We need to run below command for restarting.

service ambari-agent restart.

Or we can stop and start ambari agent.

service ambari-agent stop
service ambari-agent start


3. Restart Ambari server


We also need to restart ambari server on first master.We can run below command.

service ambari-server restart

Or we can stop and start ambari server.

service ambari-server stop
service ambari-server start

4. Restart HDFS services


In Ambari UI ,Click on HDFS , click service actions drop down and select restart all option.It will ask for confirmation once you confirm, it will restart all HDFS daemons.

Once debug logs are enabled you can check them in name node logs and data node logs.Debug logs consume lot of space you need to disable them once you collect required logs. to disable remove added export command and restart above mentioned services.