Hadoop Streaminng is a utility that comes with hadoop and it allows you to use any executable program for bigdata analysis.
We can use languages like Java,Python,PHP ,Scala ,Perl and many more .It also supports Unix commands and Shell scripts.
I was using hadoop streaming with unix or shell script extensively and I enjoyed it for several reasons.
I would like to share benefits of hadoop streaming with unix.
If you want to use a tool/technology other than mapreduce,you may be running for hive or pig .if you go for Hive or Pig you have to install and manage them separately (if you use vendor hadoop you will get it by default) .otherwise you could use hadoop streaming with Unix which you need not install it separately .
You need not learn new tool /technology like Hive or Pig if you do not have serious requirement.You can leverage your Unix skills for data analysis on hadoop .
3. Less development time
for developing java Mapreduce applications,you have to compile your code ,unit test it,package it,export jar file and run it finally. unlike Java Mapreduce you can quickly develop streaming applications with Unix by directly writing mapper and reducer code in mapper and reducer options
4. Quick conversion
As I said It has less development time, we can quickly convert data from one format to another.I heavily used it for converting data from text to sequence file and sequence file to text .we can use inputformat and outputformat options in hadoop streaming for the same.
5. Testing data
for the same reason, as I said, it has less development time we can quickly test the input data and output data by using hadoop streaming with Unix or shell script
6. Simple business requirement
for simple business requirement , we can always use Hadoop streaming with Unix .like for simple filtering operations and simple aggregation operations.
finally ,I read it somewhere hadoop streaming with Unix has better performance over mapreduce ,Hive and Pig.I personally not tested it though.
So try to use hadoop streaming with Unix if you have any above requirements.
for more details on how to use hadoop streaming with unix ,read it .
Happy Hadooping Friends.
Post a Comment