Dec 27, 2009

A WordCount Tutorial for Hadoop 0.20.1

Because the document of Hadoop 0.20.1 describes a tutorial program which uses out-of-date APIs, I decided to write the following tutorial for Hadoop 0.20.1. It is notable that in 0.20.1, org.apache.hadoop.mapred.* are deprecated and it is recommended to use org.apache.hadoop.mapreduce.*. This tutorial is based on the new API.

For how to install and configure Hadoop, you might want to refer to my previous post. After Hadoop is installed, let us create a source code directory and put the following Java source file:
package org.sogou;

import java.io.IOException;
import java.lang.InterruptedException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;

public class WordCount {
/**
* The map class of WordCount.
*/

public static class TokenCounterMapper
extends Mapper<Object, Text, Text, IntWritable> {

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}
/**
* The reducer class of WordCount
*/

public static class TokenCounterReducer
extends Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
context.write(key, new IntWritable(sum));
}
}
/**
* The main entry point.
*/

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();
Job job = new Job(conf, "Example Hadoop 0.20.1 WordCount");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenCounterMapper.class);
job.setReducerClass(TokenCounterReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}
Then, we build this file and pack the result into a jar file:
mkdir classes
javac -classpath /Users/wyi/hadoop-0.20.1/hadoop-0.20.1-core.jar:/Users/wyi/hadoop-0.20.1//lib/commons-cli-1.2.jar -d classes WordCount.java && jar -cvf wordcount.jar -C classes/ .
Finally, we run the jar file in standalone mode of Hadoop
echo "hello world bye world" > /Users/wyi/tmp/in/0.txt
echo "hello hadoop goodebye hadoop" > /Users/wyi/tmp/in/1.txt
hadoop jar wordcount.jar org.sogou.WordCount /Users/wyi/tmp/in /Users/wyi/tmp/out

86 comments:

yipeng said...
This comment has been removed by the author.
Andrew said...

Thanks! Great post, thanks for helping us newbies get started using the right APIs! Exactly what I was looking for.

Neil said...

Thank you , It was usefull

Philippe Adjiman said...

Thanks for that code! I've updated an hadoop tutorial (http://philippeadjiman.com/blog/2009/12/07/hadoop-tutorial-part-1-setting-up-your-mapreduce-learning-playground/) that I wrote one year ago with a link to your post.

Best,
-Philippe.

Field Cady said...

Thanks so much for the post! On my computer, however, it is failing with the following error:

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.sogou.WordCount$TokenCounterMapper

There was also a warning:

11/02/21 15:57:31 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

Do you know what might be happening? I came to your post because I'm having the same problem with the standard tutorials, and hoped it was due to the API problem you references. Thank you in advance for any suggestions!

Field Cady said...

Thanks so much for the post! On my computer, however, it is failing with the following error:

java.lang.RuntimeException: java.lang.ClassNotFoundException: org.sogou.WordCount$TokenCounterMapper

There was also a warning:

11/02/21 15:57:31 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).

Do you know what might be happening? I came to your post because I'm having the same problem with the standard tutorials, and hoped it was due to the API problem you references. Thank you in advance for any suggestions!

shrish said...

Thanks a lot for the post..it really helped

shrish said...

Great post....ive been struggling for long to find this

shrish said...

Great post...dats wat I wanted ;)

Radhika said...

Thanks a lot..Great post...

Radhika said...
This comment has been removed by the author.
Arunraj Nair said...

How do I add use -libjars option and set a queue name?

I am using the command-
hadoop jar MyJar com.ClassName -Dmapred.job.queuename= -libjars jarPath

which is failing

Thanks,
Arun

Wonhui said...

I have a question on your code. Well, in the "main" function, in which part the JOB starts to be executed?
I mean, all the functions called in the main are about the "setting"(such as setMapperClass, setCombinerClass, setReducerClass, etc.), thus, i don't understand where the implementation gets started.

james said...

thanks for share nice information.great post!
hadoop online courses is having good demand in these days.

james said...

great post!!
thanks for the tips.hadoop online tutorial

Amith Nambiar said...

Had to setJarByClass to get that working with CDH5.

job.setJarByClass(WordCount.class)

Thank you for the post.

besantvignesh M said...

Hai author.That Excellent posts.i have learn to lot of hadoop.Thanks a lot.



Hadoop Training in Chennai



sundara rami reddy said...

This information was really very helpful for hadoopers I am going to share this with some of my class mates and friends coz I want them to see that this blog is having such nice content.
Hadoop Training in hyderabad

jack wilson said...

This website is very helpful for the students who need info about the Hadoop courses.i appreciate for your post. thanks for shearing it with us. keep it up.

Hadoop Course in Chennai

ramya parvathaneni said...

Hi,
The best information from you and we are providing online training with total modules
hadoop online training

divi m said...

Thanks a lot.i hope to understand to your information.hadoop training in chennai

Hadoop online training said...

Hi,
nice to share information and hadoop real time online training with real time experts on
hadoop online training
industry based projects

Joseph A. Wallace said...

Thanks for InformationHadoop Course will provide the basic concepts of MapReduce applications developed using Hadoop, including a close look at framework components, use of Hadoop for a variety of data analysis tasks, and numerous examples of Hadoop in action. This course will further examine related technologies such as Hive, Pig, and Apache Accumulo. HADOOP Online Training

Rajesh Ravi said...

Really is very interesting, I saw your website and get more details..Nice work. Thanks regards,
Refer this link below,
SAS Training in Chennai

Ram Charan said...

reviews-complaints-testimonials

vignesh m said...

Hi this is vignesh i am having 3 years of experience as a android developer and i am certified. i have knowledge on OOPS concepts in android but dont know indepth. After learning hadoop will be enough to get a good career in IT with good package? and i crossed hadoop training in chennai website where someone please help me to identity the syllabus covers everything or not??

Thanks,
vignesh

Victoria John said...

Hi I am Victoria lives in Chennai. I am a technology freak. Recently I did Java Training in Chennai at a leading Java Training Institutes in Chennai. This is really helpful for me to make a bright career in IT industry.

Roshini RS said...

Thanks for sharing your view to our knowledge’s, its helps me plenty keep sharing…
Best Informatica Training In Chennai

john son said...

Dot Net Training Chennai

Thanks for your wonderful post.It is really very helpful for us and I have gathered some important information from this blog.If anyone wants to get Dot Net Training in Chennai reach FITA, rated as No.1 Dot Net Training Institute in Chennai.

Dot Net Course in Chennai


Andria BZ said...

Thanks for sharing these niche piece of coding to our knowledge. Here, I had a solution for my inconclusive problems & it’s really helps me a lot keep updates…
sas training in Chennai

Martina Christy said...

Testing Training in Chennai

Hi, I have read your blog. Your information is really useful for me.Thanks for sharing this blog. I did QTP Course in Chennai at Fita training and placement academy which offer best Selenium Course in Chennai with years of experienced professionals. This is really useful for me to make a bright career.

rebeka christy said...

PHP Training Chennai

I get a lot of great information from this blog. Thank you for your sharing this informative blog. Recently I did PHP course at a leading academy. If you are looking for best PHP Training Center in Chennai visit FITA IT training academy which offer real time and Best PHP Training in Chennai.

PHP Course in Chennai

jack wilson said...

Automation Training in Chennai

Its really awesome blog..If anyone wants to get Software Testing Training in Chennai visit FITA IT academy located at Chennai. Rated as No.1 Software Testing Training Institutes in Chennai

Software Testing Course in Chennai

jack wilson said...

QTP Course in Chennai

Hi, I wish to be a regular contributor of your blog. I have read your blog. Your information is really useful for beginner. I did Testing Training in Chennai at Fita training and placement academy which offer best Software Testing Training in Chennai with years of experienced professionals. This is really useful for me to make a bright career.

Regards...

Software Testing Training Institutes in Chennai

Emi Jackson said...

HTML5 Training

Hi, Thanks for sharing this valuable blog.I was really impressed by reading this blog. I did HTML5 Training in Chennai at reputed HTML5 Training Institutes in Chennai. This is really useful for me to make a bright future in designing field.

Best HTML5 Training in Chennai

Roshini RS said...


The information you have given here is truly helpful to me. sas training in Chennai

dhanamlakshmi palu said...

Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me.. AWS Training in chennai | AWS Training chennai | AWS course in chennai

Amirtha rao said...

Hi Yi Wang ,
Thanks for this informative post, this piece of coding was error free...
sas course in Chennai

gokul saran said...

Thanks for sharing such informative article on Load runner Automation testing tool. This load testing tool will provide most precise information about the quality of software.Hadoop Courses in Chennai

surangacloud said...

Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me.. cloud computing training in chennai | cloud computing training chennai | cloud computing course in chennai | cloud computing course chennai

dhanalakshmi palu said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing. VMWare Training in chennai | VMWare Training chennai | VMWare course in chennai | VMWare course chennai

Jhon anderson said...

Thank you so much for sharing this worthwhile to spent time on. You are running a really awesome blog. Keep up this good work Big Data Course in Chennai

Andria BZ said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
Regards,
sas course in Chennai|sas training institute in Chennai|sas training chennai

Pooja Doss said...

Oracle DBA Training in Chennai
Thanks for sharing this informative blog. I did Oracle DBA Certification in Greens Technology at Adyar. This is really useful for me to make a bright career..

Pooja Doss said...

Whatever we gathered information from the blogs, we should implement that in practically then only we can understand that exact thing clearly, but it’s no need to do it, because you have explained the concepts very well. It was crystal clear, keep sharing..
Websphere Training in Chennai

Pooja Doss said...

Data warehousing Training in Chennai
I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly..

Pooja Doss said...

Selenium Training in Chennai
Wonderful blog.. Thanks for sharing informative blog.. its very useful to me..

Pooja Doss said...

Oracle Training in chennai
Thanks for sharing such a great information..Its really nice and informative..

Pooja Doss said...

SAP Training in Chennai
This post is really nice and informative. The explanation given is really comprehensive and informative..

Pooja Doss said...

This information is impressive..I am inspired with your post writing style & how continuously you describe this topic. After reading your post,thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
Android Training In Chennai In Chennai

Pooja Doss said...

Pretty article! I found some useful information in your blog, it was awesome to read,thanks for sharing this great content to my vision, keep sharing..
Unix Training In Chennai

Pooja Doss said...

I found some useful information in your blog, it was awesome to read, thanks for sharing this great content to my vision, keep sharing..
SalesForce Training in Chennai

Pooja Doss said...

There are lots of information about latest technology and how to get trained in them, like Best Hadoop Training In Chennai in Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies Hadoop Training in Chennai By the way you are running a great blog. Thanks for sharing this blogs..

Alia Kumar said...

This is really an awesome article. Thank you for sharing this.It is worth reading for everyone. Visit us:
Oracle Training in Chennai

Alia Kumar said...

very nice blogs!!! i have to learning for lot of information for this sites...Sharing for wonderful information.Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.
Oracle DBA Training in Chennai

Alia Kumar said...

great article!!!!!This is very importent information for us.I like all content and information.I have read it.You know more about this please visit again.
Oracle RAC Training in Chennai

Alia Kumar said...

Wonderful tips, very helpful well explained. Your post is definitely incredible. I will refer this to my friend.
SalesForce Training in Chennai

Alia Kumar said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
Java Training in Chennai

Alia Kumar said...

Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
PHP Training in Chennai

Alia Kumar said...

Thanks for sharing this valuable information to our vision. You have posted a trust worthy blog keep sharing.Nice article i was really impressed by seeing this article, it was very interesting and it is very useful for me..
Android Training in Chennai

Alia Kumar said...

Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
SAP Training in Chennai

Alia Kumar said...

Excellent information with unique content and it is very useful to know about the information based on blogs.
Hadoop Training in Chennai

Alia Kumar said...

It is really very helpful for us and I have gathered some important information from this blog.If anyone wants to Selenium Training in Chennai reach Greens Technology training and placement academy.
selenium Training in Chennai

James Brown said...

That appears to be excellent however i am still not too sure that I it. At any rate will look far more into it and decide personally!
online word count

Sai Santosh said...

As we also follow this blog along with attending hadoop online training center, our knowledge about the hadoop increased in manifold ways. Thanks for the way information is presented on this blog.

Santhiya said...

I am reading your post from the beginning, it was so interesting to read & I feel thanks to you for posting such a good blog, keep updates regularly.
selenium training in chennai

Santhiya said...

Really awesome blog. Your blog is really useful for me. Thanks for sharing this informative blog. Keep update your blog.
QTP Training in Chennai

Dinju Thomas said...

This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..
Selenium Training in Chennai | QTP Training in Chennai

Dinju Thomas said...


Thanks for Information Oracle Apps Technical is a collection of a bunch of collected applications like accounts payables, purchasing, inventory, accounts receivables, human resources, order management, general ledger and fixed assets, etc which have its own functionality for serving the business
Oracle Apps Training In Chennai

Dinju Thomas said...

Oracle Training in chennai | Oracle D2K Training In chennai
This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..

Akasha said...

I am very impressed with the article I have just read,so nice.......
Manual Testing Training in Chennai

Andrew Son said...

Learning new technology would give oneself a true confidence in the current emerging Information Technology domain. With the knowledge of big data the most magnificent cloud computing technology one can go the peek of data processing. As there is a drastic improvement in this field everyone are showing much interest in pursuing this technology. Your content tells the same about evolving technology. Thanks for sharing this.

Hadoop Training in Chennai | Hadoop Training Chennai | Big Data Training in Chennai | Big Data Training Chennai

PLC Training In Chennai said...

PLC Training in Chennai
PLC Training Institute in Chennai
PLC Training Center in Chennai
PLC Training Centre in Chennai
PLC SCADA Training in Chennai
SCADA Training in Chennai
Best PLC Training in Chennai
Best PLC Training Institute in Chennai
DCS Training in Chennai
Automation Training in Chennai
Automation Training Institute in Chennai
Process Automation Training in Chennai
Industrial Automation Training in Chennai
HMI Training in Chennai
VFD Training in Chennai
Inplant Training in Chennai
Embedded Training in Chennai
VLSI Training in Chennai
IT Training in Chennai
AutoCad Training in Chennai
Web Designing Training in Chennai

Akula Rahul said...

ESIC Haryana Paramedical Staff Nurse Recruitment 2016

I like the valuable information you provide in your articles...........

Ramya Kolluru said...

JSSC Kakshpal Recruitment 2016

Hi everyone, it’s my first visit at this site, and post is genuinely fruitful for me, keep up posting these types of articles........

Arjun kumar said...

I have finally found a Worth able content to read. The way you have presented information here is quite impressive. I have bookmarked this page for future use. Thanks for sharing content like this once again. Keep sharing content like this.

Software testing training in chennai | Software testing training | Software testing course chennai

Dhivya Shree said...

There is a huge demand for professional big data analysts who are able to use the software which is used to process the big data in order to get accurate results. MNC's are looking for professionals who can process their data so that they can get into a accurate business decision which would eventually help them to earn more profits, they can serve their customers better, and their risk is lowered.
big data training in chennai|big data training|big data course in chennai|big data training chennai|big data hadoop training in chennai

Mathew Stephen said...

I have finally found a Worth able content to read. The way you have presented information here is quite impressive. I have bookmarked this page for future use. Thanks for sharing content like this once again. Keep sharing content like this.

Software testing training in chennai | Software testing training | Automation testing courses in chennai

Savitha said...

Great post and informative blog.it was awesome to read, thanks for sharing this great content to my vision.
Informatica Training In Chennai
Hadoop Training In Chennai
Oracle Training In Chennai
SAS Training In Chennai

Diya Patel said...

Best SQL Query Tuning Training Center In Chennai This information is impressive; I am inspired with your post writing style & how continuously you describe this topic. After reading your post, thanks for taking the time to discuss this, I feel happy about it and I love learning more about this topic..

Anna said...

Great and Useful Article.

Online Java Training | Java Training in Chennai | Java Training Institutes in Chennai | Online Java Course

Amit sirswa said...

IP CAmera in jaipur at Rajasthan
Home security system in jaipur
Wireless Home Security System in jaipur
Realtime attendance machine in jaipur
cctv camera dealer in jaipur
Boom Barriers system in jaipur at Rajasthan
security system solutions in jaipur

its really great information Thank you sir And keep it up More Post

Dhivya Shree said...

SAS stands for statistical analysis system which is a analysis tool developed by SAS institute and with the help of this tool data driven decisions can be taken which is helpful for the bsuiness.
SAS training in Chennai | SAS course in Chennai | SAS training institute in Chennai

rakesh said...

nice posts..

Informatica training, in the recent times has acquired a wide scope of popularity amongst the youngsters at the forefront of their career.
Informatica online training in hyderabad



Rohith said...

Yes, you are absolutely correct...And it is very informative and very clear and easy to understand.. seo training in chennai

Kits Online Training said...

nice blog, thanks for sharing
Sap SD Corporate Training Institute

Oracle Golden Gate Online Training By Real Time Faculties

Best Office 365 Online Training Institute By RealTime Faculties

Abinitio Online Training & Certification

Best Oracle RAC Online Training From US|UK|CANADA|