Makefile for Hadoop Wordcount Example

hadoop-makefile-225x225The Hadoop Map/Reduce Tutorial is a good starting to learn how Hadoop works. The tutorial uses the classic WordCount sample app to illustrate how to build and run a Hadoop application but the instructions involve several manual steps.

So I thought it would be handy create a Makefile to build the WordCount jar that could be adapted to building other Hadoop Map/Reduce applications.

Here is the Makefile I came up with:

HADOOP = ${HOME}/hadoop-2.0.2-alpha/bin/hadoop

APP = WordCount
SRC = src/*.java 
OUT = out

$(APP): $(SRC) 
        mkdir -p $(OUT) 
        javac -classpath `$(HADOOP) classpath` -d $(OUT) $(SRC) 
        jar -cvf $(APP).jar -C $(OUT) .

clean: 
        rm -rf $(OUT) *.jar .

To adapt this Makefile to other applications, follow these instructions:

  1. Create a directory for your application.
  2. cd to this directory and create a directory src.
  3. Place your java source code into src.
  4. Set HADOOP to point to your hadoop binary.
  5. Set APP to the name of your application.
  6. Set SRC to the list of source files for your project.
  7. Set OUT to the directory where the compiled class files will be placed.
  8. Type src to build your app.
  9. Run WordCount by typing (assuming your package name is example):
hadoop jar WordCount.jar <package>.WordCount <hdfs input path> <hdfs output path>

The nice thing about this Makefile is I added a clean directive to get rid of residual class files and the finished jar so you can build from a clean slate.

You can download a copy of this Makefile by right clicking and saving this link to your system.  Get the Makefile from this link instead of copying and pasting the previous text because the file from the link has the right tab spaces that the make program insists upon.

Author:

Article by Vic Hargrave

Software developer, blogger and family man enjoying life one cup of coffee at a time. I like programming and writing articles on tech topics. And yeah, I like coffee.

Leave a Reply

Your email address will not be published. Required fields are marked *