I have been studying Hadoop – The Definitive Guide by Tom White and started building the sample applications with the Makefile I discussed in my last blog. Although the Makefile approach works, I decided to try using the IntelliJ Community Edition IDE to build the examples in any given chapter all at once.
This time around I’ll walk you through a procedure to create an IntelliJ project for building Hadoop applications.
If you don’t have it already, you can get the latest version of IntelliJ Community Edition here. Select the package for your operating system of choice, either Mac OS or Linux, then install IntelliJ by placing the package contents in your directory of choice.
Install the Sample Code
- First you need the sample code from the Hadoop book. You get it from GitHub.
- Place the code in a directory called hadoop-book.
- NOTE: Since writing this article the lib directory has gone missing in the code for the book. To make your life easier, click on this link hadoop-book-lib.tgz to get a copy of what I used. Unpack this code placing it into the hadoop-book directory with this command:
tar -zxvf hadoop-book-lib.tgz
- If you don’t have IntelliJ Community Edition you can go here to download the version for your particular operating system. As of this writing the latest version of IntelliJ is 12.0.
- If you don’t have Hadoop you’ll need to download one of those distributions. For the project in this article I’m using Hadoop 2.0.2-alpha.
- When you get the Hadoop tarball unpack it in your home directory somewhere with this command:
tar -zxvf hadoop-2.0.2-alpha-src.tar.gz
Create a Project
For this example we build the apps from Chapter 3.
- Start up your IntelliJ IDE.
- Select Create New Project in the Quick Start panel.
- Set the project type to Java Module .
- Set the project name to ch03
- Place the project in your hadoop-book directory which in my case is located here on my Mac:
- Click on More Settings .
- Set the project format to .ipr (file based).
- Make sure the Create source root box is checked. IntelliJ will use the code that is already located in the ch03/src directory then click Next .
- We will not add any other coding frameworks so just click on Finish in the next screen.
- The project we just created includes the Java libraries but we need to add the Hadoop jars and the jars that come with the Hadoop book code. Select File > Project Structure menu item.
- IntelliJ adds the JDK jars but we need to add the library jars that come with the book sample code and the Hadoop jars. Select File > Project Structure .
- Click on Modules .
- Click on the Dependencies tab then click on the + button at the bottom of the screen.
- Select Jars or directories…
- Open the hadoop-2.0.2-alpha/share/hadoop directory, select all the subdirectories there then click OK.
- Repeat step 15, this time select the hadoop-2.0.2-alpha/share/hadoop/common/lib then click OK.
- Repeat step 15, this time select the hadoop-book/lib directory then click OK. Your Project Structure screen should look something like this:
- Now we have to configure the project to build the chapter 3 jar file. Select Artifacts in the Project Structure screen.
- Click on the ‘+’ symbol at the top of the screen.
- Select Jar then Empty in the drop down menus.
- Set the jar Name to ch03.
- Set the Output directory to put the ch03.jar file in the hadoop-book/ch03 directory.
- Initially the ch03.jar has no contents. To add the ch03 class files to the output jar, click on the ‘+’ button in the Output Layout tab and select Module Output. This will add an the ‘ch03 compile output entry under ch03.jar.
- Click on OK.
- To build the jar select Build > Build Artifacts…
- You will see a menu that offers four build actions: Build, Rebuild, Clean or Edit… the last of which takes you to the Project Structure – Artifacts screen to choose different build options. Select Rebuild to clean the hadoop-book.ch03 directory and build the ch03.jar.
Using this procedure you can create projects for any of the chapters in the Hadoop book source code which can save you a lot of time building and experimenting with the example applications.
Author: Vic Hargrave