Finally I have finished testing my first hadoop experiment,altought it’s just a very easy&simple word count program which included in hadoop example code itself, to configure a correct enviornment for hadoop developing is so hard for a noob as me.

Hadoop development needs an java developing environment of course. so I prepared it myself a open-jdk-1.7.0 version, which affect some configure side in my procedures.
here is my experimental environment:

  1. Hadoop-1.0.4
  2. jdk-1.7.0
  3. scientific linux 6.3 (carbon)
  4. eclipse jee-juno-SR1-linux-gtk-x86_64

After googling two days later,hoaxed by too many bad text&information, I have finally got the right key to successfully configure a environment for hadoop program developing.

NOW let’s begin!

  1. after finish scientific linux 6.3 installation,the first thing you need to do is to install a update-to-dated open-jdk-1.7.0
    [root@localhost ~]# yum search java-1.7.0-openjdk.x86_64
    to ensure that your develop environment is correct for further configuration and I have encounter a problem here that depict a little restriction of denying uncompatible jdk jar package from windows to linux.

  2. download a eclipse for fast&cosy hadoop program development
    unarchived eclipse to your own IDE directory, for me is /opt so the complete directory of eclipse is /opt/eclipse and you can startup eclipse by
    [root@localhost ~]# /opt/eclipse/eclipse
    or you can just add a symbol link to your user binary path
    [root@localhost ~]# ln -s /opt/eclipse/eclipse /usr/bin/eclipse

  3. download the hadoop and that is what I have downloaded ,the version is 1.0.4 a little trended one. very hard to find on other blog
    unarchive the hadoop to your tools directory ,for me is the same as eclipse
    [root@localhost ~]# tar -zxf hadoop-1.0.4.tar.gz -C /opt/

  4. then you shoud make some change on hadoop’s configuration files.
    edit the conf/core-site.xml


    this configure allow your datanode to listen on the port of :8000 on localhost and decided to change temporary directory to /opt/tmp
    edit conf/mapred-site.xml ,as:


    this file configure the job tracker node in cluster,remember port :9001
    next edit the file named conf/hdfs-site.xml


    the configuration above is to determeine now many replication of hdfs will be stored, because HADOOP is a distributed system, distributed and replication strategy must be the best method to store large scale of data.

  5. initialize the hdfs and startup HADOOP daemon
    [root@localhost hadoop-1.0.4]# ./bin/hadoop namenode -format
    [root@localhost hadoop-1.0.4]# ./bin/

  6. get corresponding eclipse plugin
    As my hadoop version is 1.0.4, the version of plugin must be 1.0.4
    I have to compile a plugin jar file myself, because the version after 0.20 in hadoop release, the hadoop releaser will not compile the eclipse plugin in advance, whilst they just put source code in
    Hence you can use ant or ivy to compile yourself accordent with your operation system or other parameters in developing environment.
    you can also download 1.0.4 plugin on internet
    that is much easier!

  7. put eclipse plugin in $ECLIPSE_HOME/plugin
    in my situation, /opt/eclipse/plugins/
    now your eclipse have the capability to develop hadoop project.

  8. launch eclipse
    then startup your eclipse and click this little button named “open perspective”
    then enter the Map/reduce location tab
    right click and new or edit a hadoop location
    edit this configuration window to ensure your eclipse connect to the right port and host ,do you remember that two ports I have mentioned?
    the port on left is in the mapred-site.xml file other is in core-site.xml file
    now you can see the project explore,double click the hadoop icon in this DFS locations
    if everthing ok. there will be a progress tell you how this connection perform.

Beware your directory may not similar to mine,thus dont get annoyed if your pattern is different to mine.
now it seems your HADOOP development environment is complete now lets test it to as if this work!

test our hadoop env

  1. sample data

     java c++ python c  
     java c++ javascript  
     helloworld hadoop  
     mapreduce java hadoop hbase
  2. create HDFS folder create folder on HDFS through hadoop command named /tmp/workcount notice this named folder is virtually mounted in HDFS, thus it is located in the folder you have defined in configuration file.
    bin/hadoop fs -mkdir /tmp/wordcount
    then transmit this data file into HDFS
    bin/hadoop fs -copyFromLocal /home/rugal/word.txt /tmp/wordcount/word.txt

  3. execute hadoop job run this project but you have to make some change in parameters
    right click on file in eclipse –>Run as–>Run Configurations
    add parameter below seperated by space

    now you can run this project,is that correct run ?
    if this project encounter java.lang.OutOfMemoryError: Java heap space,just configure VM arguments in (program arguments)
    -Xms512m -Xmx1024m -XX:MaxPermSize=256m

    there are many line printed on eclipse console
    after the completion of running,you can see that result of program by hadoop command
    bin/hadoop fs -ls /tmp/wordcount/out
    you should know folder is refer to your command above so be careful create folders

  4. display result let see the result by HADOOP command
    bin/hadoop fs -cat /tmp/wordcount/out/part-r-00000

     c         1
     c++       2
     hadoop    2
     hbase     1
     helloworld 1
     java       3
     javascript  1
     mapreduce  1
     python    1

    ok, here we have finished all the hadoop configuration and developing environment deployment, it spent me 2 days in total!


20 December 2013