Differences

This shows you the differences between two versions of the page.

--- tutorials:cb1:customization:deploying_hadoop_cluster_on_cubieboard_guide [2013/10/17 10:51]
admin
+++ tutorials:cb1:customization:deploying_hadoop_cluster_on_cubieboard_guide [2013/12/23 14:50] (current)
@@ Line 10: / Line 10: @@
 Hadoop is a free, Java-based programming framework that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation.You can use A10 Cubieboards to build Hadoop cluster base on Lubuntu(v1.04) image which was integrated jdk-1.8 armhf.Need to do configuration: JDK environment, slaves and master ssh sever, hadoop configuration parameters.
-{{:tutorials:cb1:customization:img_0564.jpg?800|}}
+{{:tutorials:cb1:customization:img_0564.jpg?820|}}
 ====Just start====
-===1.tools and soft===
-Router/ hadoop_0.20.203 /Lubuntu 12.04v1.04(JDK-1.8)/ Power and network cables
+Router provide cluster with a lan
+hadoop_0.20.203  you can get it from here http://hadoop.apache.org/common/releases.html
+Lubuntu 12.04v1.04(JDK-1.8) 、Power and network cables
 {{:tutorials:cb1:customization:img_0579.jpg|780}}
 ===Users and network===
-== Create a user==
+master 3slaves both work in wlan network ,LAN connection between the nodes can ping each other.
+==For master==
+Create a user
 <code>
 sudo addgroup hadoop
@@ Line 23: / Line 32: @@
 add:hadoop ALL=(ALL:ALL) ALL
 </code>
+Host&hostname
-==Host&hostname==
 <code>
 sudo vim ./etc/hosts
@@ Line 32: / Line 40: @@
 .168.1.42 slave2
 .168.1.43 slave3
-.168.1.44 slave4
-.168.1.45 slave5
-.168.1.46 slave6
-.168.1.47 slave7
 sudo vim ./etc/hostname
-cubieboard->slaveX or master
+cubieboard -->master
 </code>
+==For each slaves==
+You should do the same things.E.for slave1 node .
+add user
+<code>
+sudo addgroup hadoop
+sudo adduser -ingroup hadoop hadoop
+sudo vim ./etc/sudoers
+add:hadoop ALL=(ALL:ALL) ALL
+</code>
+Host&hostname
+<code>
+sudo vim ./etc/hosts
+add:
+.168.1.40 master
+.168.1.41 slave1
+sudo vim ./etc/hostname
+cubieboard -->slave1
+</code>
 ==Static IP settings==
+For each node
 <code>
 sudo vim ./etc/network/interfaces
 add
 #auto lo
 # iface lo inet loopback
@@ Line 62: / Line 83: @@
 nameserver 192.168.1.1
 </code>
-====Ssh sever ====
-===slaves and master to achieve mutual login without password===
+Stay here to sure all cbs have an user with Static IP which can ping each other.
-Master
+====SSH sever ====
+master<no passwd>slave1
+master<no passwd>slave2
+master<no passwd>slave3
+B  ---no passwd-->A
+A
 <code>
 ssh-keygen –t rsa –P ''
@@ Line 72: / Line 103: @@
 scp ~/.ssh/id_rsa.pub hadoop@192.168.1.40:~/
 </code>
-Slaves
+B
 <code>
 mkdir ~/.ssh
@@ Line 80: / Line 111: @@
 rm –r ~/id_rsa.pub
 </code>
+master to slave1
+<code>
+hadoop@master:~$
+hadoop@master:~$ ssh slave1
+Welcome to Linaro 13.04 (GNU/Linux 3.4.43+ armv7l)
+ * Documentation:  https://wiki.linaro.org/
+Last login: Thu Oct 17 03:38:36 2013 from master
+</code>
+slave1 to master
+<code>
+hadoop@slave1:~$ ssh master
+Welcome to Linaro 13.04 (GNU/Linux 3.4.43+ armv7l)
+ * Documentation:  https://wiki.linaro.org/
+Last login: Thu Oct 17 03:38:58 2013 from slave1
+</code>
 ====JDK====
-===JDK path modification===
+==JDK path modification==
+''hadoop@master:~$ vim /etc/profile''
 '' #export JAVA_HOME=/lib/jdk''
+You also should add to other nodes.
 ====Hadoop configuration====
-===Hadoop configuration parameters===
+You should edit core-site.xml hdfs-site.xml mapred-site.xml on /hadoop/hadoop_0.20.203_master/conf for master
-You should edit core-site.xml hdfs-site.xml mapred-site.xml on /hadoop/hadoop_0.20.203_master/conf
+You can do Hadoop configuration on your host computer .
+<code>
+aaron@cubietech:/work$ sudo mv hadoop_0.20.203 hadoop
+aaron@cubietech:/work$ cd hadoop/conf/
+aaron@cubietech:/work/hadoop/conf$ sudo vim core-site.xml
+</code>
 core-site.xml
@@ Line 132: / Line 194: @@
 </configuration>
 </code>
-===7.How to run===
+After that ,You should copy hadoop to every node
 <code>
-bin/hadoop namenode -format
+scp -r hadoop root@192.168.1.40:/usr/local
+scp -r hadoop root@192.168.1.41:/usr/local
+scp -r hadoop root@192.168.1.42:/usr/local
+scp -r hadoop root@192.168.1.43:/usr/local
+</code>
+====How to run====
+''hadoop@master:~$ cd /usr/local/hadoop/''
+format filesys
+''bin/hadoop namenode -format ''
+<code>
+hadoop@master:/usr/local/hadoop$ bin/hadoop namenode -format
+/10/17 05:49:16 INFO namenode.NameNode: STARTUP_MSG:
+/************************************************************
+STARTUP_MSG: Starting NameNode
+STARTUP_MSG:   host = master/192.168.1.40
+STARTUP_MSG:   args = [-format]
+STARTUP_MSG:   version = 0.20.203.0
+STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
+************************************************************/
+Re-format filesystem in /usr/local/hadoop/datalog1 ? (Y or N) Y
+Re-format filesystem in /usr/local/hadoop/datalog2 ? (Y or N) Y
+/10/17 05:49:22 INFO util.GSet: VM type       = 32-bit
+/10/17 05:49:22 INFO util.GSet: 2% max memory = 19.335 MB
+/10/17 05:49:22 INFO util.GSet: capacity      = 2^22 = 4194304 entries
+/10/17 05:49:22 INFO util.GSet: recommended=4194304, actual=4194304
+/10/17 05:49:24 INFO namenode.FSNamesystem: fsOwner=hadoop
+/10/17 05:49:24 INFO namenode.FSNamesystem: supergroup=supergroup
+/10/17 05:49:24 INFO namenode.FSNamesystem: isPermissionEnabled=true
+/10/17 05:49:24 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
+/10/17 05:49:24 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
+/10/17 05:49:24 INFO namenode.NameNode: Caching file names occuring more than 10 times
+/10/17 05:49:26 INFO common.Storage: Image file of size 112 saved in 0 seconds.
+/10/17 05:49:26 INFO common.Storage: Storage directory /usr/local/hadoop/datalog1 has been successfully formatted.
+/10/17 05:49:26 INFO common.Storage: Image file of size 112 saved in 0 seconds.
+/10/17 05:49:27 INFO common.Storage: Storage directory /usr/local/hadoop/datalog2 has been successfully formatted.
+/10/17 05:49:27 INFO namenode.NameNode: SHUTDOWN_MSG:
+/************************************************************
+SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.40
+************************************************************/
+</code>
+<code>
 bin/hadoop dfsadmin -report
@@ Line 142: / Line 247: @@
 bin/stop-all.sh // stop
-./bin/hadoop jar hadoop-examples-0.20.203.0.jar pi 100 100 //Calculates PI
 </code>
-====Troubleshooting====
-===== You can also read this tutorial =====
+''./bin/hadoop jar hadoop-examples-0.20.203.0.jar pi 100 100 ''
+{{:tutorials:cb1:installation:screenshot_from_2013-07-31_18_18_35.png|800}}
+You can also see filesys on web
+http:192.168.1.2:50030
+http:192.168.1.2:50070
+{{:tutorials:cb1:installation:screenshot_from_2013-07-31_18_18_14.png|800}}
+You can also read this tutorial
 http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html
 <WRAP noprint>
 {{tag>Cubieboard Cubietruck}}
 </WRAP>

CubieBoard Docs

User Tools

Site Tools

Differences

Page Tools