Differences

This shows you the differences between two versions of the page.

--- tutorials:cb1:customization:deploying_hadoop_cluster_on_cubieboard_guide [2013/10/17 11:02]
admin
+++ tutorials:cb1:customization:deploying_hadoop_cluster_on_cubieboard_guide [2013/12/23 14:50] (current)
@@ Line 12: / Line 12: @@
 {{:tutorials:cb1:customization:img_0564.jpg?820|}}
 ====Just start====
-Router/ hadoop_0.20.203 /Lubuntu 12.04v1.04(JDK-1.8)/ Power and network cables
+Router provide cluster with a lan
+hadoop_0.20.203  you can get it from here http://hadoop.apache.org/common/releases.html
+Lubuntu 12.04v1.04(JDK-1.8) 、Power and network cables
 {{:tutorials:cb1:customization:img_0579.jpg|780}}
 ===Users and network===
-master 3slaves both work in wlan network ,LAN connection between the nodes can ping each other.
-<code>
-.168.1.40  master
-.168.1.41  slave1
-.168.1.42  slave2
-.168.1.43  slave3
-.0.0.1     localhost
-.0.1.1     cubieboard
-</code>
-== Create a user==
+master 3slaves both work in wlan network ,LAN connection between the nodes can ping each other.
+==For master==
+Create a user
 <code>
 sudo addgroup hadoop
@@ Line 33: / Line 32: @@
 add:hadoop ALL=(ALL:ALL) ALL
 </code>
+Host&hostname
-==Host&hostname==
 <code>
 sudo vim ./etc/hosts
@@ Line 42: / Line 40: @@
 .168.1.42 slave2
 .168.1.43 slave3
-.168.1.44 slave4
-.168.1.45 slave5
-.168.1.46 slave6
-.168.1.47 slave7
 sudo vim ./etc/hostname
-cubieboard->slaveX or master
+cubieboard -->master
 </code>
+==For each slaves==
+You should do the same things.E.for slave1 node .
+add user
+<code>
+sudo addgroup hadoop
+sudo adduser -ingroup hadoop hadoop
+sudo vim ./etc/sudoers
+add:hadoop ALL=(ALL:ALL) ALL
+</code>
+Host&hostname
+<code>
+sudo vim ./etc/hosts
+add:
+.168.1.40 master
+.168.1.41 slave1
+sudo vim ./etc/hostname
+cubieboard -->slave1
+</code>
 ==Static IP settings==
+For each node
 <code>
 sudo vim ./etc/network/interfaces
 add
 #auto lo
 # iface lo inet loopback
@@ Line 72: / Line 83: @@
 nameserver 192.168.1.1
 </code>
-====Ssh sever ====
-===slaves and master to achieve mutual login without password===
+Stay here to sure all cbs have an user with Static IP which can ping each other.
-Master
+====SSH sever ====
+master<no passwd>slave1
+master<no passwd>slave2
+master<no passwd>slave3
+B  ---no passwd-->A
+A
 <code>
 ssh-keygen –t rsa –P ''
@@ Line 82: / Line 103: @@
 scp ~/.ssh/id_rsa.pub hadoop@192.168.1.40:~/
 </code>
-Slaves
+B
 <code>
 mkdir ~/.ssh
@@ Line 90: / Line 111: @@
 rm –r ~/id_rsa.pub
 </code>
+master to slave1
+<code>
+hadoop@master:~$
+hadoop@master:~$ ssh slave1
+Welcome to Linaro 13.04 (GNU/Linux 3.4.43+ armv7l)
+ * Documentation:  https://wiki.linaro.org/
+Last login: Thu Oct 17 03:38:36 2013 from master
+</code>
+slave1 to master
+<code>
+hadoop@slave1:~$ ssh master
+Welcome to Linaro 13.04 (GNU/Linux 3.4.43+ armv7l)
+ * Documentation:  https://wiki.linaro.org/
+Last login: Thu Oct 17 03:38:58 2013 from slave1
+</code>
 ====JDK====
-===JDK path modification===
+==JDK path modification==
+''hadoop@master:~$ vim /etc/profile''
 '' #export JAVA_HOME=/lib/jdk''
+You also should add to other nodes.
 ====Hadoop configuration====
-===Hadoop configuration parameters===
+You should edit core-site.xml hdfs-site.xml mapred-site.xml on /hadoop/hadoop_0.20.203_master/conf for master
-You should edit core-site.xml hdfs-site.xml mapred-site.xml on /hadoop/hadoop_0.20.203_master/conf
+You can do Hadoop configuration on your host computer .
+<code>
+aaron@cubietech:/work$ sudo mv hadoop_0.20.203 hadoop
+aaron@cubietech:/work$ cd hadoop/conf/
+aaron@cubietech:/work/hadoop/conf$ sudo vim core-site.xml
+</code>
 core-site.xml
@@ Line 142: / Line 194: @@
 </configuration>
 </code>
-===7.How to run===
+After that ,You should copy hadoop to every node
 <code>
-bin/hadoop namenode -format
+scp -r hadoop root@192.168.1.40:/usr/local
+scp -r hadoop root@192.168.1.41:/usr/local
+scp -r hadoop root@192.168.1.42:/usr/local
+scp -r hadoop root@192.168.1.43:/usr/local
+</code>
+====How to run====
+''hadoop@master:~$ cd /usr/local/hadoop/''
+format filesys
+''bin/hadoop namenode -format ''
+<code>
+hadoop@master:/usr/local/hadoop$ bin/hadoop namenode -format
+/10/17 05:49:16 INFO namenode.NameNode: STARTUP_MSG:
+/************************************************************
+STARTUP_MSG: Starting NameNode
+STARTUP_MSG:   host = master/192.168.1.40
+STARTUP_MSG:   args = [-format]
+STARTUP_MSG:   version = 0.20.203.0
+STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-security-203 -r 1099333; compiled by 'oom' on Wed May  4 07:57:50 PDT 2011
+************************************************************/
+Re-format filesystem in /usr/local/hadoop/datalog1 ? (Y or N) Y
+Re-format filesystem in /usr/local/hadoop/datalog2 ? (Y or N) Y
+/10/17 05:49:22 INFO util.GSet: VM type       = 32-bit
+/10/17 05:49:22 INFO util.GSet: 2% max memory = 19.335 MB
+/10/17 05:49:22 INFO util.GSet: capacity      = 2^22 = 4194304 entries
+/10/17 05:49:22 INFO util.GSet: recommended=4194304, actual=4194304
+/10/17 05:49:24 INFO namenode.FSNamesystem: fsOwner=hadoop
+/10/17 05:49:24 INFO namenode.FSNamesystem: supergroup=supergroup
+/10/17 05:49:24 INFO namenode.FSNamesystem: isPermissionEnabled=true
+/10/17 05:49:24 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
+/10/17 05:49:24 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
+/10/17 05:49:24 INFO namenode.NameNode: Caching file names occuring more than 10 times
+/10/17 05:49:26 INFO common.Storage: Image file of size 112 saved in 0 seconds.
+/10/17 05:49:26 INFO common.Storage: Storage directory /usr/local/hadoop/datalog1 has been successfully formatted.
+/10/17 05:49:26 INFO common.Storage: Image file of size 112 saved in 0 seconds.
+/10/17 05:49:27 INFO common.Storage: Storage directory /usr/local/hadoop/datalog2 has been successfully formatted.
+/10/17 05:49:27 INFO namenode.NameNode: SHUTDOWN_MSG:
+/************************************************************
+SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.40
+************************************************************/
+</code>
+<code>
 bin/hadoop dfsadmin -report
@@ Line 152: / Line 247: @@
 bin/stop-all.sh // stop
-./bin/hadoop jar hadoop-examples-0.20.203.0.jar pi 100 100 //Calculates PI
 </code>
-====Troubleshooting====
-===== You can also read this tutorial =====
+''./bin/hadoop jar hadoop-examples-0.20.203.0.jar pi 100 100 ''
+{{:tutorials:cb1:installation:screenshot_from_2013-07-31_18_18_35.png|800}}
+You can also see filesys on web
+http:192.168.1.2:50030
+http:192.168.1.2:50070
+{{:tutorials:cb1:installation:screenshot_from_2013-07-31_18_18_14.png|800}}
+You can also read this tutorial
 http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html
 <WRAP noprint>
 {{tag>Cubieboard Cubietruck}}
 </WRAP>

CubieBoard Docs

User Tools

Site Tools

Differences

Page Tools