部署上,很简单,就是分成两部分:修改配置文件盒启动脚本。
一:修改配置文件:
hadoop2.2的配置文件在/opt/hadoop-2.2.0/etc/hadoop文件夹下,具体配置文件修改如下:
1、修改/etc/hosts文件(sudo gedit /etc/hosts)
192.168.222.154 hd2-single
2、修改core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name> <value>/home/sujx/hadoop/tmp</value> </property> <property> <name>fs.defaultFS</name> <value>hdfs://hd2-single:9000</value> <final>true</final> </property></configuration>
fs.defaultFS:HDFS文件系统的URL
3. 修改hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.name.dir</name> <value>file:/home/sujx/hadoop/dfs/name</value> <final>true</final> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/home/sujx/hadoop/dfs/data</value> <final>true</final> </property> <property> <name>dfs.replication</name> <value>1</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> </configuration>4. 修改mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapred.system.dir</name> <value>file:/home/sujx/hadoop/mapred/system</value> <final>true</final> </property> <property> <name>mapred.local.dir</name> <value>file:/home/sujx/hadoop/mapred/local</value> <final>true</final> </property></configuration>
5. 修改yarn-site.xml
<configuration>
<property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> <description>shuffle service that needsto be set for Map Reduce to run </description> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hd2-single</value> <description>hostanem of RM</description> </property></configuration>
6. 修改slave
hd2-single
至此,配置文件修改完毕,比较多,挺麻烦的。
二:启动脚本。
启动hadoop脚本,需呀用到一些环境变量,所以需要先修改的profile文件。
使用命令:sudo /etc/profile
export HADOOP_HOME=/opt/hadoop-2.2.0
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbinexport HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOMEexport HADOOP_HDFS_HOME=$HADOOP_HOMEexport YARN_HOME=$HADOOP_HOMEexport HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexport YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop在初次运行Hadoop的时候需要初始化Hadoop文件系统,命令如下:
hdfs namenode -format
1. 启动脚本一:
sujx@ubuntu:~$ hadoop-daemon.sh start namenode
starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.outsujx@ubuntu:~$ hadoop-daemon.sh start datanodestarting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.outsujx@ubuntu:~$ hadoop-daemon.sh start secondarynamenodestarting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.outsujx@ubuntu:~$ jps9310 SecondaryNameNode9345 Jps9140 NameNode9221 DataNodesujx@ubuntu:~$ yarn-daemon.sh start resourcemanagerstarting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.outsujx@ubuntu:~$ yarn-daemon.sh start nodemanagerstarting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.outsujx@ubuntu:~$ jps9310 SecondaryNameNode9651 NodeManager9413 ResourceManager9140 NameNode9709 Jps9221 DataNodesujx@ubuntu:~$2. 启动脚本二:
sujx@ubuntu:~$ start-dfs.sh
Starting namenodes on [hd2-single]hd2-single: starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.outhd2-single: starting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.outsujx@ubuntu:~$ start-yarn.sh starting yarn daemonsstarting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.outhd2-single: starting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.outsujx@ubuntu:~$ jps11414 SecondaryNameNode10923 NameNode11141 DataNode12038 Jps11586 ResourceManager11811 NodeManagersujx@ubuntu:~$3. 启动脚本三:
sujx@ubuntu:~$ start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.shStarting namenodes on [hd2-single]hd2-single: starting namenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-namenode-ubuntu.outhd2-single: starting datanode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-datanode-ubuntu.outStarting secondary namenodes [0.0.0.0]0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.2.0/logs/hadoop-sujx-secondarynamenode-ubuntu.outstarting yarn daemonsstarting resourcemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-resourcemanager-ubuntu.outhd2-single: starting nodemanager, logging to /opt/hadoop-2.2.0/logs/yarn-sujx-nodemanager-ubuntu.outsujx@ubuntu:~$ jps14156 NodeManager14445 Jps13267 NameNode13759 SecondaryNameNode13485 DataNode13927 ResourceManagersujx@ubuntu:~$其实这三种方式最终效果都是相同,他们内部也都是相互调用关系。对应的结束脚本也简单:
1. 结束脚本一:sujx@ubuntu:~$ hadoop-daemon.sh stop nodemanagersujx@ubuntu:~$ hadoop-daemon.sh stop resourcemanagersujx@ubuntu:~$ hadoop-daemon.sh stop secondarynamenodesujx@ubuntu:~$ hadoop-daemon.sh stop datanodesujx@ubuntu:~$ hadoop-daemon.sh stop namenode 2. 结束脚本二:sujx@ubuntu:~$ stop-yarn.sh sujx@ubuntu:~$ stop-dfs.sh 3. 结束脚本三:sujx@ubuntu:~$ stop-all.sh 至此,单机伪分布就已经部署完毕。转自:http://www.linuxidc.com/Linux/2014-01/95570p2.htm