hadoop（2.x）以hadoop2.2为例完全分布式最新高可靠安装文档

Utor 发表于 2014-9-5 21:26
**** 作者被禁止或删除 内容自动屏蔽 ****

防火墙没关

学习 学习 学习

好文要顶，膜拜。

谢谢楼主的分享~

学习一下，谢谢楼主~

学习一下，谢谢楼主~

大神 有问题请教 我五台机器 一主四从
启动时提示：
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
15/11/03 01:08:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
master.hadoop]
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
master.hadoop: starting namenode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-namenode-master.hadoop.out
library: ssh: Could not resolve hostname library: Name or service not known
which: ssh: Could not resolve hostname which: Name or service not known
You: ssh: Could not resolve hostname You: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
master.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
master.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
64-Bit: ssh: Could not resolve hostname 64-Bit: Name or service not known
loaded: ssh: Could not resolve hostname loaded: Name or service not known
warning:: ssh: Could not resolve hostname warning:: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
guard.: ssh: Could not resolve hostname guard.: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
to: ssh: Could not resolve hostname to: Name or service not known
disabled: ssh: Could not resolve hostname disabled: Name or service not known
Java: ssh: Could not resolve hostname Java: Name or service not known
Server: ssh: Could not resolve hostname Server: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known
guard: ssh: Could not resolve hostname guard: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
will: ssh: Could not resolve hostname will: Name or service not known
recommended: ssh: Could not resolve hostname recommended: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
The: ssh: Could not resolve hostname The: Name or service not known
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
<libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
or: ssh: Could not resolve hostname or: Name or service not known
it: ssh: Could not resolve hostname it: Name or service not known
now.: ssh: Could not resolve hostname now.: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
link: ssh: Could not resolve hostname link: Name or service not known
'-z: ssh: Could not resolve hostname '-z: Name or service not known
slave4.hadoop: starting datanode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-datanode-slave4.hadoop.out
slave1.hadoop: starting datanode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-datanode-slave1.hadoop.out
slave2.hadoop: starting datanode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-datanode-slave2.hadoop.out
slave3.hadoop: starting datanode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-datanode-slave3.hadoop.out
Starting secondary namenodes [Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
master.hadoop]
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
You: ssh: Could not resolve hostname You: Name or service not known
master.hadoop: starting secondarynamenode, logging to /usr/local/hadoop-2.6.2/logs/hadoop-root-secondarynamenode-master.hadoop.out
master.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
master.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
64-Bit: ssh: Could not resolve hostname 64-Bit: Name or service not known
loaded: ssh: Could not resolve hostname loaded: Name or service not known
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
which: ssh: Could not resolve hostname which: Name or service not known
Java: ssh: Could not resolve hostname Java: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
to: ssh: Could not resolve hostname to: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
Server: ssh: Could not resolve hostname Server: Name or service not known
it: ssh: Could not resolve hostname it: Name or service not known
will: ssh: Could not resolve hostname will: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
warning:: ssh: Could not resolve hostname warning:: Name or service not known
disabled: ssh: Could not resolve hostname disabled: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
The: ssh: Could not resolve hostname The: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
guard: ssh: Could not resolve hostname guard: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
guard.: ssh: Could not resolve hostname guard.: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
recommended: ssh: Could not resolve hostname recommended: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
'-z: ssh: Could not resolve hostname '-z: Name or service not known
or: ssh: Could not resolve hostname or: Name or service not known
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
now.: ssh: Could not resolve hostname now.: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
<libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
link: ssh: Could not resolve hostname link: Name or service not known
'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
15/11/03 01:09:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop-2.6.2/logs/yarn-root-resourcemanager-master.hadoop.out
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
slave4.hadoop: starting nodemanager, logging to /usr/local/hadoop-2.6.2/logs/yarn-root-nodemanager-slave4.hadoop.out
slave1.hadoop: starting nodemanager, logging to /usr/local/hadoop-2.6.2/logs/yarn-root-nodemanager-slave1.hadoop.out
slave3.hadoop: starting nodemanager, logging to /usr/local/hadoop-2.6.2/logs/yarn-root-nodemanager-slave3.hadoop.out
slave2.hadoop: starting nodemanager, logging to /usr/local/hadoop-2.6.2/logs/yarn-root-nodemanager-slave2.hadoop.out
slave4.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
slave4.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
slave1.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
slave1.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
slave3.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
slave3.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
slave2.hadoop: Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /usr/local/hadoop-2.6.2/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
slave2.hadoop: It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.

安装完毕启动完成 用jps命令显示如下 正常吗
[root@master sbin]# jps
1579 NameNode
1845 SecondaryNameNode
1995 ResourceManager
2257 Jps
[root@master sbin]#


[root@slave1 ~]# jps
1495 NodeManager
1613 Jps
[root@slave1 ~]#


[root@slave2 ~]# jps
1494 NodeManager
1612 Jps
1401 DataNode
[root@slave2 ~]#


[root@slave3 ~]# jps
1832 NodeManager
1950 Jps
1737 DataNode
[root@slave3 ~]#


[root@slave4 ~]# jps
1837 NodeManager
1955 Jps
[root@slave4 ~]#

桥接模式下的IP总是换，一旦重启的时候就变了，用起来很不方便，有什么好的解决办法？

膜拜楼主，根据文章搭建了hadoop环境，jps进程列表都正常，但web界面无法打开，请问这个一般是什么原因啊？谢谢

leoinnoahs 发表于 2015-11-9 03:12
膜拜楼主，根据文章搭建了hadoop环境，jps进程列表都正常，但web界面无法打开，请问这个一般是什么原因啊？ ...

yarn-site.xml各个端口，hosts保证正确
[mw_shl_code=bash,true]<configuration>
        <property>
               <name>yarn.nodemanager.aux-services</name>
               <value>mapreduce_shuffle</value>
        </property>
        <property>                                                               
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
               <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
        <property>
               <name>yarn.resourcemanager.address</name>
               <value>master:8032</value>
       </property>
       <property>
               <name>yarn.resourcemanager.scheduler.address</name>
               <value>master:8030</value>
       </property>
       <property>
            <name>yarn.resourcemanager.resource-tracker.address</name>
             <value>master:8031</value>
      </property>
      <property>
              <name>yarn.resourcemanager.admin.address</name>
               <value>master:8033</value>
       </property>
       <property>
               <name>yarn.resourcemanager.webapp.address</name>
               <value>master:8088</value>
       </property>
</configuration>
[/mw_shl_code]
如果还不行，建议重新搭建，下次肯定不会这个问题，也就是说，是疏忽造成的

更多：

使用hadoop集群，更详细内容，可以查看
hadoop2.X使用手册1：通过web端口查看主节点、slave1节点及集群运行状态
hadoop2.X使用手册2：如何运行自带wordcount
hadoop2.2运行mapreduce（wordcount）问题总结