日志
Hadoop-2.6.0在Centos6.5 64位系统编译前准备工作
热度 5 |
hadoop2.6.0伪分布搭建参考文档：
http://www.aboutyun.com/thread-10554-1-1.html[root@lsn-linux hadoop-2.6.0]# hadoop fs -ls /
14/12/09 19:43:06 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@lsn-linux hadoop-2.6.0]#
原因是hadoop-2.6.0.tar.gz安装包是在32位机器上编译的，64位的机器加载本地库.so文件时出错，不影响使用。
解决：
1、重新编译源码后将新的lib/native替换到集群中原来的lib/native
2、修改hadoop-env.sh ，增加
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native"
需要进行编译准备工作：
0.安装JDK,使用 java -version 查看jdk版本，确定JDK版本是64位。
a. 解压jdk
$ tar -xvzf jdk-7u60-linux-x64.tar.gz
b. 设置环境变量 vim /etc/profile
export JAVA_HOME=/usr/local/jdk1.7
export HADOOP_HOME=/usr/local/hadoop-2.6.0
export MAVEN_HOME=/opt/apache-maven
export FINDBUG_HOME=/opt/findbugs-3.0.0
export ANT_HOME=/opt/apache-ant-1.9.4
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
:$MAVEN_HOME/bin:$FINDBUG_HOME/bin:$ANT_HOME/bin (备注：不能换行)
c.使配置文件生效
$ source /etc/profile
1.安装gcc|gc++
yum install gcc
yum install gcc-c++
验证
2.安装Apache-Maven。
tar -zxvf apache-maven-3.2.1.tar.gz
配置环境变量 vim /etc/profile
export MAVEN_HOME=/opt/apache-maven
export PATH=.:PATH:$MAVEN_HOME/bin
验证： mvn --version
3.安装Apache-ant(重要).
tar -zxvf apache-ant-1.9.4-bin.tar.gz
配置环境变量 vim /etc/profile
export MAVEN_HOME=/opt/apache-maven
export PATH=$PATH:$MAVEN_HOME/bin
3.安装protobuf(goole序列化工具)
tar -zxvf protobuf-2.5.0.tar.gz
./configuration
make #编译
make install
验证：protoc --version
4.安装CMake2.6 or newer
安装 yum install cmake
安装 yum install openssl-devel
安装 yum install ncurses-devel
验证：cmake --version
5.安装make
yum install make
验证： make --version
6.Hadoop - hadoop-common-project中的pom.xml添加依赖（hadoop-2.2.0需要修改，hadoop2.6.0版本不需要）
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
7.在编译之前防止 java.lang.OutOfMemoryError: Java heap space 堆栈问题,在centos系统中执行命令：
$ export MAVEN_OPTS="-Xms256m -Xmx512m"
8.解压压缩包
tar -zxvf hadoop-2.6.0-src.tar.gz
a.执行命令 cd ${hostname_Local}/hadoop-2.6.0/ 目录下
b.编译
mvn package -DskipTests -Pdist,native
c.编译好的项目放在 hadoop-2.6.0-src/hadoop-dist/target目录下。
/root/Downloads/hadoop-2.6.0-src/hadoop-dist/target 即 hadoop-2.6.0就是编译好的包。
=====================================================================
编译日志：
[INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-dist ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: /root/Downloads/hadoop-2.6.0-src/hadoop-dist/target/hadoop-dist-2.6.0.jar
[INFO]
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] No sources in project. Archive not created.
[INFO]
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] No sources in project. Archive not created.
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-dist ---
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (tar) @ hadoop-dist ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /root/Downloads/hadoop-2.6.0-src/hadoop-dist/target/hadoop-dist-2.6.0-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 13.582 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 9.846 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 24.408 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 1.967 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 6.443 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 20.692 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 14.250 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 23.716 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 13.714 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [08:46 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 47.127 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 48.790 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.316 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [14:58 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [11:10 min]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [01:43 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 27.438 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.146 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.165 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [07:03 min]
[INFO] hadoop-yarn-common ................................. SUCCESS [03:31 min]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.827 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [01:11 min]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [02:25 min]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 17.129 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 39.350 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [01:44 min]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 32.941 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 44.664 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.197 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 15.165 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 9.604 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.149 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 31.971 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 22.195 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.673 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [02:08 min]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [01:38 min]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 24.796 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [01:02 min]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 43.043 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [01:09 min]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 9.662 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 40.439 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 13.894 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 32.797 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [01:00 min]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 11.333 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 35.122 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 22.939 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 17.568 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 12.339 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 18.325 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 27.889 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 30.148 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [01:28 min]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 25.086 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.657 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 25.302 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 23.268 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.156 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:06 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:17 h
[INFO] Finished at: 2014-12-29T20:45:54-08:00
[INFO] Final Memory: 94M/193M
[INFO] ------------------------------------------------------------------------
[root@Master hadoop-2.6.0-src]#
/2 
