图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › HDFS › 在Linux上编译Hadoop-2.4.0
分享

在Linux上编译Hadoop-2.4.0

laozi888 发表于 2014-10-20 00:41:11 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 3 9381
About云VIP帮助找工作和提升
本帖最后由 pig2 于 2014-10-20 10:02 编辑

目录目录
1
1. 前言
1
2. 安装依赖
1
2.1. 安装ProtocolBuffer
2
2.2. 安装CMake
2
2.3. 安装JDK
2
2.4. 安装Maven
3
3. 编译Hadoop源代码
3
附1:编译环境
5
附2：版本信息
5
附3：常见错误
5
1) unexpected end tag: </ul>
5

1. 前言
Hadoop-2.4.0的源码目录下有个BUILDING.txt文件，它介绍了如何在Linux和Windows下编译源代码，本文基本是遵照BUILDING.txt指示来操作的，这里再做一下简单的提炼。
第一次编译要求能够访问互联网，Hadoop的编译依赖非常多的东西，一定要保证机器可访问互联网，否则难逐一解决所有的编译问题，但第一次之后的编译则不用再下载了。

2. 安装依赖
在编译Hadoop 2.4.0源码之前，需要将下列几个依赖的东西安装好：
1) JDK 1.6或更新版本（本文使用JDK1.7，请不要安装JDK1.8版本，JDK1.8和Hadoop 2.4.0不匹配，编译Hadoop 2.4.0源码时会报很多错误）
2) Maven 3.0或更新版本
3) ProtocolBuffer 2.5.0
4) CMake 2.6或更新版本
5) Findbugs 1.3.9，可选的（本文编译时未安装）

在安装好之后，还需要设置一下环境变量，可以修改/etc/profile，也可以是修改~/.profile，增加如下内容：


  1. export JAVA_HOME=/root/jdk

  2. export CLASSPATH=$JAVA_HOME/lib/tools.jar

  3. export PATH=$JAVA_HOME/bin:$PATH

  4. 

  5. export CMAKE_HOME=/root/cmake

  6. export PATH=$CMAKE_HOME/bin:$PATH

  7. 

  8. export PROTOC_HOME=/root/protobuf

  9. export PATH=$PROTOC_HOME/bin:$PATH

  10. 

  11. export MAVEN_HOME=/root/maven

  12. export PATH=$MAVEN_HOME/bin:$PATH
复制代码



本文以root用户在/root目录下进行安装，但实际可以选择非root用户及非/root目录进行安装。

2.1. 安装ProtocolBuffer标准的automake编译安装方式：
1) cd /root
2) tar xzf protobuf-2.5.0.tar.gz
3) cd protobuf-2.5.0
4) ./conigure --prefix=/root/protobuf
5) make
6) make install

2.2. 安装CMake1) cd /root
2) tar xzf cmake-2.8.12.2.tar.gz
3) cd cmake-2.8.12.2
4) ./bootstrap --prefix=/root/cmake
5) make
6) make install

2.3. 安装JDK1) cd /root
2) tar xzf jdk-7u55-linux-x64.gz
3) cd jdk1.7.0_55
4) ln -s jdk1.7.0_55 jdk

2.4. 安装Maven


1) cd /root
2) tar xzf apache-maven-3.0.5-bin.tar.gz
3) ln -s apache-maven-3.0.5 maven

3.编译Hadoop源代码

完成上述准备工作后，即可通过执行命令：mvn package -Pdist -DskipTests -Dtar，启动对Hadoop源代码的编译。请注意一定不要使用JDK1.8。

编译成功后，jar文件会放在target子目录下，可以在Hadoop源码目录下借用find命令搜索各个target子目录。
编译成功后，会生成Hadoop二进制安装包hadoop-2.4.0.tar.gz，放在源代码的hadoop-dist/target子目录下：
  1. main:

  2.      [exec] $ tar cf hadoop-2.4.0.tar hadoop-2.4.0

  3.      [exec] $ gzip -f hadoop-2.4.0.tar

  4.      [exec]

  5.      [exec] Hadoop dist tar available at: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-2.4.0.tar.gz

  6.      [exec]

  7. [INFO] Executed tasks

  8. [INFO]

  9. [INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---

  10. [INFO] Building jar: /root/hadoop-2.4.0-src/hadoop-dist/target/hadoop-dist-2.4.0-javadoc.jar

  11. [INFO] ------------------------------------------------------------------------

  12. [INFO] Reactor Summary:

  13. [INFO]

  14. [INFO] Apache Hadoop Main ................................ SUCCESS [4.647s]

  15. [INFO] Apache Hadoop Project POM ......................... SUCCESS [5.352s]

  16. [INFO] Apache Hadoop Annotations ......................... SUCCESS [7.239s]

  17. [INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.424s]

  18. [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [2.918s]

  19. [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [6.261s]

  20. [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [5.321s]

  21. [INFO] Apache Hadoop Auth ................................ SUCCESS [5.953s]

  22. [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [3.783s]

  23. [INFO] Apache Hadoop Common .............................. SUCCESS [1:54.010s]

  24. [INFO] Apache Hadoop NFS ................................. SUCCESS [9.721s]

  25. [INFO] Apache Hadoop Common Project ...................... SUCCESS [0.048s]

  26. [INFO] Apache Hadoop HDFS ................................ SUCCESS [4:15.270s]

  27. [INFO] Apache Hadoop HttpFS .............................. SUCCESS [6:18.553s]

  28. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [16.237s]

  29. [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [6.543s]

  30. [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.036s]

  31. [INFO] hadoop-yarn ....................................... SUCCESS [0.051s]

  32. [INFO] hadoop-yarn-api ................................... SUCCESS [1:35.227s]

  33. [INFO] hadoop-yarn-common ................................ SUCCESS [43.216s]

  34. [INFO] hadoop-yarn-server ................................ SUCCESS [0.055s]

  35. [INFO] hadoop-yarn-server-common ......................... SUCCESS [16.476s]

  36. [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [19.942s]

  37. [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.926s]

  38. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [9.804s]

  39. [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [23.320s]

  40. [INFO] hadoop-yarn-server-tests .......................... SUCCESS [1.208s]

  41. [INFO] hadoop-yarn-client ................................ SUCCESS [9.177s]

  42. [INFO] hadoop-yarn-applications .......................... SUCCESS [0.113s]

  43. [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [4.106s]

  44. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [3.265s]

  45. [INFO] hadoop-yarn-site .................................. SUCCESS [0.056s]

  46. [INFO] hadoop-yarn-project ............................... SUCCESS [5.552s]

  47. [INFO] hadoop-mapreduce-client ........................... SUCCESS [0.096s]

  48. [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [37.231s]

  49. [INFO] hadoop-mapreduce-client-common .................... SUCCESS [27.135s]

  50. [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [4.886s]

  51. [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [17.876s]

  52. [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [14.140s]

  53. [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [11.305s]

  54. [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [3.083s]

  55. [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [9.855s]

  56. [INFO] hadoop-mapreduce .................................. SUCCESS [5.110s]

  57. [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [7.778s]

  58. [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [12.973s]

  59. [INFO] Apache Hadoop Archives ............................ SUCCESS [3.265s]

  60. [INFO] Apache Hadoop Rumen ............................... SUCCESS [11.060s]

  61. [INFO] Apache Hadoop Gridmix ............................. SUCCESS [7.412s]

  62. [INFO] Apache Hadoop Data Join ........................... SUCCESS [4.221s]

  63. [INFO] Apache Hadoop Extras .............................. SUCCESS [4.771s]

  64. [INFO] Apache Hadoop Pipes ............................... SUCCESS [0.032s]

  65. [INFO] Apache Hadoop OpenStack support ................... SUCCESS [8.030s]

  66. [INFO] Apache Hadoop Client .............................. SUCCESS [7.730s]

  67. [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.158s]

  68. [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [7.485s]

  69. [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [6.912s]

  70. [INFO] Apache Hadoop Tools ............................... SUCCESS [0.029s]

  71. [INFO] Apache Hadoop Distribution ........................ SUCCESS [40.425s]

  72. [INFO] ------------------------------------------------------------------------

  73. [INFO] BUILD SUCCESS

  74. [INFO] ------------------------------------------------------------------------

  75. [INFO] Total time: 21:57.892s

  76. [INFO] Finished at: Mon Apr 21 14:33:22 CST 2014

  77. [INFO] Final Memory: 88M/243M

  78. [INFO] ------------------------------------------------------------------------
复制代码





附1:编译环境
整个过程是在阿里云64位主机上进行的，2.30GHz单核1G内存：
  1. 

  2. [root@AY140408105805619186Z hadoop-2.4.0-src]# uname -a

  3. Linux AY140408105805619186Z 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

  4. 

  5. [root@AY140408105805619186Z ~]# cat /etc/redhat-release

  6. CentOS release 5.8 (Final)
复制代码



附2：版本信息

名称

版本

包名

说明

Maven

3.0.5

apache-maven-3.0.5-bin.tar.gz

使用3.2.1可能会有问题

CMake

2.8.12.2

cmake-2.8.12.2.tar.gz


JDK

1.7.0

jdk-7u55-linux-x64.gz

不能使用JDK1.8.0

Protocol Buffers

2.5.0

protobuf-2.5.0.tar.gz


Hadoop

2.4.0

hadoop-2.4.0-src.tar.gz




附3：

常见错误1) unexpected end tag: 
  1. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-annotations: MavenReportException: Error while creating archive:

  2. [ERROR] Exit code: 1 - /root/hadoop-2.4.0-src/hadoop-common-project/hadoop-annotations/src/main/java/org/apache/hadoop/classification/InterfaceStability.java:27: error: unexpected end tag: </ul>

  3. [ERROR] * </ul>

  4. [ERROR] ^

  5. [ERROR]

  6. [ERROR] Command line was: /root/jdk1.8.0/jre/../bin/javadoc @options @packages
复制代码


原因是InterfaceStability.java中的注释问题：

解决办法，将JDK换成1.7版本，使用JDK1.8编译就会遇到上述问题，将行删除可以解决问题，但后续还会遇到类似的问题，所以不要使用JDK1.8编译Hadoop 2.4.0。




回复

使用道具 举报

自定义广告语

已有(3)人评论

跳转到指定楼层
正序浏览
pig2 发表于 2014-10-20 10:47:04
string2020 发表于 2014-10-20 09:53
谢谢
我之前用的都是最新的版本去编译（各个源代码），死活没成功。
现在想想，应该是版本不匹配。回去再 ...
特别是 protobuf不能正确返回版本，需要jdk1.7，对于1.6版本，即使protobuf安装成功，也不会返回版本

回复

使用道具 举报

evababy 发表于 2014-11-28 10:04:17
直至2.5.2都需要在pom.xml中增加，否侧commons就错误无了
<plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>2.4.2</version>
          <configuration>
            <skipTests>true</skipTests>
          </configuration>
        </plugin>
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
laozi888

新手上路

关注

  • 主题

  • 帖子

  • 粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000