分享

hadoop2.7汇总:新增功能最新编译64位安装、源码包、API、eclipse插件下载

nettman 2015-4-22 17:19:01 发表于 总结型 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 29 167945
hadoop2.7新增功能

版本更新说明:
http://hadoop.apache.org/docs/r2.7.0/

Common
当使用HTTP proxy server时,认证有所加强.这是对于使用proxy server访问WebHDFS是非常有用的
一种hadoop新的允许直接写Graphite的标准库
规范了HCFS(Hadoop Compatible Filesystem )的相关帮助


HDFS
支持POSIX风格的FileSystem扩展属性。.更多查看用户文档
使用 OfflineImageViewer, 客户端可以通过 WebHDFS API查看fsimage.
NFS网关做了些改进和bug修复。 Hadoop portmapper 不在需要运行网关, 网关可以拒绝非特权端口连接。
SecondaryNameNode, JournalNode, and DataNode 已经可以可视化

YARN
YARN’s REST APIs 现在支持 写/修改 操作. 用户可以通过REST APIs提交和 kill应用程序
时间存储在yarn中,为applications的存储和应用的通用具体信息,支持通过Kerberos认证。
公平调度器支持动态的按等级划分队列,用户队列可以动态的创建在其它父队列里


安装部署:
hadoop2.7集群完全分布式安装

hadoop2.7【单节点】单机、伪分布、分布式安装指导

下载地址

http://apache.fayea.com/hadoop/common/hadoop-2.7.0/

官网下载安装包
hadoop-2.7.0.tar.gz
链接:http://pan.baidu.com/s/1eQ1sUjS 密码:0h97

maven包
hadoop-2.7.0-src.tar
链接:http://pan.baidu.com/s/1pJ7gmlt 密码:xqyw

64位编译包
hadoop-2.7.0-aboutyun-64bit.tar.gz
链接:http://pan.baidu.com/s/1o6kNDGY 密码:jpuj

hadoop2.7-api文档
链接:http://pan.baidu.com/s/1kTDZYPD 密码:j1r9


eclipse插件Linux及window操作系统都不太一样,导致插件使用不了,
最好自己制作适合自己的插件
如果真不会制作也有办法开发,可查看
hadoop开发方式总结及操作指导
############################################

编译64位包[官网已支持64位]

1.编译按照这篇帖子即可成功:
从零教你在Linux环境下(ubuntu)如何编译hadoop2.4
最好在root下编译,否则可能会遇到权限问题




2.总结编译中遇到的两大问题:
1.插件下载不完全

表现:Could not resolve dependencies
例如:
Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0:

解决办法:
这种情况很常见,而且很多都碰到了,他们也是完全按照文档来配置的,但是就不成功,这就是因为插件没有下载完毕造成的。所以尽量多执行几次下面命令
  1. mvn package -Pdist,native -DskipTests -Dtar
复制代码
2.权限问题

权限一般报找不到文件
表现
No such file or directory

例如:
Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml (No such file or directory) -> [Help 1]
解决办法:
在root下编译。然后确保所属权限属于root。
剩下的就是多编译几次


3.相关工具:

1.vmware10
链接:http://pan.baidu.com/s/1bIBuA 密码:3ss9

2.winscp516setup

链接:http://pan.baidu.com/s/1nt64rzj 密码:2109


3.jdk-7u67-linux-x64.tar
链接:http://pan.baidu.com/s/1pJ60syf 密码:9kee

4.protobuf-2.5.0.tar.gz
链接:http://pan.baidu.com/s/1dDgWe6P 密码:xgiu

5.apache-ant-1.9.4-bin.tar.gz
http://pan.baidu.com/s/1c0vjhBy





4.编译遇到问题
1.权限问题
本来已经改变了文件夹的权限,可是文件夹里面的还是没有改变
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................ SUCCESS [4:34.701s]
[INFO] Apache Hadoop Project POM ......................... FAILURE [29.710s]
[INFO] Apache Hadoop Annotations ......................... SKIPPED
[INFO] Apache Hadoop Assemblies .......................... SKIPPED
[INFO] Apache Hadoop Project Dist POM .................... SKIPPED
[INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
[INFO] Apache Hadoop MiniKDC ............................. SKIPPED
[INFO] Apache Hadoop Auth ................................ SKIPPED
[INFO] Apache Hadoop Auth Examples ....................... SKIPPED
[INFO] Apache Hadoop Common .............................. SKIPPED
[INFO] Apache Hadoop NFS ................................. SKIPPED
[INFO] Apache Hadoop KMS ................................. SKIPPED
[INFO] Apache Hadoop Common Project ...................... SKIPPED
[INFO] Apache Hadoop HDFS ................................ SKIPPED
[INFO] Apache Hadoop HttpFS .............................. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
[INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
[INFO] Apache Hadoop HDFS Project ........................ SKIPPED
[INFO] hadoop-yarn ....................................... SKIPPED
[INFO] hadoop-yarn-api ................................... SKIPPED
[INFO] hadoop-yarn-common ................................ SKIPPED
[INFO] hadoop-yarn-server ................................ SKIPPED
[INFO] hadoop-yarn-server-common ......................... SKIPPED
[INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
[INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
[INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
[INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
[INFO] hadoop-yarn-server-tests .......................... SKIPPED
[INFO] hadoop-yarn-client ................................ SKIPPED
[INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
[INFO] hadoop-yarn-applications .......................... SKIPPED
[INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
[INFO] hadoop-yarn-site .................................. SKIPPED
[INFO] hadoop-yarn-registry .............................. SKIPPED
[INFO] hadoop-yarn-project ............................... SKIPPED
[INFO] hadoop-mapreduce-client ........................... SKIPPED
[INFO] hadoop-mapreduce-client-core ...................... SKIPPED
[INFO] hadoop-mapreduce-client-common .................... SKIPPED
[INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
[INFO] hadoop-mapreduce-client-app ....................... SKIPPED
[INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
[INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
[INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
[INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
[INFO] hadoop-mapreduce .................................. SKIPPED
[INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
[INFO] Apache Hadoop Distributed Copy .................... SKIPPED
[INFO] Apache Hadoop Archives ............................ SKIPPED
[INFO] Apache Hadoop Rumen ............................... SKIPPED
[INFO] Apache Hadoop Gridmix ............................. SKIPPED
[INFO] Apache Hadoop Data Join ........................... SKIPPED
[INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
[INFO] Apache Hadoop Extras .............................. SKIPPED
[INFO] Apache Hadoop Pipes ............................... SKIPPED
[INFO] Apache Hadoop OpenStack support ................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
[INFO] Apache Hadoop Azure support ....................... SKIPPED
[INFO] Apache Hadoop Client .............................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
[INFO] Apache Hadoop Tools Dist .......................... SKIPPED
[INFO] Apache Hadoop Tools ............................... SKIPPED
[INFO] Apache Hadoop Distribution ........................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:34.724s
[INFO] Finished at: Wed Apr 22 03:40:41 PDT 2015
[INFO] Final Memory: 28M/67M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (create-testdirs) on project hadoop-project: Error executing ant tasks: /home/aboutyun/hadoop-2.7.0-src/hadoop-project/target/antrun/build-main.xml (No such file or directory) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluen ... oExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :hadoop-project

查看下权限,都是root
  1. aboutyun@ubuntu:~/hadoop-2.7.0-src$ ll
复制代码

  1. total 120
  2. drwxr-xr-x 15 aboutyun aboutyun  4096 Apr 10 11:39 ./
  3. drwxr-xr-x 24 aboutyun aboutyun  4096 Apr 22 03:32 ../
  4. -rw-r--r--  1 root     root     13110 Apr 10 11:35 BUILDING.txt
  5. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 dev-support/
  6. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-assemblies/
  7. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-client/
  8. drwxr-xr-x 10 root     root      4096 Apr 22 03:31 hadoop-common-project/
  9. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-dist/
  10. drwxr-xr-x  6 root     root      4096 Apr 22 03:31 hadoop-hdfs-project/
  11. drwxr-xr-x  9 root     root      4096 Apr 22 03:31 hadoop-mapreduce-project/
  12. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-maven-plugins/
  13. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-minicluster/
  14. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-project/
  15. drwxr-xr-x  2 root     root      4096 Apr 22 03:31 hadoop-project-dist/
  16. drwxr-xr-x 16 root     root      4096 Apr 22 03:31 hadoop-tools/
  17. drwxr-xr-x  3 root     root      4096 Apr 22 03:31 hadoop-yarn-project/
  18. -rw-r--r--  1 root     root     15429 Apr 10 11:35 LICENSE.txt
  19. -rw-r--r--  1 root     root       101 Apr 10 11:35 NOTICE.txt
  20. -rw-r--r--  1 root     root     18893 Apr 10 11:35 pom.xml
  21. -rw-r--r--  1 root     root      1366 Apr 10 11:35 README.txt
复制代码
修改为aboutyun.
执行下面命令:
  1. sudo chown aboutyun:aboutyun hadoop-2.7.0-src/*
复制代码





修改完毕

错误2
  1. Downloaded: http://repo.maven.apache.org/maven2/org/codehaus/plexus/plexus-archiver/2.4.4/plexus-archiver-2.4.4.jar (161 KB at 34.7 KB/sec)
  2. [INFO] ------------------------------------------------------------------------
  3. [INFO] Reactor Summary:
  4. [INFO]
  5. [INFO] Apache Hadoop Main ................................ SUCCESS [2.569s]
  6. [INFO] Apache Hadoop Project POM ......................... FAILURE [2:16.305s]
  7. [INFO] Apache Hadoop Annotations ......................... SKIPPED
  8. [INFO] Apache Hadoop Assemblies .......................... SKIPPED
  9. [INFO] Apache Hadoop Project Dist POM .................... SKIPPED
  10. [INFO] Apache Hadoop Maven Plugins ....................... SKIPPED
  11. [INFO] Apache Hadoop MiniKDC ............................. SKIPPED
  12. [INFO] Apache Hadoop Auth ................................ SKIPPED
  13. [INFO] Apache Hadoop Auth Examples ....................... SKIPPED
  14. [INFO] Apache Hadoop Common .............................. SKIPPED
  15. [INFO] Apache Hadoop NFS ................................. SKIPPED
  16. [INFO] Apache Hadoop KMS ................................. SKIPPED
  17. [INFO] Apache Hadoop Common Project ...................... SKIPPED
  18. [INFO] Apache Hadoop HDFS ................................ SKIPPED
  19. [INFO] Apache Hadoop HttpFS .............................. SKIPPED
  20. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SKIPPED
  21. [INFO] Apache Hadoop HDFS-NFS ............................ SKIPPED
  22. [INFO] Apache Hadoop HDFS Project ........................ SKIPPED
  23. [INFO] hadoop-yarn ....................................... SKIPPED
  24. [INFO] hadoop-yarn-api ................................... SKIPPED
  25. [INFO] hadoop-yarn-common ................................ SKIPPED
  26. [INFO] hadoop-yarn-server ................................ SKIPPED
  27. [INFO] hadoop-yarn-server-common ......................... SKIPPED
  28. [INFO] hadoop-yarn-server-nodemanager .................... SKIPPED
  29. [INFO] hadoop-yarn-server-web-proxy ...................... SKIPPED
  30. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SKIPPED
  31. [INFO] hadoop-yarn-server-resourcemanager ................ SKIPPED
  32. [INFO] hadoop-yarn-server-tests .......................... SKIPPED
  33. [INFO] hadoop-yarn-client ................................ SKIPPED
  34. [INFO] hadoop-yarn-server-sharedcachemanager ............. SKIPPED
  35. [INFO] hadoop-yarn-applications .......................... SKIPPED
  36. [INFO] hadoop-yarn-applications-distributedshell ......... SKIPPED
  37. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SKIPPED
  38. [INFO] hadoop-yarn-site .................................. SKIPPED
  39. [INFO] hadoop-yarn-registry .............................. SKIPPED
  40. [INFO] hadoop-yarn-project ............................... SKIPPED
  41. [INFO] hadoop-mapreduce-client ........................... SKIPPED
  42. [INFO] hadoop-mapreduce-client-core ...................... SKIPPED
  43. [INFO] hadoop-mapreduce-client-common .................... SKIPPED
  44. [INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED
  45. [INFO] hadoop-mapreduce-client-app ....................... SKIPPED
  46. [INFO] hadoop-mapreduce-client-hs ........................ SKIPPED
  47. [INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED
  48. [INFO] hadoop-mapreduce-client-hs-plugins ................ SKIPPED
  49. [INFO] Apache Hadoop MapReduce Examples .................. SKIPPED
  50. [INFO] hadoop-mapreduce .................................. SKIPPED
  51. [INFO] Apache Hadoop MapReduce Streaming ................. SKIPPED
  52. [INFO] Apache Hadoop Distributed Copy .................... SKIPPED
  53. [INFO] Apache Hadoop Archives ............................ SKIPPED
  54. [INFO] Apache Hadoop Rumen ............................... SKIPPED
  55. [INFO] Apache Hadoop Gridmix ............................. SKIPPED
  56. [INFO] Apache Hadoop Data Join ........................... SKIPPED
  57. [INFO] Apache Hadoop Ant Tasks ........................... SKIPPED
  58. [INFO] Apache Hadoop Extras .............................. SKIPPED
  59. [INFO] Apache Hadoop Pipes ............................... SKIPPED
  60. [INFO] Apache Hadoop OpenStack support ................... SKIPPED
  61. [INFO] Apache Hadoop Amazon Web Services support ......... SKIPPED
  62. [INFO] Apache Hadoop Azure support ....................... SKIPPED
  63. [INFO] Apache Hadoop Client .............................. SKIPPED
  64. [INFO] Apache Hadoop Mini-Cluster ........................ SKIPPED
  65. [INFO] Apache Hadoop Scheduler Load Simulator ............ SKIPPED
  66. [INFO] Apache Hadoop Tools Dist .......................... SKIPPED
  67. [INFO] Apache Hadoop Tools ............................... SKIPPED
  68. [INFO] Apache Hadoop Distribution ........................ SKIPPED
  69. [INFO] ------------------------------------------------------------------------
  70. [INFO] BUILD FAILURE
  71. [INFO] ------------------------------------------------------------------------
  72. [INFO] Total time: 2:25.054s
  73. [INFO] Finished at: Wed Apr 22 03:50:50 PDT 2015
  74. [INFO] Final Memory: 30M/71M
  75. [INFO] ------------------------------------------------------------------------
  76. [ERROR] Failed to execute goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) on project hadoop-project: Execution hadoop-java-sources of goal org.apache.maven.plugins:maven-source-plugin:2.3:jar-no-fork failed: Plugin org.apache.maven.plugins:maven-source-plugin:2.3 or one of its dependencies could not be resolved: Could not transfer artifact org.apache.maven.reporting:maven-reporting-api:jar:2.0.6 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/maven/reporting/maven-reporting-api/2.0.6/maven-reporting-api-2.0.6.jar from central failed: Read timed out -> [Help 1]
  77. [ERROR]
  78. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  79. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  80. [ERROR]
  81. [ERROR] For more information about the errors and possible solutions, please read the following articles:
  82. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
  83. [ERROR]
  84. [ERROR] After correcting the problems, you can resume the build with the command
  85. [ERROR]   mvn <goals> -rf :hadoop-project
复制代码

执行了下
  1. mvn install
复制代码

第二个过了,后面又遇到了问题,文件夹里,还有文件里,而且还是root,最后直接切换root,免得在遇到权限问题

切换root遇到问题

  1. [ERROR] The build could not read 1 project -> [Help 1]
  2. [ERROR]   
  3. [ERROR]   The project org.apache.hadoop:hadoop-main:2.7.0 (/root/hadoop-2.7.0-src/pom.xml) has 1 error
  4. [ERROR]     Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.5.0 or one of its dependencies could not be resolved: Failed to collect dependencies for org.apache.felix:maven-bundle-plugin:jar:2.5.0 (): Failed to read artifact descriptor for org.apache.maven:maven-plugin-parameter-documenter:jar:2.0.6: Could not transfer artifact org.apache.maven:maven-plugin-parameter-documenter:pom:2.0.6 from/to central (http://repo.maven.apache.org/maven2): Read timed out -> [Help 2]
  5. [ERROR]
  6. [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
  7. [ERROR] Re-run Maven using the -X switch to enable full debug logging.
  8. [ERROR]
  9. [ERROR] For more information about the errors and possible solutions, please read the following articles:
  10. [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
  11. [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
复制代码

又遇到
  1. [INFO] ------------------------------------------------------------------------
  2. [ERROR] Failed to execute goal on project hadoop-minikdc: Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0: The following artifacts could not be resolved: org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15, org.apache.directory.api:api-ldap-codec-core:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-aci:jar:1.0.0-M20, org.apache.directory.api:api-ldap-extras-util:jar:1.0.0-M20, org.apache.directory.api:api-ldap-model:jar:1.0.0-M20, org.apache.directory.api:api-util:jar:1.0.0-M20, org.apache.mina:mina-core:jar:2.0.0-M5, net.sf.ehcache:ehcache-core:jar:2.4.4, org.apache.directory.server:apacheds-interceptors-admin:jar:2.0.0-M15: Could not transfer artifact org.apache.directory.server:apacheds-core-api:jar:2.0.0-M15 from/to central (http://repo.maven.apache.org/maven2): GET request of: org/apache/directory/server/apacheds-core-api/2.0.0-M15/apacheds-core-api-2.0.0-M15.jar from central failed: Read timed out -> [Help 1]
复制代码


总结:
遇到这种情况,一般都是插件没有下载
  1. Could not resolve dependencies for project org.apache.hadoop:hadoop-minikdc:jar:2.7.0:
复制代码
可以多次执行,下载完毕。




参考:

从零教你在Linux环境下(ubuntu 12.04)如何编译hadoop2.4















加微信w3aboutyun,可拉入技术爱好者群

已有(30)人评论

跳转到指定楼层
pig2 发表于 2015-4-26 17:13:53
Apache Hadoop 2.7.0 发布下载,包括大量显著改进,值得关注的改进如下:
  • 重大改进
    • 此版本不再支持 JDK 6 运行时,仅支持 JDK 7+
    • 此版本不适用于生产环境!还有一些重要的问题需要通过测试,用于生产环境的用户请等待 2.7.1/2.7.2

Hadoop Common



    • 支持 Windows Azure 存储 —— Blob
Hadoop HDFS



    • 支持文件截断
    • 支持每个存储类型配额
    • 支持可变长度的文件块
Hadoop YARN



    • YARN 认证可插拔
    • 自动分享,全局缓存 YARN 本地化资源(测试阶段)
Hadoop MapReduce



    • 限制一个作业运行的 Map/Reduce 任务
    • 加快大量输出文件时大型作业的 FileOutputCommitter 速度
完整改进请看发行说明
回复

使用道具 举报

深沉 发表于 2015-4-23 09:22:31
很好很给力
回复

使用道具 举报

wnan1981 发表于 2015-4-23 10:03:10
资源不错,谢谢分享

回复

使用道具 举报

hb1984 发表于 2015-4-23 18:14:49
谢谢楼主分享。         
回复

使用道具 举报

feng01301218 发表于 2015-4-23 19:03:36
如果您要查看本帖隐
回复

使用道具 举报

cochou 发表于 2015-4-24 12:12:15
回复

使用道具 举报

839565022 发表于 2015-4-24 12:57:38
好想学习学习呀
回复

使用道具 举报

oppohome 发表于 2015-4-24 14:31:23
正需要资料呢,谢谢
回复

使用道具 举报

li5953150 发表于 2015-4-24 15:09:33
你好 我想问一下  我从官方下载Hadoop  怎么知道是32位编译  在哪看
回复

使用道具 举报

123下一页
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条