立即注册 登录
About云-梭伦科技 返回首页

pig2的个人空间 https://www.aboutyun.com/?61 [收藏] [复制] [分享] [RSS]

日志

hadoop2.2运行mapreduce（wordcount）问题总结

已有 2225 次阅读2014-5-15 13:16 |个人分类:云技术基础

本文链接：
http://www.aboutyun.com/thread-7728-1-1.html
问题导读：
1.出现Not a valid JAR:可能原因是什么？
2.运行wordcount，路径不正确会报什么错误？
3.一旦输入数据丢失，mapreduce正在运行，会出现什么错误？







这里对运行mapreduce，做一个问题汇总：第一个问题：Not a valid JAR:
下面举例，对里面的内容分析：
  1. hadoop jar  /hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
复制代码

hadoop-mapreduce-examples-2.2.0.jar 位于下面路径中，所以如果路径不对会报如下错误：
  1. Not a valid JAR: /hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar



 


那么正确写法为：
hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount

同时要注意输入输出路径不正确也有可能会报这个错误,还有另外一种就是你的包真的无效了。

第二个问题：Could not complete /tmp/hadoop-yarn/staging/job_1400084979891_0001/job.jar retrying...
这里不断进行尝试，错误表现如下：
  1. Could not complete /tmp/hadoop-yarn/staging/yonghuming/.staging/job_1400084979891_0001/job.jar retrying...

解决方法：

出上面问题是因为集群处于安全模式，（集群格式化之后，一般会处于安全模式，所以这时候运行mapredcue，最好先检查一下集群模式）
  1. /hadoop dfsadmin -safemode leave


第三个问题：Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002
对于上面错误很是不能保存block，但是奇怪的172.16.77.15，并没有挂掉。错误如下：
  1. hadoop jar /usr/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /data/wordcount /output/wordcount
  2. 14/05/14 09:38:55 INFO client.RMProxy: Connecting to ResourceManager at master/172.16.77.15:8032
  3. 14/05/14 09:38:56 INFO input.FileInputFormat: Total input paths to process : 1
  4. 14/05/14 09:38:56 INFO mapreduce.JobSubmitter: number of splits:1
  5. 14/05/14 09:38:56 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
  6. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
  7. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
  8. 14/05/14 09:38:56 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
  9. 14/05/14 09:38:56 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
  10. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
  11. 14/05/14 09:38:56 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
  12. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
  13. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
  14. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
  15. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
  16. 14/05/14 09:38:56 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
  17. 14/05/14 09:38:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1400084979891_0002
  18. 14/05/14 09:38:58 INFO impl.YarnClientImpl: Submitted application application_1400084979891_0002 to ResourceManager at master/172.16.77.15:8032
  19. 14/05/14 09:38:58 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1400084979891_0002/
  20. 14/05/14 09:38:58 INFO mapreduce.Job: Running job: job_1400084979891_0002
  21. 14/05/14 09:39:19 INFO mapreduce.Job: Job job_1400084979891_0002 running in uber mode : false
  22. 14/05/14 09:39:19 INFO mapreduce.Job:  map 0% reduce 0%
  23. 14/05/14 09:40:12 INFO mapreduce.Job:  map 100% reduce 0%
  24. 14/05/14 09:40:12 INFO mapreduce.Job: Task Id : attempt_1400084979891_0002_m_000000_0, Status : FAILED
  25. Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 

  27. file=/data/wordcount/inputWord
  28.         at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
  29.         at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
  30.         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
  31.         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
  32.         at java.io.DataInputStream.read(DataInputStream.java:100)
  33.         at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  34.         at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  35.         at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
  36.         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
  37.         at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
  38.         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
  39.         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
  40.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
  41.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  42.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  43.         at java.security.AccessController.doPrivileged(Native Method)
  44.         at javax.security.auth.Subject.doAs(Subject.java:415)
  45.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  46.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

  48. 14/05/14 09:40:14 INFO mapreduce.Job:  map 0% reduce 0%
  49. 14/05/14 09:41:09 INFO mapreduce.Job: Task Id : attempt_1400084979891_0002_m_000000_1, Status : FAILED
  50. Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 

  52. file=/data/wordcount/inputWord
  53.         at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
  54.         at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
  55.         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
  56.         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
  57.         at java.io.DataInputStream.read(DataInputStream.java:100)
  58.         at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  59.         at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  60.         at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
  61.         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
  62.         at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
  63.         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
  64.         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
  65.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
  66.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  67.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  68.         at java.security.AccessController.doPrivileged(Native Method)
  69.         at javax.security.auth.Subject.doAs(Subject.java:415)
  70.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  71.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

  73. 14/05/14 09:41:33 INFO mapreduce.Job: Task Id : attempt_1400084979891_0002_m_000000_2, Status : FAILED
  74. Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 

  76. file=/data/wordcount/inputWord
  77.         at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
  78.         at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
  79.         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
  80.         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
  81.         at java.io.DataInputStream.read(DataInputStream.java:100)
  82.         at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)
  83.         at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
  84.         at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:164)
  85.         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:532)
  86.         at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
  87.         at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
  88.         at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
  89.         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
  90.         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
  91.         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
  92.         at java.security.AccessController.doPrivileged(Native Method)
  93.         at javax.security.auth.Subject.doAs(Subject.java:415)
  94.         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
  95.         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

  97. 14/05/14 09:41:59 INFO mapreduce.Job:  map 100% reduce 100%
  98. 14/05/14 09:41:59 INFO mapreduce.Job: Job job_1400084979891_0002 failed with state FAILED due to: Task failed task_1400084979891_0002_m_000000
  99. Job failed as tasks failed. failedMaps:1 failedReduces:0

  101. 14/05/14 09:41:59 INFO mapreduce.Job: Counters: 5
  102.         Job Counters 
  103.                 Failed map tasks=4
  104.                 Launched map tasks=4
  105.                 Other local map tasks=4
  106.                 Total time spent by all maps in occupied slots (ms)=152716
  107.                 Total time spent by all reduces in occupied slots (ms)=0

出现上面错误，你可以使用下面命令来进行检查：
  1. hadoop fs -text /data/wordcount/inputWord

解释一下上面内容：
inputWord是上传文件的内容，如果查看不了，如下错误，说明说明DataNode失去连接，或则已经不能使用了


  1. hadoop fs -text /data/wordcount/inputWord
  2. 14/05/14 10:29:23 INFO hdfs.DFSClient: No node available for BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 file=/data/wordcount/inputWord
  3. 14/05/14 10:29:23 INFO hdfs.DFSClient: Could not obtain BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 from any node: java.io.IOException: No live nodes 

  5. contain current block. Will get new block locations from namenode and retry...
  6. 14/05/14 10:29:23 WARN hdfs.DFSClient: DFS chooseDataNode: got # 1 IOException, will wait for 1611.781101650108 msec.
  7. 14/05/14 10:29:24 INFO hdfs.DFSClient: No node available for BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 file=/data/wordcount/inputWord
  8. 14/05/14 10:29:24 INFO hdfs.DFSClient: Could not obtain BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 from any node: java.io.IOException: No live nodes 

  10. contain current block. Will get new block locations from namenode and retry...
  11. 14/05/14 10:29:24 WARN hdfs.DFSClient: DFS chooseDataNode: got # 2 IOException, will wait for 3653.3994797694913 msec.
  12. 14/05/14 10:29:28 INFO hdfs.DFSClient: No node available for BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 file=/data/wordcount/inputWord
  13. 14/05/14 10:29:28 INFO hdfs.DFSClient: Could not obtain BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 from any node: java.io.IOException: No live nodes 

  15. contain current block. Will get new block locations from namenode and retry...
  16. 14/05/14 10:29:28 WARN hdfs.DFSClient: DFS chooseDataNode: got # 3 IOException, will wait for 6618.009385385317 msec.
  17. 14/05/14 10:29:35 WARN hdfs.DFSClient: DFS Read
  18. org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 file=/data/wordcount/inputWord
  19.         at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:838)
  20.         at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:526)
  21.         at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:749)
  22.         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:793)
  23.         at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:601)
  24.         at java.io.DataInputStream.readShort(DataInputStream.java:312)
  25.         at org.apache.hadoop.fs.shell.Display$Text.getInputStream(Display.java:130)
  26.         at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:98)
  27.         at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
  28.         at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
  29.         at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
  30.         at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
  31.         at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
  32.         at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
  33.         at org.apache.hadoop.fs.FsShell.run(FsShell.java:255)
  34.         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  35.         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
  36.         at org.apache.hadoop.fs.FsShell.main(FsShell.java:305)
  37. text: Could not obtain block: BP-1908912651-172.16.77.15-1399795457132:blk_1073741826_1002 file=/data/wordcount/inputWord



路过
雷人
握手
鲜花
鸡蛋
收藏 分享邀请 举报

全部作者的其他最新日志

评论 (0 个评论)

facelist doodle 涂鸦板

您需要登录后才可以评论 登录 | 立即注册

关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000