分享

新手求助,MapReduce程序运行出错,非计算机专业,恳请大神介绍得浅显点,感激不尽!

1601365 发表于 2017-4-13 21:06:01 [显示全部楼层] 只看大图 回帖奖励 阅读模式 关闭右栏 10 8056
david@s0:~/Downloads$ hadoop jar HadoopDemo2.jar /user/it18zhang/ncdc_data/19*.gz /user/it18zhang/out
17/04/13 20:14:34 INFO client.RMProxy: Connecting to ResourceManager at s0/192.168.20.128:8032
17/04/13 20:14:34 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/04/13 20:14:35 INFO input.FileInputFormat: Total input paths to process : 2
17/04/13 20:14:35 INFO mapreduce.JobSubmitter: number of splits:2
17/04/13 20:14:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1492080262949_0002
17/04/13 20:14:36 INFO impl.YarnClientImpl: Submitted application application_1492080262949_0002
17/04/13 20:14:36 INFO mapreduce.Job: The url to track the job: http://s0:8088/proxy/application_1492080262949_0002/
17/04/13 20:14:36 INFO mapreduce.Job: Running job: job_1492080262949_0002
17/04/13 20:14:48 INFO mapreduce.Job: Job job_1492080262949_0002 running in uber mode : false
17/04/13 20:14:48 INFO mapreduce.Job:  map 0% reduce 0%
17/04/13 20:15:23 INFO mapreduce.Job: Task Id : attempt_1492080262949_0002_m_000000_0, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741826_1002 file=/user/it18zhang/ncdc_data/1902.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:184)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
...........(略)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/04/13 20:15:23 INFO mapreduce.Job: Task Id : attempt_1492080262949_0002_m_000001_0, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741825_1001 file=/user/it18zhang/ncdc_data/1901.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)

.............(略)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/04/13 20:15:50 INFO mapreduce.Job: Task Id : attempt_1492080262949_0002_m_000000_1, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741826_1002 file=/user/it18zhang/ncdc_data/1902.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
..........(略)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/04/13 20:16:10 INFO mapreduce.Job:  map 50% reduce 0%
17/04/13 20:16:10 INFO mapreduce.Job: Task Id : attempt_1492080262949_0002_m_000000_2, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741826_1002 file=/user/it18zhang/ncdc_data/1902.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:144)
..............(略)
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/04/13 20:16:11 INFO mapreduce.Job:  map 0% reduce 0%
17/04/13 20:16:12 INFO mapreduce.Job: Task Id : attempt_1492080262949_0002_m_000001_1, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741825_1001 file=/user/it18zhang/ncdc_data/1901.gz
at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:983)
at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:642)
at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:882)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:934)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:159)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:143)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)
at java.io.InputStream.read(InputStream.java:101)
at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)
at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)
at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)
............(略)

17/04/13 20:16:38 INFO mapreduce.Job:  map 100% reduce 100%
17/04/13 20:16:39 INFO mapreduce.Job: Job job_1492080262949_0002 failed with state FAILED due to: Task failed task_1492080262949_0002_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

17/04/13 20:16:40 INFO mapreduce.Job: Counters: 16
Job Counters
Failed map tasks=6
Killed map tasks=1
Killed reduce tasks=1
Launched map tasks=7
Other local map tasks=7
Total time spent by all maps in occupied slots (ms)=200749
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=200749
Total time spent by all reduce tasks (ms)=0
Total vcore-milliseconds taken by all map tasks=200749
Total vcore-milliseconds taken by all reduce tasks=0
Total megabyte-milliseconds taken by all map tasks=205566976
Total megabyte-milliseconds taken by all reduce tasks=0
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
false

已有(10)人评论

跳转到指定楼层
wqkenqing 发表于 2017-4-14 09:28:20
你看看你集群中的DataNode节点是否正常哟。
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block
回复

使用道具 举报

tanxl 发表于 2017-4-14 09:59:28
问一下:你的hadoop集群能跑基本的wordcount程序吗,先试一下跑一下wordcount程序
回复

使用道具 举报

einhep 发表于 2017-4-14 13:15:27
jps看下datanode可能有挂掉的。
最好重启下集群。
有的datanode在刚启动的时候是没有问题的,但是一会就挂掉了。
回复

使用道具 举报

1601365 发表于 2017-4-14 15:58:57
wqkenqing 发表于 2017-4-14 09:28
你看看你集群中的DataNode节点是否正常哟。
org.apache.hadoop.hdfs.BlockMissingException: Could not ob ...

通过jps命令查看过了,都活的好好的,我搭建的是一个NameNode节点和两个DataNode节点,但DataNode节点的日志文件出现了一个错误,都在图片里了
求助.jpg
求助1.jpg
回复

使用道具 举报

1601365 发表于 2017-4-14 16:05:32
tanxl 发表于 2017-4-14 09:59
问一下:你的hadoop集群能跑基本的wordcount程序吗,先试一下跑一下wordcount程序

WordCount程序也一样出错,而且有时候还有Interruptedexception,哎,卡了好几天,要崩溃了
回复

使用道具 举报

sstutu 发表于 2017-4-14 16:07:01
1601365 发表于 2017-4-14 15:58
通过jps命令查看过了,都活的好好的,我搭建的是一个NameNode节点和两个DataNode节点,但DataNode节点的 ...

一个个解决,你这里明显有问题。下面红字部分先解决掉。为什么还有gz文件。datanode根本不能保存。
attempt_1492080262949_0002_m_000000_0, Status : FAILED
Error: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1808748719-192.168.20.128-1488809333278:blk_1073741826_1002 file=/user/it18zhang/ncdc_data/1902.gz

回复

使用道具 举报

1601365 发表于 2017-4-14 16:11:22
einhep 发表于 2017-4-14 13:15
jps看下datanode可能有挂掉的。
最好重启下集群。
有的datanode在刚启动的时候是没有问题的,但是一会就 ...

jps过了,DataNode节点也都还在,目前发现在运行日志里有错误,还有异常,哎,好悲催
警告.jpg
求助1.jpg
回复

使用道具 举报

1601365 发表于 2017-4-14 16:24:11
sstutu 发表于 2017-4-14 16:07
一个个解决,你这里明显有问题。下面红字部分先解决掉。为什么还有gz文件。datanode根本不能保存。
atte ...

我原先看教学视频上是gz文件,如果不是gz文件,我最好应该上传什么文件好啊。
回复

使用道具 举报

tanxl 发表于 2017-4-14 17:43:19
1601365 发表于 2017-4-14 16:05
WordCount程序也一样出错,而且有时候还有Interruptedexception,哎,卡了好几天,要崩溃了

那最好是先把hadoop集群搭建正确了 ,先跑wordcount程序确保集群能够正常运行,再完成自己的mapreduce程序为好。
回复

使用道具 举报

12下一页
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条