分享

hive运行job的时候报错

旧收音机 发表于 2015-7-7 11:50:15 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 11 85130
hive> select count(*) from techbbs;
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
   set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
   set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
   set mapreduce.job.reduces=<number>
Starting Job = job_1436192701429_0004, Tracking URL = http://hiter:8088/proxy/application_1436192701429_0004/
Kill Command = /usr/hadoop-2.5.1/bin/hadoop job  -kill job_1436192701429_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
2015-07-07 00:11:21,693 Stage-1 map = 0%,  reduce = 0%
2015-07-07 00:11:37,009 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.33 sec
2015-07-07 00:11:53,332 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 9.2 sec
2015-07-07 00:11:58,933 Stage-1 map = 0%,  reduce = 0%
MapReduce Total cumulative CPU time: 9 seconds 200 msec
Ended Job = job_1436192701429_0004 with errors
Error during job, obtaining debugging information...
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched:
Job 0: Map: 1  Reduce: 1   Cumulative CPU: 9.2 sec   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 9 seconds 200 msec

日志信息:

2015-07-07 00:12:03,378 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1436192701429_0004_000002
2015-07-07 00:12:07,243 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-07-07 00:12:07,260 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-07-07 00:12:07,426 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-07-07 00:12:07,460 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-07-07 00:12:07,461 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@c791b9)
2015-07-07 00:12:07,568 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: The specific max attempts: 2 for application: 4. Attempt num: 2 is last retry: true
2015-07-07 00:12:07,899 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-07-07 00:12:07,906 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-07-07 00:12:09,499 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Attempt num: 2 is last retry: true because a commit was started.
2015-07-07 00:12:09,509 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$NoopEventHandler
2015-07-07 00:12:09,519 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-07-07 00:12:09,523 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-07-07 00:12:09,630 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Will not try to recover. recoveryEnabled: true recoverySupportedByCommitter: false numReduceTasks: 1 shuffleKeyValidForRecovery: true ApplicationAttemptID: 2
2015-07-07 00:12:09,671 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at hdfs://hiter:9000/tmp/hadoop-yarn/staging/root/.staging/job_1436192701429_0004/job_1436192701429_0004_1.jhist
2015-07-07 00:12:11,154 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-07-07 00:12:11,379 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2015-07-07 00:12:11,684 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2015-07-07 00:12:11,684 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
2015-07-07 00:12:11,741 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
2015-07-07 00:12:11,741 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
2015-07-07 00:12:11,742 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
2015-07-07 00:12:12,040 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
2015-07-07 00:12:12,056 WARN [main] org.apache.hadoop.conf.Configuration: job.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
2015-07-07 00:12:12,079 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at hiter/192.168.1.204:8030
2015-07-07 00:12:12,302 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: 8192
2015-07-07 00:12:12,303 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: default
2015-07-07 00:12:12,322 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryCopyService: History file is at hdfs://hiter:9000/tmp/hadoop-yarn/staging/root/.staging/job_1436192701429_0004/job_1436192701429_0004_1.jhist
2015-07-07 00:12:12,734 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1436192701429_0004, File: hdfs://hiter:9000/tmp/hadoop-yarn/staging/root/.staging/job_1436192701429_0004/job_1436192701429_0004_2.jhist
2015-07-07 00:12:13,016 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.io.IOException: Was asked to shut down.
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1488)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1482)
        at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1415)
2015-07-07 00:12:13,027 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1


stderr:

log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

已有(11)人评论

跳转到指定楼层
leo_1989 发表于 2015-7-7 18:23:10
楼主首先确保,运行其它是没有问题的,否则可能就是hive配置问题。如果hive没有问题,在看看yarn。MRAppMaster可能出问题了
回复

使用道具 举报

nextuser 发表于 2015-7-7 22:29:44
试一下select * from table,能否查询成功
回复

使用道具 举报

旧收音机 发表于 2015-7-8 10:31:32
nextuser 发表于 2015-7-7 22:29
试一下select * from table,能否查询成功

能查询,但是要走mapreduce就不行,就报错
回复

使用道具 举报

bob007 发表于 2015-7-8 12:36:41
旧收音机 发表于 2015-7-8 10:31
能查询,但是要走mapreduce就不行,就报错

MRAppMaster启动失败了,看来这个问题还是比较复杂的,跟yarn有关系。重启yarn试试。

这个资料,可能对楼主可能有帮助
Hadoop - MapReduce MRAppMaster-剖析
回复

使用道具 举报

旧收音机 发表于 2015-7-9 09:28:25
bob007 发表于 2015-7-8 12:36
MRAppMaster启动失败了,看来这个问题还是比较复杂的,跟yarn有关系。重启yarn试试。

这个资料,可能 ...

重启不下十次,但是就是不行
回复

使用道具 举报

Alkaloid0515 发表于 2015-7-10 23:00:22
旧收音机 发表于 2015-7-9 09:28
重启不下十次,但是就是不行

整合的,还是单独安装的。整合注意版本兼容
回复

使用道具 举报

旧收音机 发表于 2015-7-10 23:13:42
整合的,但是现在就启动hadoop,然后执行自带的example 中的pi 也报错了,我感觉是任务提交的时候报错的
回复

使用道具 举报

Bill_ 发表于 2015-7-10 23:13:52
明天翻翻源码。
回复

使用道具 举报

Alkaloid0515 发表于 2015-7-10 23:58:48
旧收音机 发表于 2015-7-10 23:13
整合的,但是现在就启动hadoop,然后执行自带的example 中的pi 也报错了,我感觉是任务提交的时候报错的

说的详细点,hadoop既然执行了,那么肯定能执行完毕。除非资源不足了。
说下你的各个版本,还有配置情况
回复

使用道具 举报

12下一页
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条