分享

cdh hive任务出现异常

JME 发表于 2015-4-7 00:07:00 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 15 50701
我使用的cdh 版本是5.3.0  ,hive 的版本是0.13.1 .在执行hive任务的时候,时不时出现以下异常.重新跑,又不会,并不是 每次都会出现,但基本上每天都出现,我分析了,没找出具体原因,求大神指点.
-----------------------------------------------------------
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 109
Serialization trace:
parentOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:366)
at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:277)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:258)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:451)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:444)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:588)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:169)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 109
Serialization trace:
parentOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:767)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:131)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:112)
at org.apache.hive.com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:776)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:139)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:17)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:694)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:943)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:851)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:865)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:335)
... 13 more


FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-15: Map: 2  Reduce: 1   Cumulative CPU: 13.47 sec   HDFS Read: 101706433 HDFS Write: 8804 SUCCESS
Stage-Stage-11: Map: 2  Reduce: 1   Cumulative CPU: 14.2 sec   HDFS Read: 101706433 HDFS Write: 345 SUCCESS
Stage-Stage-1: Map: 2  Reduce: 1   Cumulative CPU: 17.88 sec   HDFS Read: 101706433 HDFS Write: 40943 SUCCESS
Stage-Stage-16: Map: 1  Reduce: 1   Cumulative CPU: 2.51 sec   HDFS Read: 9164 HDFS Write: 4883 SUCCESS
Stage-Stage-12: Map: 1  Reduce: 1   Cumulative CPU: 2.53 sec   HDFS Read: 705 HDFS Write: 301 SUCCESS
Stage-Stage-2: Map: 1  Reduce: 1   Cumulative CPU: 2.96 sec   HDFS Read: 41303 HDFS Write: 18015 SUCCESS
Stage-Stage-17: Map: 1  Reduce: 1   Cumulative CPU: 3.25 sec   HDFS Read: 5243 HDFS Write: 4883 SUCCESS
Stage-Stage-13: Map: 1  Reduce: 1   Cumulative CPU: 2.62 sec   HDFS Read: 661 HDFS Write: 301 SUCCESS
Stage-Stage-3: Map: 1  Reduce: 1   Cumulative CPU: 2.8 sec   HDFS Read: 18375 HDFS Write: 18015 SUCCESS
Stage-Stage-18: Map: 1  Reduce: 1   Cumulative CPU: 2.52 sec   HDFS Read: 5243 HDFS Write: 4883 SUCCESS
Stage-Stage-14: Map: 1  Reduce: 1   Cumulative CPU: 2.17 sec   HDFS Read: 661 HDFS Write: 301 SUCCESS
Stage-Stage-4: Map: 1  Reduce: 1   Cumulative CPU: 2.35 sec   HDFS Read: 18375 HDFS Write: 18015 SUCCESS
Stage-Stage-21: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 minutes 9 seconds 260 msec

已有(14)人评论

跳转到指定楼层
rsgg03 发表于 2015-4-7 07:13:21
会不会是读取的时候,读取失败或则序列化失败
楼主试试
JavaSerializer
回复

使用道具 举报

JME 发表于 2015-4-7 09:22:52
rsgg03 发表于 2015-4-7 07:13
会不会是读取的时候,读取失败或则序列化失败
楼主试试
JavaSerializer

我使用的hive sql 操作,不是每次都出现这样的情况,但我那几个任务 ,基本每天都会出现.1-2次这样的情况.不知道为何,困扰我很久了.
回复

使用道具 举报

sstutu 发表于 2015-4-7 14:07:35
JME 发表于 2015-4-7 09:22
我使用的hive sql 操作,不是每次都出现这样的情况,但我那几个任务 ,基本每天都会出现.1-2次这样的情况.不 ...

尝试小数据量试试,如果小数据量出现问题,可能是配置、或则程序的问题

回复

使用道具 举报

JME 发表于 2015-4-7 14:34:49
sstutu 发表于 2015-4-7 14:07
尝试小数据量试试,如果小数据量出现问题,可能是配置、或则程序的问题

我觉的和数据量没有关系,因为 有比他更大的数据操作 都没有问题,而且也不是每次 都出现,我定时了每小时跑的任务,一天中可能会出现这样的问题
回复

使用道具 举报

bbhhhh 发表于 2015-4-27 10:20:20
和lz遇到了同样的问题,确认和数据量无关,不是每次都出现,我有20个hive sql需要执行,出现错误的sql也不是固定的。
我的环境是hadoop2.5-cdh5.2.1, hive0.13.1-cdh5.2.1,难道其他人就没碰到过?
回复

使用道具 举报

JME 发表于 2015-4-27 14:01:14
bbhhhh 发表于 2015-4-27 10:20
和lz遇到了同样的问题,确认和数据量无关,不是每次都出现,我有20个hive sql需要执行,出现错误的sql也不 ...

确实,不是 每次 都出现,请问楼上,你们 是怎么解决这个问题的,求指教啊
回复

使用道具 举报

bbhhhh 发表于 2015-4-27 14:17:02
我不也正在问吗,目前一头雾水,刚才又跑了一次,一切又正常了,太妖了。
回复

使用道具 举报

whmshop 发表于 2015-6-21 08:25:15
这个没人解决吗?我也碰到了
回复

使用道具 举报

JME 发表于 2015-6-21 12:21:55
我也是,找过好多资料,都是一头雾水,
回复

使用道具 举报

12下一页
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条