分享

reduce卡住不动了,如何查问题

ananan36 发表于 2015-9-15 18:03:00 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 2 12358
select row_number() over(PARTITION BY 1) as seq,1,2 from test_04
2015-09-15 17:48:34,856 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2237.58 sec
2015-09-15 17:49:35,174 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2299.08 sec
2015-09-15 17:50:35,484 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2361.36 sec
2015-09-15 17:51:35,769 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2421.21 sec
2015-09-15 17:52:36,062 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2481.82 sec
2015-09-15 17:53:36,294 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2544.86 sec
2015-09-15 17:54:36,556 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2605.48 sec
2015-09-15 17:55:36,888 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2669.11 sec
2015-09-15 17:56:37,150 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2727.11 sec
2015-09-15 17:57:37,393 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2790.74 sec
2015-09-15 17:58:37,569 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2851.31 sec
2015-09-15 17:59:37,738 Stage-1 map = 100%,  reduce = 98%, Cumulative CPU 2914.3 sec
2015-09-15 18:00:11,415 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 2947.08 sec
2015-09-15 18:01:12,037 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 2957.91 sec
2015-09-15 18:02:12,518 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 2957.91 sec

已有(2)人评论

跳转到指定楼层
jixianqiuxue 发表于 2015-9-15 18:38:17
这个可能数据倾斜了

hive数据倾斜原因分析及解决方案
http://www.aboutyun.com/thread-8296-1-1.html

回复

使用道具 举报

ananan36 发表于 2015-9-16 10:29:42
jixianqiuxue 发表于 2015-9-15 18:38
这个可能数据倾斜了

hive数据倾斜原因分析及解决方案

怎么看redue占内存
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条