图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › Hadoop|YARN › hadoop2的jobhistory没有显示job问题
分享

hadoop2的jobhistory没有显示job问题

jttsai 发表于 2014-8-21 17:06:38 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 4 86610
About云VIP帮助找工作和提升
运行完mapreduce后，在JobHistory提供的页面上找不到任何任务，这是什么情况。
下面是我配置部分：
<property>
  <name>mapreduce.jobhistory.address</name>
  <value>master1.hadoop:10020</value>
</property>
<property>
  <name>mapreduce.jobhistory.webapp.address</name>
  <value>master1.hadoop:19888</value>
</property>
<property>
  <name>mapreduce.jobhistory.intermediate-done-dir</name>
  <value>${hadoop.tmp.dir}/mr/history-tmp</value>
</property>
<property>
  <name>mapreduce.jobhistory.done-dir</name>
  <value>${hadoop.tmp.dir}/mr/history-done</value>
</property>

下面是jobhistory的日志信息：
2014-08-21 16:47:09,940 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer: registered UNIX signal handlers for [TERM, HUP, INT]
2014-08-21 16:47:11,082 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: JobHistory Init
2014-08-21 16:47:21,738 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Initializing Existing Jobs...
2014-08-21 16:47:21,756 INFO org.apache.hadoop.mapreduce.v2.hs.CachedHistoryStorage: CachedHistoryStorage Init
2014-08-21 16:47:21,800 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2014-08-21 16:47:21,817 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 10033
2014-08-21 16:47:21,897 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2014-08-21 16:47:21,975 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2014-08-21 16:47:21,975 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobHistoryServer metrics system started
2014-08-21 16:47:21,984 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2014-08-21 16:47:21,988 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2014-08-21 16:47:21,988 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2014-08-21 16:47:22,079 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2014-08-21 16:47:22,084 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.jobhistory is not defined
2014-08-21 16:47:22,100 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2014-08-21 16:47:22,104 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context jobhistory
2014-08-21 16:47:22,104 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2014-08-21 16:47:22,104 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2014-08-21 16:47:22,108 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /jobhistory/*
2014-08-21 16:47:22,109 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2014-08-21 16:47:22,124 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 19888
2014-08-21 16:47:22,124 INFO org.mortbay.log: jetty-6.1.26
2014-08-21 16:47:22,156 INFO org.mortbay.log: Extract jar:file:/home/hdfs/bch/share/hadoop/yarn/hadoop-yarn-common-2.3.0-cdh5.0.0.jar!/webapps/jobhistory to /tmp/Jetty_master1_hadoop_19888_jobhistory____qptoa8/webapp
2014-08-21 16:47:22,476 INFO org.mortbay.log: Started SelectChannelConnector@master1.hadoop:19888
2014-08-21 16:47:22,478 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888
2014-08-21 16:47:23,018 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2014-08-21 16:47:23,036 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2014-08-21 16:47:23,044 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 10020
2014-08-21 16:47:23,055 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.HSClientProtocolPB to the server
2014-08-21 16:47:23,113 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-08-21 16:47:23,113 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 10020: starting
2014-08-21 16:47:23,121 INFO org.apache.hadoop.mapreduce.v2.hs.HistoryClientService: Instantiated MRClientService at master1.hadoop/20.26.19.16:10020
2014-08-21 16:47:23,124 INFO org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService: aggregated log deletion started.
2014-08-21 16:47:23,124 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-08-21 16:47:23,124 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 10033: starting
2014-08-21 16:47:23,201 INFO org.apache.hadoop.yarn.logaggregation.AggregatedLogDeletionService: aggregated log deletion finished.
2014-08-21 16:47:23,626 INFO logs: Aliases are enabled
2014-08-21 16:47:51,990 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner started
2014-08-21 16:47:51,998 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: History Cleaner complete
2014-08-21 16:50:21,989 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-08-21 16:53:20,872 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:53:21,988 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-08-21 16:53:29,237 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:54:10,688 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:54:12,525 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:54:13,198 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:54:13,604 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:54:13,929 INFO org.apache.hadoop.yarn.webapp.View: Getting list of all Jobs.
2014-08-21 16:56:21,988 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-08-21 16:59:21,988 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-08-21 17:02:21,988 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
2014-08-21 17:05:21,988 INFO org.apache.hadoop.mapreduce.v2.hs.JobHistory: Starting scan to move intermediate done files
回复

使用道具 举报

自定义广告语

已有(4)人评论

跳转到指定楼层
正序浏览
gwgyk 发表于 2015-7-16 14:53:19
howtodown 发表于 2014-8-21 22:33
本帖最后由 howtodown 于 2014-8-21 22:34 编辑
首先保证已经启动 historyserver

hadoop2能查看到map阶段和reduce阶段的执行时间吗？不是平均执行时间

我启动了history server，但是只能看到平均的map和reduce时间，并不能看到map和reduce各自执行了多长时间
这个可以看到吗？
回复

使用道具 举报

howtodown 发表于 2014-8-21 22:33:40
本帖最后由 howtodown 于 2014-8-21 22:34 编辑
首先保证已经启动 historyserver
  1. $ sbin/mr-jobhistory-daemon.sh   start historyserver
复制代码




然后看看你在你所配置的目录中，查看是否有hdfs文件，如果没有说明你的配置有问题。

最好按照下面参数来配置

  1. <property>

  2.     <name>mapreduce.jobhistory.done-dir</name>

  3.     <value>${yarn.app.mapreduce.am.staging-dir}/history/done</value>

  4. </property>

  5. 

  6. <property>

  7.     <name>mapreduce.jobhistory.intermediate-done-dir</name>

  8.     <value>${yarn.app.mapreduce.am.staging-dir}

  9.                          /history/done_intermediate</value>

  10. </property>

  11. 

  12. <property>

  13.     <name>yarn.app.mapreduce.am.staging-dir</name>

  14.     <value>/tmp/hadoop-yarn/staging</value>

  15. </property>
复制代码

详细参考：
Hadoop jobhistory历史服务器介绍





回复

使用道具 举报

jttsai 发表于 2014-8-22 09:25:55
howtodown 发表于 2014-8-21 22:33
本帖最后由 howtodown 于 2014-8-21 22:34 编辑
首先保证已经启动 historyserver

谢谢您的指导！可以找到了。。。这是我另外一个oozie的一个问题，您还知道么？

http://www.aboutyun.com/forum.ph ... amp;page=1#pid40743

回复

使用道具 举报

斯巴达之狐 发表于 2016-8-24 18:47:11
jttsai 发表于 2014-8-22 09:25
谢谢您的指导！可以找到了。。。这是我另外一个oozie的一个问题，您还知道么？

http://www.aboutyun.c ...

难道就是配置的问题吗？？按楼上说的配置就可以解决了？？
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
jttsai

中级会员

关注

  • 21

    主题

  • 67

    帖子

  • 4

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000