cloudera monitor1个诡异的问题: master主机2分钟挂掉1次

问题具体描述: cloudera monitor1个诡异的问题: master主机2分钟挂掉1次(包括DataNode/hive等全报问题)。
版本: cdh 5.12
服务: hdfs/yarn/hive等
结构: 1台master 3个slave
         master上含有cm的所有角色(包括monitor)，并且含有DataNode等
         slave上就是单纯的DataNode等slave角色。

主要现象:
     如图001:

具体的报错:
[mw_shl_code=bash,true]    代理状态   该主机已与 Cloudera Manager Server 建立联系。 该主机未与 Host Monitor 建立联系。
    The health test result for MGMT_SERVICE_MONITOR_HEALTH has become bad: The health of the Service Monitor is bad. The following health tests are bad: host health.
    The health test result for MGMT_HOST_MONITOR_HEALTH has become bad: The health of the Host Monitor is bad. The following health tests are bad: host health.
    The health test result for MGMT_EVENT_SERVER_HEALTH has become bad: The health of the Event Server is bad. The following health tests are bad: host health.
    The health test result for MGMT_ALERT_PUBLISHER_HEALTH has become bad: The health of the Alert Publisher is bad. The following health tests are bad: host health.
    The health test result for HOST_NETWORK_INTERFACES_SLOW_MODE has become unknown: Not enough data to test: Test of whether the host has network interfaces that appear to be operating at less than full speed. This health test is currently suppressed.[/mw_shl_code]

之前好好的，不知怎么就这样了。
ntp/selinux/hosts/iptables/等等能检查的都检查过了。
虽然提示这么多，但是服务都是没问题的。网络测试也是通的。
给人的感觉  跟monitor 有关似的。
大家谁有建议，提一下呗

感觉有点像  跟  monitor这个进程有关，但是不知道架构，不知道咋具体排查。。。现在还在找

检查  mgmt-cmf-mgmt-HOSTMONITOR-hostname.log.out 日志也没东西···

chendoubao 发表于 2018-7-19 10:21
检查  mgmt-cmf-mgmt-HOSTMONITOR-hostname.log.out 日志也没东西···

The health test result for HOST_NETWORK_INTERFACES_SLOW_MODE has become unknown: Not enough data to test: Test of whether the host has network interfaces that appear to be operating at less than full speed. This health test is currently suppressed.
应该没什么关系，可能就是数据不够的问题
由于数据不足，所以cloudera的健康测试可能被禁止或则抑制

s060403072 发表于 2018-7-19 12:31
The health test result for HOST_NETWORK_INTERFACES_SLOW_MODE has become unknown: Not enough data t ...

如果单看这点像是这样，但是网络部分都没有啥问题。并且服务也是ok的。(其实隐含的这个地方的测试方式不知道是如何的，看文档有的地方提是ethtool，但是没有具体的测试方式)

很奇葩的现象。
然后由于中午hive突然变慢了好多，于是就把hive重启了下。
然后···集群就突然正常了·····正常了·····

chendoubao 发表于 2018-7-19 15:29
很奇葩的现象。
然后由于中午hive突然变慢了好多，于是就把hive重启了下。
然后···集群就突然正常了· ...

厉害了，老铁

s060403072 发表于 2018-7-19 15:36
厉害了，老铁

严格意义，从监控图上来讲，是有一些变化的





例如: cpu在之前的systime 和iotime 都是有1点小高的(但是也不至于导致这样吧)。
      mem变化是因为hive重启了1次。
网络没啥变化，我就没贴图。So。。。奇怪呦

记录一下，以后别人遇到重启服务吧,都重启重启