本人CM5.7+CDH5.5.3在运行过程中zookeeper的连接数一直持续性增加,直到突破zookeeper最大连接限制导致zookeeper异常,hbase无法建立新的连接。经查看zookeeper上的连接数,全部是从SERVICE_MONITORING进程连接过来的。查看日志里面全部都是连接zookeeper获取hbase连接
2016-05-18 16:01:29,887 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2ef85713 connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
2016-05-18 16:01:29,918 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x3b57f651 connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
2016-05-18 16:02:29,888 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x24be87ba connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
2016-05-18 16:02:29,922 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x2623f77a connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
2016-05-18 16:03:29,891 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x5564004d connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
2016-05-18 16:03:29,922 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x1a13fd2d connecting to ZooKeeper ensemble=datanode03:2181,datanode01:2181,datanode02:2181
却一直没见到关闭连接,并且连接建立后一直在发送包,也不会被zookeeper超时关闭。
问题找了好几天了,一直没有头绪,还希望有大神能指点一下。谢谢!
|