又是一个难解的storm问题，拜谢各位看看

我发布到storm上的topology运行良好，但奇怪的时候，我第二天过来，总是发现整个stormshutdown

我查看了nimbus.log,supervisor.log日志：
nimbus：
2015-04-22T19:12:58.967+0800 b.s.d.nimbus [INFO] Cleaning inbox ... deleted: stormjar-59f060a6-0ddd-460b-b599-c015a55617d5.jar
2015-04-22T19:24:11.066+0800 b.s.d.nimbus [INFO] Shutting down master
2015-04-22T19:24:11.078+0800 o.a.s.z.ClientCnxn [INFO] EventThread shut down
2015-04-22T19:24:11.078+0800 o.a.s.z.ZooKeeper [INFO] Session: 0x14ce054623a0020 closed
2015-04-22T19:24:11.084+0800 b.s.d.nimbus [INFO] Shut down master

supervisor.log
2015-04-22T18:46:13.640+0800 b.s.d.supervisor [INFO] 2d3a5712-5083-428c-9b7c-3738cb50e08c still hasn't started
2015-04-22T18:46:14.146+0800 b.s.d.supervisor [INFO] 9e736ea7-4504-4fb5-9555-7a1c4e4007a8 still hasn't started
2015-04-22T18:46:14.647+0800 b.s.d.supervisor [INFO] 9e736ea7-4504-4fb5-9555-7a1c4e4007a8 still hasn't started
2015-04-22T19:24:11.068+0800 b.s.d.supervisor [INFO] Shutting down supervisor 779763af-312f-4ba3-a7f8-13b3484c08a3
2015-04-22T19:24:11.094+0800 b.s.event [INFO] Event manager interrupted
2015-04-22T19:24:11.094+0800 b.s.event [INFO] Event manager interrupted
2015-04-22T19:24:11.102+0800 o.a.s.z.ZooKeeper [INFO] Session: 0x4ce05462480020 closed
2015-04-22T19:24:11.102+0800 o.a.s.z.ClientCnxn [INFO] EventThread shut down

请问有哪位大侠能解答下吗
~
~

caiyifeng · 发表于 2015-4-23 10:05:11

我观察了同一时间点的zookeeper日志，报如下错误，不知是否有关系：
2015-04-22 19:24:11,097 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.16:38668 which had sessionid 0x4ce05462480021
2015-04-22 19:24:11,100 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.15:45085 which had sessionid 0x4ce05462480020
2015-04-22 19:24:11,175 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.17:38168 which had sessionid 0x4ce0546248002c
2015-04-22 19:24:11,179 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.17:38166 which had sessionid 0x4ce05462480029
2015-04-22 19:24:11,185 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.16:38778 which had sessionid 0x4ce0546248002f
2015-04-22 19:24:11,195 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.16:38775 which had sessionid 0x4ce0546248002e
2015-04-22 19:24:12,393 [myid:0] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x4ce05462480030, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
      at java.lang.Thread.run(Thread.java:745)
2015-04-22 19:24:12,395 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.15:45698 which had sessionid 0x4ce05462480030
2015-04-22 19:24:12,407 [myid:0] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x4ce0546248002d, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
      at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
      at java.lang.Thread.run(Thread.java:745)
2015-04-22 19:24:12,408 [myid:0] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1007] - Closed socket connection for client /10.50.5.16:38769 which had sessionid 0x4ce0546248002d
2015-04-22 19:24:12,408 [myid:0] - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@357] - caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x4ce05462480032, likely client has closed socket
      at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)

arsenduan · 发表于 2015-4-23 10:39:39

caiyifeng 发表于 2015-4-23 10:05
我观察了同一时间点的zookeeper日志，报如下错误，不知是否有关系：
2015-04-22 19:24:11,097 [myid:0] - ...

上面可能是连接的问题
尝试下面解决办法：

1.删除storm/lib下的log4j-over-slf4j

2.Storm中关于ZMQ和ZooKeeper连接错误的问题，
一般都是本机的host配置有问题导致无法连接。需要在Storm集群中的所有节点，进行如下修改：
1)添加本机IP和主机名的信息，如192.168.0.2 node1
2)添加Strom Cluster中其他主机的信息，
如
192.168.0.3 node2
192.168.0.4 node3
从而使ZMQ或Zookeeper在连接时能解析到正确的主机。

caiyifeng · 发表于 2015-4-23 11:21:48

caiyifeng · 发表于 2015-4-24 09:47:31

arsenduan 发表于 2015-4-23 10:39
上面可能是连接的问题
尝试下面解决办法：

还是不行啊，我忘了说一件事了：我的3个节点都是虚拟机，会不会有关系？？？

rsgg03 · 发表于 2015-4-24 15:08:49

caiyifeng 发表于 2015-4-24 09:47
还是不行啊，我忘了说一件事了：我的3个节点都是虚拟机，会不会有关系？？？

应该没有关系

Unable to read additional data from client sessionid 0x4ce0546248002d, likely client has closed socket

根据错误来看是不能读zookeeper客户端的数据，因为客户端不能通信了。
可能zookeeper的问题