图文精华

About云-梭伦科技»专题 › 技术学习(版主发帖区) › 大数据学习 › Hbase › 总结型› Master只能控制其中1台RegionServer，无法控制其它Regio ...
分享

Master只能控制其中1台RegionServer，无法控制其它RegionServer原因分析

pig2 2014-5-9 14:01:38 发表于 总结型 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 0 7335
About云VIP帮助找工作和提升
问题导读：
集群中hosts配置错误，可能会遇到什么问题





最近在测试HBase时遇到一个非常奇怪的问题：集群有7台机器，其中1台Master，6台RegionServer。但是Master只能控制其中1台RegionServer，而无法控制其他5台RegionServer。

打开master的日志文件，发现以下错误信息：
  1. 2011-04-22 16:37:21,242 WARN org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of -ROOT-,,0.70236052 to serverName=hp2,60020,1303461559353, load=(requests=0, regions=0, usedHeap=28, maxHeap=3979), trying to assign elsewhere instead; retry=0 

  2. org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed setting up proxy interface org.apache.hadoop.hbase.ipc.HRegionInterface to /10.131.18.3:60020 after attempts=1 

  3.     at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:355) 

  4.     at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:965) 

  5.     at org.apache.hadoop.hbase.master.ServerManager.getServerConnection(ServerManager.java:606) 

  6.     at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:541) 

  7.     at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:920) 

  8.     at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:730) 

  9.     at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:710) 

  10.     at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1189) 

  11.     at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:432) 

  12.     at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:389) 

  13.     at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:283) 

  14. Caused by: java.net.ConnectException: Connection refused 

  15.     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) 

  16.     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) 

  17.     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) 

  18.     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408) 

  19.     at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328) 

  20.     at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883) 

  21.     at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750) 

  22.     at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257) 

  23.     at $Proxy7.getProtocolVersion(Unknown Source) 

  24.     at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419) 

  25.     at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393) 

  26.     at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444) 

  27.     at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349) 

  28.     ... 10 more
复制代码


在这个日志中，master机器无法与IP地址为10.131.18.3的regionserver进行通信。

然后找到10.131.18.3机器，查看这台机器的regionserver日志，查看regionserver的启动信息：
  1. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020: starting 

  2. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 60020: starting 

  3. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60020: starting 

  4. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 60020: starting 

  5. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60020: starting 

  6. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 15 on 60020: starting 

  7. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60020: starting 

  8. 2011-04-14 18:32:05,122 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60020: starting 

  9. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020: starting 

  10. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60020: starting 

  11. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 0 on 60020: starting2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 1 on 60020: starting 

  12. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 2 on 60020: starting2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 3 on 60020: starting 

  13. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 4 on 60020: starting2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 5 on 60020: starting 

  14. 2011-04-14 18:32:05,123 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 6 on 60020: starting2011-04-14 18:32:05,124 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 7 on 60020: starting 

  15. 2011-04-14 18:32:05,124 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 8 on 60020: starting2011-04-14 18:32:05,124 INFO org.apache.hadoop.ipc.HBaseServer: PRI IPC Server handler 9 on 60020: starting2011-04-14 18:32:05,124 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Serving as dell4,60020,1302777124101, RPC listening on /127.0.0.1:60020, sessionid=0x12f535856620004
复制代码

可以看出，这台regionserver机器启动成功了，但是RPC的监听ip地址却是本机的地址（127.0.0.1）。这样的话，master机器就无法与这台regionserver正常通信了，正确的监听地址应该是10.131.18.3才对。

查看代码，RPC监听地址的代码如下：
  1. /** @return Bind address */ 

  2.   public String getBindAddress() { 

  3.     final InetAddress addr = address.getAddress(); 

  4.     if (addr != null) { 

  5.       return addr.getHostAddress(); 

  6.     } else { 

  7.       LogFactory. getLog(HServerAddress.class).error( "Could not resolve the" 

  8.           + " DNS name of " + stringValue ); 

  9.       return null; 

  10.     } 

  11.   }
复制代码
代码没有错，看来是机器的某些配置导致java读取本机的ip地址出现了错误。最后查看这台机器的hosts文件：
[hadoop@hp2 logs]$ vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               hp2 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
10.131.18.8     dell1
10.131.18.5     dell2
10.131.18.6     dell3
10.131.18.7     dell4
10.131.18.2     hp1
10.131.18.3     hp2
10.131.18.4     hp3
问题找到了，其实是hosts文件的配置原因，接下来修改hosts文件为如下：
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
10.131.18.8     dell1
10.131.18.5     dell2
10.131.18.6     dell3
10.131.18.7     dell4
10.131.18.2     hp1
10.131.18.3     hp2
10.131.18.4     hp3
再次启动整个集群，问题解决。





关注公众号，获取大数据、人工智能20套、区块链资源5阶段等资源，随时更新，获取最新技术资源
回复

使用道具 举报

自定义广告语

没找到任何评论，期待你打破沉寂

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
pig2

超级版主

关注

  • 2329

    主题

  • 3339

    帖子

  • 1972

    粉丝

欢迎大家如about云官方2群322273151，更新咨询，更新资源，随时关注
TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000