图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › HDFS › hdfs内网集群的访问问题
分享

hdfs内网集群的访问问题

scorpionxie 发表于 2016-11-15 15:55:06 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 3 11392
About云VIP帮助找工作和提升
集群在一个内网环境
机器名内网ip 外网ip部署情况
bigmaster192.168.10.1010.1.1.10 namenode
bignode01192.168.10.11 10.1.1.11 datanode
bignode02192.168.10.12 10.1.1.12 datanode
bignode03192.168.10.13 10.1.1.13 datanode

java程序在10.1.1.55上，已修改10.1.1.55上的hosts，把五个机器名与集群节点的外网IP做了映射。

[mw_shl_code=applescript,true]10.1.1.10 bigmaster
10.1.1.11 bignode01
10.1.1.12 bignode02
10.1.1.13 bignode03[/mw_shl_code]

通过java进行文件上传到hdfs的时候，报连接超时。
分析日志看到以下代码：
[mw_shl_code=bash,true]hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.11:50010,DS-fb1f06fa-03f0-4384-ab19-249319fcda06,DISK]
hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[[192.168.10.12:50010,DS-a1f3f581-5969-43a3-a133-c038f062300b,DISK]
hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[[192.168.10.13:50010,DS-77af5965-1d15-4331-8b53-5099f68a61d7,DISK][/mw_shl_code]

应该是java连接到namenode后，得到的是各个datanode的内网ip，所以无法访问datanode。
求教应该如何解决？
回复

使用道具 举报

自定义广告语

已有(3)人评论

跳转到指定楼层
正序浏览
einhep 发表于 2016-11-15 16:12:56
这里面楼主描述的需更加清晰，下面内容
1.集群是如何配置的，用的是内网还是外网ip。
2.开发客户端与集群是什么关系。
3.链接超时，贴出来。
4.这个日志只是显示了一些信息，并非报错。
对于提供的信息，不能确定集群是如何配置的，比如你是统一用的外网配置的集群，还是统一用的内网配置的集群。

回复

使用道具 举报

scorpionxie 发表于 2016-11-15 16:24:45
einhep 发表于 2016-11-15 16:12
这里面楼主描述的需更加清晰，下面内容
1.集群是如何配置的，用的是内网还是外网ip。
2.开发客户端与集群 ...

1.集群是如何配置的，用的是内网还是外网ip。Q集群是用ambari部署的，是用的机器名。各个节点的hosts文件如下
[mw_shl_code=bash,true]192.168.10.10 bigmaster
192.168.10.11 bignode01
192.168.10.12 bignode02
192.168.10.13 bignode03[/mw_shl_code]

2.开发客户端与集群是什么关系。
Q客户端就是java程序，或者说是开发平台吧，与集群只能通过10.1.1.x进行访问。

3.链接超时，贴出来。

4.这个日志只是显示了一些信息，并非报错。
对于提供的信息，不能确定集群是如何配置的，比如你是统一用的外网配置的集群，还是统一用的内网配置的集群。



日志如下
[mw_shl_code=bash,true]2016-11-15 15:13:58,816 TRACE [Thread-4] ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: addBlock {block { b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100545 generationStamp: 359890 numBytes: 0 } offset: 0 locs { id { ipAddr: "192.168.10.13" hostName: "bignode02" datanodeUuid: "f7046005-f515-4b48-b059-88adddb5fd37" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574724321 remaining: 1111921903831 blockPoolUsed: 361574724321 lastUpdate: 1479194019243 xceiverCount: 9 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518160 } locs { id { ipAddr: "192.168.10.11" hostName: "bignode03" datanodeUuid: "f190501c-1309-42f2-89e7-e04b040bc23e" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361572066017 remaining: 1112808610007 blockPoolUsed: 361572066017 lastUpdate: 1479194019136 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518053 } locs { id { ipAddr: "192.168.10.12" hostName: "bignode04" datanodeUuid: "9099adc6-9f68-4a29-ba2a-bb154168c3e4" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574056673 remaining: 1112210184407 blockPoolUsed: 361574056673 lastUpdate: 1479194019207 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518124 } corrupt: false blockToken { identifier: "\212\001Xh\372\331\r\214\r\275)@\000\000\000\006hadoopuser\000\000\000)BP-1901003557-192.168.1.221-1454661240805\214@\005yA\001\005WRITE" password: "9\335\370D^\222\256\031r\031$\321\353-\370\027Np\356}" kind: "HDFS_BLOCK_TOKEN" service: "" } isCached: false isCached: false isCached: false storageTypes: DISK storageTypes: DISK storageTypes: DISK storageIDs: "DS-fb1f06fa-03f0-4384-ab19-249319fcda06" storageIDs: "DS-a1f3f581-5969-43a3-a133-c038f062300b" storageIDs: "DS-77af5965-1d15-4331-8b53-5099f68a61d7" }}
2016-11-15 15:13:58,819 DEBUG [Thread-4] hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.13:50010,DS-fb1f06fa-03f0-4384-ab19-249319fcda06,DISK]
2016-11-15 15:13:58,819 DEBUG [Thread-4] hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.11:50010,DS-a1f3f581-5969-43a3-a133-c038f062300b,DISK]
2016-11-15 15:13:58,819 DEBUG [Thread-4] hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.12:50010,DS-77af5965-1d15-4331-8b53-5099f68a61d7,DISK]
2016-11-15 15:13:58,819 DEBUG [Thread-4] hdfs.DFSClient: Connecting to datanode 192.168.10.13:50010
2016-11-15 15:14:19,817 INFO  [Thread-4] hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection timed out: no further information
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
        at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1605)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1346)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1299)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
2016-11-15 15:14:19,819 INFO  [Thread-4] hdfs.DFSClient: Abandoning BP-1901003557-192.168.1.221-1454661240805:blk_1074100545_359890
2016-11-15 15:14:19,822 TRACE [Thread-4] ipc.ProtobufRpcEngine: 14: Call -> bigmaster/10.1.1.10:8020: abandonBlock {b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100545 generationStamp: 359890 numBytes: 0 } src: "/apps/bocai/uploadfile/145-500000-0001-20161114170257716/curl.zip" holder: "DFSClient_NONMAPREDUCE_-486330154_1" fileId: 991761}
2016-11-15 15:14:19,823 DEBUG [IPC Parameter Sending Thread #0] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #7
2016-11-15 15:14:19,835 DEBUG [IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #7
2016-11-15 15:14:19,835 DEBUG [Thread-4] ipc.ProtobufRpcEngine: Call: abandonBlock took 14ms
2016-11-15 15:14:19,838 TRACE [Thread-4] ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: abandonBlock {}
2016-11-15 15:14:19,838 INFO  [Thread-4] hdfs.DFSClient: Excluding datanode DatanodeInfoWithStorage[192.168.10.13:50010,DS-fb1f06fa-03f0-4384-ab19-249319fcda06,DISK]
2016-11-15 15:14:19,844 TRACE [Thread-4] ipc.ProtobufRpcEngine: 14: Call -> bigmaster/10.1.1.10:8020: addBlock {src: "/apps/bocai/uploadfile/145-500000-0001-20161114170257716/curl.zip" clientName: "DFSClient_NONMAPREDUCE_-486330154_1" excludeNodes { id { ipAddr: "192.168.10.13" hostName: "bignode02" datanodeUuid: "f7046005-f515-4b48-b059-88adddb5fd37" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574724321 remaining: 1111921903831 blockPoolUsed: 361574724321 lastUpdate: 1479194019243 xceiverCount: 9 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10518160 } fileId: 991761}
2016-11-15 15:14:19,845 DEBUG [IPC Parameter Sending Thread #0] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #8
2016-11-15 15:14:19,853 DEBUG [IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #8
2016-11-15 15:14:19,853 DEBUG [Thread-4] ipc.ProtobufRpcEngine: Call: addBlock took 10ms
2016-11-15 15:14:19,855 TRACE [Thread-4] ipc.ProtobufRpcEngine: 14: Response <- bigmaster/10.1.1.10:8020: addBlock {block { b { poolId: "BP-1901003557-192.168.1.221-1454661240805" blockId: 1074100546 generationStamp: 359891 numBytes: 0 } offset: 0 locs { id { ipAddr: "192.168.10.12" hostName: "bignode04" datanodeUuid: "9099adc6-9f68-4a29-ba2a-bb154168c3e4" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361574056673 remaining: 1112210208983 blockPoolUsed: 361574056673 lastUpdate: 1479194040208 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10539126 } locs { id { ipAddr: "192.168.10.11" hostName: "bignode03" datanodeUuid: "f190501c-1309-42f2-89e7-e04b040bc23e" xferPort: 50010 infoPort: 50075 ipcPort: 8010 infoSecurePort: 0 } capacity: 1556750635008 dfsUsed: 361572066017 remaining: 1112808642775 blockPoolUsed: 361572066017 lastUpdate: 1479194040140 xceiverCount: 8 location: "/default-rack" adminState: NORMAL cacheCapacity: 0 cacheUsed: 0 lastUpdateMonotonic: 10539057 } corrupt: false blockToken { identifier: "\212\001Xh\373+B\214\r\275)@\000\000\000\006hadoopuser\000\000\000)BP-1901003557-192.168.1.221-1454661240805\214@\005yB\001\005WRITE" password: "\336\212\233\221\264\006\245\257\356\232U\266[\206\357A\242\352\313R" kind: "HDFS_BLOCK_TOKEN" service: "" } isCached: false isCached: false storageTypes: DISK storageTypes: DISK storageIDs: "DS-77af5965-1d15-4331-8b53-5099f68a61d7" storageIDs: "DS-a1f3f581-5969-43a3-a133-c038f062300b" }}
2016-11-15 15:14:19,855 DEBUG [Thread-4] hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.12:50010,DS-77af5965-1d15-4331-8b53-5099f68a61d7,DISK]
2016-11-15 15:14:19,855 DEBUG [Thread-4] hdfs.DFSClient: pipeline = DatanodeInfoWithStorage[192.168.10.11:50010,DS-a1f3f581-5969-43a3-a133-c038f062300b,DISK]
2016-11-15 15:14:19,855 DEBUG [Thread-4] hdfs.DFSClient: Connecting to datanode 192.168.10.12:50010
2016-11-15 15:14:28,757 TRACE [LeaseRenewer:hadoopuser@bigmaster, clients=[DFSClient_NONMAPREDUCE_-486330154_1], created at java.lang.Throwable: TRACE
        at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
        at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
        at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
        at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
        at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
        at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1707)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
        at com.zd.bocai.hdfs.HdfsUtils.copyFile(HdfsUtils.java:17)
        at com.zd.bocai.BocaiServer.main(BocaiServer.java:24)
] ipc.ProtobufRpcEngine: 15: Call -> bigmaster/10.1.1.10:8020: renewLease {clientName: "DFSClient_NONMAPREDUCE_-486330154_1"}
2016-11-15 15:14:28,758 DEBUG [IPC Parameter Sending Thread #0] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser sending #9
2016-11-15 15:14:28,760 DEBUG [IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser] ipc.Client: IPC Client (1327981586) connection to bigmaster/10.1.1.10:8020 from hadoopuser got value #9
2016-11-15 15:14:28,760 DEBUG [LeaseRenewer:hadoopuser@bigmaster, clients=[DFSClient_NONMAPREDUCE_-486330154_1], created at java.lang.Throwable: TRACE
        at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:202)
        at org.apache.hadoop.hdfs.LeaseRenewer.<init>(LeaseRenewer.java:71)
        at org.apache.hadoop.hdfs.LeaseRenewer$Factory.get(LeaseRenewer.java:143)
        at org.apache.hadoop.hdfs.LeaseRenewer$Factory.access$100(LeaseRenewer.java:90)
        at org.apache.hadoop.hdfs.LeaseRenewer.getInstance(LeaseRenewer.java:80)
        at org.apache.hadoop.hdfs.DFSClient.getLeaseRenewer(DFSClient.java:818)
        at org.apache.hadoop.hdfs.DFSClient.beginFileLease(DFSClient.java:824)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1707)
        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:356)
        at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
        at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
        at com.zd.bocai.hdfs.HdfsUtils.copyFile(HdfsUtils.java:17)
        at com.zd.bocai.BocaiServer.main(BocaiServer.java:24)
] ipc.ProtobufRpcEngine: Call: renewLease took 3ms[/mw_shl_code]

回复

使用道具 举报

qcbb001 发表于 2016-11-15 20:00:53
scorpionxie 发表于 2016-11-15 16:24
1.集群是如何配置的，用的是内网还是外网ip。Q集群是用ambari部署的，是用的机器名。各个节点的hosts文件 ...

用10段去链接192段肯定不行的。即使bigmaster是双ip也不行啊。两个ip不能通信。
bigmaster/10.1.1.10:8020

需要将下面修改为10段的
192.168.10.10 bigmaster
192.168.10.11 bignode01
192.168.10.12 bignode02
192.168.10.13 bignode03


或则你的开发机器是与192段的同属一个网段才可以
回复

使用道具 举报

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
scorpionxie

新手上路

关注

  • 主题

  • 帖子

  • 粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000