datanode无法正常启动

我的CDH集群，由一个datanode无法正常启动，出现报错如下
Info FsDatasetImpl Adding replicas to map for block pool BP-1180296529- 1491016364627 on volume /data_e/dfs/dn/current...
Info FsDatasetImpl Caught exception while adding replicas from /data_c/dfs/dn/current. Will throw later.
java.io.IOException: block pool BP- -1491016364627 is not found
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getBlockPoolSlice(FsVolumeImpl.java:408)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.getVolumeMap(FsVolumeImpl.java:865)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeList$1.run(FsVolumeList.java:191)

进行如下：排查和操作
查看current下的VERSION 以及 BP-1180296529- 1491016364627下的VERSION文件，内容没有问题
删除current目录 ，重启，问题存在
修改目录的权限为777 ，重启问题依旧存在
hdfs进行重新平衡，datanode重启问题依旧

谁知道这个问题怎么处理么？ 或者有什么思路方向么？

附：hdfs dfsadmin -report
提示：
Configured Capacity: 135335249657856 (123.09 TB)
Present Capacity: 126442052813507 (115.00 TB)
DFS Remaining: 60736844544007 (55.24 TB)
DFS Used: 65705208269500 (59.76 TB)
DFS Used%: 51.96%
Under replicated blocks: 12774
Blocks with corrupt replicas: 0
Missing blocks: 7
Missing blocks (with replication factor 1): 15
但是整体显示健康

你删除就不对了，因为删掉所以报错找不到数据了。这样就可以解释了。另外无法正常启动，应该是另外的错误，而不是这个错误造成的。在找找其他日志。

附新的信息：
除了上面的报错，日志下还有
2019-01-22 22:05:32,222 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to analyze storage directories for block p
ool BP-1180296529- 1491016364627
java.io.IOException: BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block storage: /data_c/dfs/dn/curr
ent/BP-1180296529-
        at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:
217)
        at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250
)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:470)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:399)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1561)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1521)
2019-01-22 22:05:32,504 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage directory [DISK]file:/data_
m/dfs/dn/ for block pool BP-1180296529-1491016364627
java.io.IOException: BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used block storage: /data_m/dfs/dn/curr
ent/BP-1180296529-1491016364627
        at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.loadBpStorageDirectories(BlockPoolSliceStorage.java:
217)
        at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceStorage.recoverTransitionRead(BlockPoolSliceStorage.java:250
)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.loadBlockPoolSliceStorage(DataStorage.java:470)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.addStorageLocations(DataStorage.java:399)
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:575)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1561)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1521)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
        at java.lang.Thread.run(Thread.java:745)
2019-01-22 22:05:32,505 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for Block pool <registe
ring> (Datanode Uuid f945f506-3856-4c05-b6af-e0923ed5bbed) service to Exit
ing.
java.io.IOException: All specified directories are failed to load.
        at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:576)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1561)
        at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1521)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:313)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
        at java.lang.Thread.run(Thread.java:745)
2019-01-22 22:05:32,843 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool <reg
istering> (Datanode Uuid f945f506-3856-4c05-b6af-e0923ed5bbed) service to
2019-01-22 22:05:32,945 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool <registering> (Datanode Uui
d f945f506-3856-4c05-b6af-e0923ed5bbed)
2019-01-22 22:05:34,946 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2019-01-22 22:05:34,952 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 0
2019-01-22 22:05:34,955 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:

jsvc.err下报错
Opened info server at  
Starting regular datanode initialization
java.lang.NumberFormatException: For input string: "-"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:581)
        at java.lang.Long.parseLong(Long.java:631)
        at org.apache.hadoop.fs.DU$DUShell.parseExecResult(DU.java:84)
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:587)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.fs.DU$DUShell.startRefresh(DU.java:61)
        at org.apache.hadoop.fs.DU.refresh(DU.java:53)
        at org.apache.hadoop.fs.CachingGetSpaceUsed.init(CachingGetSpaceUsed.java:84)
        at org.apache.hadoop.fs.GetSpaceUsed$Builder.build(GetSpaceUsed.java:166)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.<init>(BlockPoolSlice.java:156)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.addBlockPool(FsVolumeImpl.java:892)
目前不清楚这个是什么原因造成的

仙本是凡 发表于 2019-1-23 10:53
jsvc.err下报错
Opened info server at
Starting re ...

要找到对应的启动日志，这些应该都是数据缺失造成的。
楼主对于日志应该细细分析。
如果不会看日志，建议把这个节点重新初始化，然后在添加。
不妥的地方，在于为什么把数据删掉。这样是非常不好的。