图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › 求助：hadnoop完全分布式集群安装问题，数据产生不了
分享

求助：hadnoop完全分布式集群安装问题，数据产生不了

ran_newstar 发表于 2014-8-22 00:08:58 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 60 61271
About云VIP帮助找工作和提升
从百度搜索结果来看是hdfs-site.xml里dfs.namenode.rpc-address的端口号不对，我设置了端口号9001是按照《hadoop-2.4.0分布式安装手册》要求，设置的地址是master的地址。
hdfs-site.xml的配置：


<property>
  <name>dfs.datanode.data.dir</name>
  <value>/home/hadoop/data/hadoop-2.4.0</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>

core-site.xml里没有dfs.datanode.data.dir这个属性，安装文档是有的，我把这个属性人为加上了。
<property>
  <name>dfs.datanode.data.dir</name>
  <value>/home/hadoop/data/hadoop-2.4.0</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>



上面配置的目录真实存在的：
[hadoop@master hadoop-2.4.0]$ pwd
/home/hadoop/data/hadoop-2.4.0


运行hadoop namenode -format 命令出现的现象：
14/08/21 08:57:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 08:57:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 08:57:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 08:57:02 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 08:57:02 WARN common.Util: Path /home/hadoop/dfs/hadoop-2.4.0/name should be specified as a URI in configuration files. Please update hdfs configuration.
14/08/21 08:57:02 WARN common.Util: Path /home/hadoop/dfs/hadoop-2.4.0/name should be specified as a URI in configuration files. Please update hdfs configuration.



回复

使用道具 举报

自定义广告语

已有(60)人评论

跳转到指定楼层
正序浏览
pig2 发表于 2014-8-22 06:25:59
core-site.xml里没有dfs.datanode.data.dir这个属性，就不要加上。没有配置文件都有自己的功能。加上反而是不正确的


按照标准来配置文件：
core-site.xml如下配置：
  1. <configuration>

  2.        <property>

  3.                 <name>fs.defaultFS</name>

  4.                 <value>hdfs://master:8020</value>

  5.        </property>

  6.        <property>

  7.                 <name>io.file.buffer.size</name>

  8.                 <value>131072</value>

  9.         </property>

  10.        <property>

  11.                <name>hadoop.tmp.dir</name>

  12.                <value>file:/home/aboutyun/tmp</value>

  13.                <description>Abase for other temporary   directories.</description>

  14.        </property>

  15.         <property>

  16.                <name>hadoop.proxyuser.aboutyun.hosts</name>

  17.                <value>*</value>

  18.        </property>

  19.        <property>

  20.                <name>hadoop.proxyuser.aboutyun.groups</name>

  21.                <value>*</value>

  22.        </property>

  23. </configuration>

复制代码



hdfs-site.xml如下配置
  1. <configuration>

  2.        <property>

  3.                 <name>dfs.namenode.secondary.http-address</name>

  4.                <value>master:9001</value>

  5.        </property>

  6.      <property>

  7.              <name>dfs.namenode.name.dir</name>

  8.              <value>file:/home/aboutyun/dfs/name</value>

  9.        </property>

  10.       <property>

  11.               <name>dfs.datanode.data.dir</name>

  12.               <value>file:/home/aboutyun/dfs/data</value>

  13.        </property>

  14.        <property>

  15.                <name>dfs.replication</name>

  16.                <value>3</value>

  17.         </property>

  18.         <property>

  19.                  <name>dfs.webhdfs.enabled</name>

  20.                   <value>true</value>

  21.          </property>

  22. </configuration>

复制代码







回复

使用道具 举报

ran_newstar 发表于 2014-8-22 10:18:44
本帖最后由 ran_newstar 于 2014-8-22 10:23 编辑
pig2 发表于 2014-8-22 06:25
core-site.xml里没有dfs.datanode.data.dir这个属性，就不要加上。没有配置文件都有自己的功能。加上反而是 ...

core-site.xml里没有下面属性，没加：
  •         <property>
  •                <name>hadoop.proxyuser.aboutyun.hosts</name>
  •                <value>*</value>
  •        </property>
  •        <property>
  •                <name>hadoop.proxyuser.aboutyun.groups</name>
  •                <value>*</value>
  •        </property>
结果出来一样，没解决。

其实昨天是tmp目录和name目录都有文件产生，就是datanode.data目录没有文件产生。

另外，file:/这个格式正确？而不是file:///   ？
回复

使用道具 举报

pig2 发表于 2014-8-22 10:33:22

这个格式是正确的，经过验证的。
看看自己的文件夹权限，肯定自己在某个地方忽略了。
详细参考：

hadoop2.2完全分布式最新高可靠安装文档
http://www.aboutyun.com/thread-7684-1-1.html
回复

使用道具 举报

ran_newstar 发表于 2014-8-22 10:40:05
pig2 发表于 2014-8-22 10:33
这个格式是正确的，经过验证的。
看看自己的文件夹权限，肯定自己在某个地方忽略了。
详细参考：

重新格式化，现在三个目录都不会产生文件了。
[hadoop@master ~]$ ls -l
total 48
drwxrwxr-x. 3 hadoop hadoop 4096 Aug 21 02:56 data
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 21:18 Desktop
drwxrwxr-x. 3 hadoop hadoop 4096 Aug 21 02:55 dfs
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Documents
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 10:31 Downloads
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Music
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Pictures
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Public
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Templates
drwxrwxr-x. 3 hadoop hadoop 4096 Aug 21 04:27 tmp
drwxr-xr-x. 2 hadoop hadoop 4096 Aug 20 09:01 Videos
drwxrwxr-x. 4 hadoop hadoop 4096 Aug 20 11:49 workspace


回复

使用道具 举报

sstutu 发表于 2014-8-22 10:43:02
ran_newstar 发表于 2014-8-22 10:40
重新格式化，现在三个目录都不会产生文件了。
[hadoop@master ~]$ ls -l
total 48

跟产生数据没有关系，你看看上传文件到hadoop，是否能查看到。
回复

使用道具 举报

ran_newstar 发表于 2014-8-22 10:47:06
STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common -r 1583262; compiled by 'jenkins' on 2014-03-31T08:29Z
STARTUP_MSG:   java = 1.7.0_67
************************************************************/
14/08/21 19:45:52 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
14/08/21 19:45:52 INFO namenode.NameNode: createNameNode [-format, master]
14/08/21 19:45:53 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 19:45:53 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 19:45:53 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 19:45:53 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
Formatting using clusterid: CID-7a10952a-39f6-430b-9a05-daabd622fa7a
14/08/21 19:45:53 INFO namenode.FSNamesystem: fsLock is fair:true
14/08/21 19:45:53 INFO namenode.HostFileManager: read includes:
HostSet(
)
14/08/21 19:45:53 INFO namenode.HostFileManager: read excludes:
HostSet(
)
14/08/21 19:45:53 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
14/08/21 19:45:53 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
14/08/21 19:45:53 INFO util.GSet: Computing capacity for map BlocksMap
14/08/21 19:45:53 INFO util.GSet: VM type       = 32-bit
14/08/21 19:45:53 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
14/08/21 19:45:53 INFO util.GSet: capacity      = 2^22 = 4194304 entries
14/08/21 19:45:53 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
14/08/21 19:45:53 INFO blockmanagement.BlockManager: defaultReplication         = 2
14/08/21 19:45:53 INFO blockmanagement.BlockManager: maxReplication             = 512
14/08/21 19:45:53 INFO blockmanagement.BlockManager: minReplication             = 1
14/08/21 19:45:53 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
14/08/21 19:45:53 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
14/08/21 19:45:53 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
14/08/21 19:45:53 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
14/08/21 19:45:53 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
14/08/21 19:45:53 INFO namenode.FSNamesystem: fsOwner             = hadoop (auth:SIMPLE)
14/08/21 19:45:53 INFO namenode.FSNamesystem: supergroup          = supergroup
14/08/21 19:45:53 INFO namenode.FSNamesystem: isPermissionEnabled = true
14/08/21 19:45:53 INFO namenode.FSNamesystem: HA Enabled: false
14/08/21 19:45:53 INFO namenode.FSNamesystem: Append Enabled: true
14/08/21 19:45:54 INFO util.GSet: Computing capacity for map INodeMap
14/08/21 19:45:54 INFO util.GSet: VM type       = 32-bit
14/08/21 19:45:54 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
14/08/21 19:45:54 INFO util.GSet: capacity      = 2^21 = 2097152 entries
14/08/21 19:45:54 INFO namenode.NameNode: Caching file names occuring more than 10 times
14/08/21 19:45:54 INFO util.GSet: Computing capacity for map cachedBlocks
14/08/21 19:45:54 INFO util.GSet: VM type       = 32-bit
14/08/21 19:45:54 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
14/08/21 19:45:54 INFO util.GSet: capacity      = 2^19 = 524288 entries
14/08/21 19:45:54 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
14/08/21 19:45:54 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
14/08/21 19:45:54 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
14/08/21 19:45:54 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
14/08/21 19:45:54 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
14/08/21 19:45:54 INFO util.GSet: Computing capacity for map NameNodeRetryCache
14/08/21 19:45:54 INFO util.GSet: VM type       = 32-bit
14/08/21 19:45:54 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
14/08/21 19:45:54 INFO util.GSet: capacity      = 2^16 = 65536 entries
14/08/21 19:45:54 INFO namenode.AclConfigFlag: ACLs enabled? false
Re-format filesystem in Storage Directory /home/hadoop/dfs/hadoop-2.4.0/name ? (Y or N) Y
14/08/21 19:45:56 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 19:45:56 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 19:45:56 INFO namenode.FSImage: Allocated new BlockPoolId: BP-698057704-192.168.1.121-1408675556549
14/08/21 19:45:56 INFO common.Storage: Storage directory /home/hadoop/dfs/hadoop-2.4.0/name has been successfully formatted.
14/08/21 19:45:57 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
14/08/21 19:45:57 INFO util.ExitUtil: Exiting with status 0
14/08/21 19:45:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master/192.168.1.121
************************************************************/
[hadoop@master sbin]$
回复

使用道具 举报

howtodown 发表于 2014-8-22 10:54:36
ran_newstar 发表于 2014-8-22 10:47
STARTUP_MSG:   build = http://svn.apache.org/repos/asf/hadoop/common -r 1583262; compiled by 'jenkin ...
看看上传文件之后，能否看到
回复

使用道具 举报

ran_newstar 发表于 2014-8-22 11:01:34
本帖最后由 ran_newstar 于 2014-8-22 11:31 编辑
howtodown 发表于 2014-8-22 10:54
看看上传文件之后，能否看到

格式化都没成功还能上传文件吗？

现在使用hadoop dfs -mkdir /home/hadoop/data/hadoop-2.4.0/test就name.dir目录里有文件产生。
[hadoop@master current]$ pwd
/home/hadoop/dfs/hadoop-2.4.0/name/current
[hadoop@master current]$ ls -l
total 2064
-rw-rw-r--. 1 hadoop hadoop 1048576 Aug 21 19:49 edits_0000000000000000001-0000000000000000001
-rw-rw-r--. 1 hadoop hadoop 1048576 Aug 21 19:52 edits_inprogress_0000000000000000002
-rw-rw-r--. 1 hadoop hadoop     353 Aug 21 19:45 fsimage_0000000000000000000
-rw-rw-r--. 1 hadoop hadoop      62 Aug 21 19:45 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 hadoop hadoop       2 Aug 21 19:52 seen_txid
-rw-rw-r--. 1 hadoop hadoop     204 Aug 21 19:45 VERSION
[hadoop@master current]$ date
Thu Aug 21 19:57:01 PDT 2014
无文件：
[hadoop@master hadoop-2.4.0]$ pwd
/home/hadoop/tmp/hadoop-2.4.0
[hadoop@master hadoop-2.4.0]$ ls

无文件：
[hadoop@master ~]$ cd data/hadoop-2.4.0/
[hadoop@master hadoop-2.4.0]$ ls
[hadoop@master hadoop-2.4.0]$ pwd
/home/hadoop/data/hadoop-2.4.0

另外rpc-address端口号是9001还是9000？我按照论坛里的2.4.0完全分布式按照文档是9001.
-----------------------------------------------
hdfs-site.xml:
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/home/hadoop/data/hadoop-2.4.0</value>
  <description>Determines where on the local filesystem an DFS data node
  should store its blocks.  If this is a comma-delimited
  list of directories, then data will be stored in all named
  directories, typically on different devices.
  Directories that do not exist are ignored.
  </description>
</property>



<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/home/hadoop/dfs/hadoop-2.4.0/name</value>
  <description>Determines where on the local filesystem the DFS name node
      should store the name table(fsimage).  If this is a comma-delimited list
      of directories then the name table is replicated in all of the
      directories, for redundancy. </description>
</property>


<property>
  <name>dfs.namenode.rpc-address</name>
  <value>master:9001</value>
  <description>
    RPC address that handles all clients requests. In the case of HA/Federation where multiple namenodes exist,
    the name service id is added to the name e.g. dfs.namenode.rpc-address.ns1
    dfs.namenode.rpc-address.EXAMPLENAMESERVICE
    The value of this property will take the form of nn-host1:rpc-port.
  </description>
</property>

---------------------------------------------------------------------------
core-site.xml:
<property>
  <name>hadoop.tmp.dir</name>
  <value>file:/home/hadoop/tmp/hadoop-2.4.0</value>
  <description>A base for other temporary directories.</description>
</property>


<property>
  <name>fs.defaultFS</name>
  <value>hdfs://master:9000</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
  <description>Deprecated. Use (fs.defaultFS) property
  instead</description>
</property>


回复

使用道具 举报

ran_newstar 发表于 2014-8-22 11:07:43
本帖最后由 ran_newstar 于 2014-8-22 11:18 编辑

[hadoop@master sbin]$ hadoop dfs -mkdir /home/hadoop/data/hadoop-2.4.0/test
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

14/08/21 20:05:40 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 20:05:40 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts;  Ignoring.
14/08/21 20:05:41 WARN conf.Configuration: mapred-site.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval;  Ignoring.
mkdir: Call From master/192.168.1.121 to master:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

[hadoop@master current]$ ls -l
total 2064
-rw-rw-r--. 1 hadoop hadoop 1048576 Aug 21 19:49 edits_0000000000000000001-0000000000000000001
-rw-rw-r--. 1 hadoop hadoop 1048576 Aug 21 19:52 edits_inprogress_0000000000000000002
-rw-rw-r--. 1 hadoop hadoop     353 Aug 21 19:45 fsimage_0000000000000000000
-rw-rw-r--. 1 hadoop hadoop      62 Aug 21 19:45 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 hadoop hadoop       2 Aug 21 19:52 seen_txid
-rw-rw-r--. 1 hadoop hadoop     204 Aug 21 19:45 VERSION
[hadoop@master current]$ date
Thu Aug 21 20:06:18 PDT 2014


如果格式化完全成功，那么三个目录tmp,data,name都应该有相应文件产生。你说的格式化只是name成功了。
回复

使用道具 举报

1234567下一页
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
ran_newstar

注册会员

关注

  • 2

    主题

  • 58

    帖子

  • 3

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000