Hadoop 2.6.3 示例中sort运行出错

Hadoop 2.6.3自带示例中的sort，运行
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar sort -totalOrder 0.1 100000 10 /input /usr/output
运行后出现下面的错误：
16/03/10 23:05:53 INFO mapreduce.Job: Task Id : attempt_1456453931065_0029_m_000009_2, Status : FAILED
Error: java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:698)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.FileNotFoundException: File file:/home/cloud/tech/hadoop-2.6.3/tmp/nm-local-dir/usercache/cloud/appcache/application_1456453931065_0029/container_1456453931065_0029_01_000050/_partition.lst does not exist
        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1817)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841)
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:301)
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:88)
        ... 10 more


查了下Sort源码，有下面一段
//如果设置了抽样的话
                if (sampler != null) {
                        System.out.println("Sampling input to effect total-order sort...");
                        //TotalOrderPartitioner
                        job.setPartitionerClass(TotalOrderPartitioner.class);
                        Path inputDir = FileInputFormat.getInputPaths(job)[0];
                        inputDir = inputDir.makeQualified(inputDir.getFileSystem(conf));
                        Path partitionFile = new Path(inputDir, "_sortPartitioning");
                        TotalOrderPartitioner.setPartitionFile(conf, partitionFile);
                        InputSampler.<K, V> writePartitionFile(job, sampler);
                        URI partitionUri = new URI(partitionFile.toString() + "#" + "_sortPartitioning");
                        DistributedCache.addCacheFile(partitionUri, conf);
                }

这里为了全局排序，用了TotalOrderPartitioner，sampler写writePartitionFile的时候发现hdfs里根本没有写成功，各位有知道原因的吗？

hadoop权限修改了吗

hyj 发表于 2016-3-11 12:17
hadoop权限修改了吗

你是指什么权限啊？
我hadoop的安装用户、运行程序用户都是一样的

hyj 发表于 2016-3-11 12:17
hadoop权限修改了吗

用了下面两个命令

• hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar randomwriter /input  
• hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar sort -totalOrder 0.1 100000 10 /input /usr/output
  

按理说如果权限有问题，第一条命令应该不能写入随机数啊，但是第一条执行没问题，生成了/input文件夹和下面的数据文件

圣袭wgzly 发表于 2016-3-11 14:07
用了下面两个命令

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.3.jar randomw ...

File file:/home/cloud/tech/hadoop-2.6.3/tmp/nm-local-dir/usercache/cloud/appcache/application_1456453931065_0029/container_1456453931065_0029_01_000050/_partition.lst does not exist

hadoop读取本地文件？？？
还是hdfs文件。这是没有找到文件

atsky123 发表于 2016-3-11 20:04
File file:/home/cloud/tech/hadoop-2.6.3/tmp/nm-local-dir/usercache/cloud/appcache/application_1456 ...

是这样的，sort程序用了DistributedCache，并把正常情况下会生成的partition file加进去，所以会读取本地路径
但是问题在于这个原始partition file根本没产生。。