分享

新手指导,该如何在开发环境中,创建mapreduce程序

langke93 发表于 2015-4-19 15:55:20
yunmin 发表于 2015-4-19 15:05
楼上那个什么问题造成的?每台机的InputPath和OutputPath都应该不一样吗?



下面改成自己的路径

static final String INPUT_PATH = "hdfs://aboutyun:9000/yy1";
static final String OUT_PATH = "hdfs://aboutyun:9000/Output";
回复

使用道具 举报

yunmin 发表于 2015-4-22 09:24:20
langke93 发表于 2015-4-19 15:55
下面改成自己的路径

static final String INPUT_PATH = "hdfs://aboutyun[/backcolor ...

已经解决了!谢谢Qq

回复

使用道具 举报

fzu3144 发表于 2015-7-15 16:24:41
[mw_shl_code=java,true]SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/I:/%e5%ae%89%e8%a3%85%e9%83%a8%e7%bd%b2/%e5%bc%95%e5%85%a5jar%e5%8c%85/hadoop-eclipse-plugin-2.2.0%e4%be%9d%e8%b5%96%e5%8c%85/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/I:/%e5%ae%89%e8%a3%85%e9%83%a8%e7%bd%b2/%e5%bc%95%e5%85%a5jar%e5%8c%85/hadoop-2.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/F:/master/hbase-0.98.12.1-hadoop2/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-07-15 16:21:20,201 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2015-07-15 16:21:21,279 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at master/192.168.37.136:8032
2015-07-15 16:21:23,270 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(64)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2015-07-15 16:21:23,442 WARN  [main] mapreduce.JobResourceUploader (JobResourceUploader.java:uploadFiles(171)) - No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
2015-07-15 16:21:23,609 INFO  [main] input.FileInputFormat (FileInputFormat.java:listStatus(283)) - Total input paths to process : 1
2015-07-15 16:21:25,497 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(201)) - number of splits:1
2015-07-15 16:21:26,388 INFO  [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(290)) - Submitting tokens for job: job_1436614419410_0031
2015-07-15 16:21:26,838 INFO  [main] mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(371)) - Job jar is not present. Not adding any jar to the list of resources.
2015-07-15 16:21:27,433 INFO  [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(273)) - Submitted application application_1436614419410_0031
2015-07-15 16:21:27,537 INFO  [main] mapreduce.Job (Job.java:submit(1294)) - The url to track the job: http://master:8088/proxy/application_1436614419410_0031/
2015-07-15 16:21:27,539 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1339)) - Running job: job_1436614419410_0031
2015-07-15 16:21:32,640 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_1436614419410_0031 running in uber mode : false
2015-07-15 16:21:32,646 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) -  map 0% reduce 0%
2015-07-15 16:21:32,710 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Job job_1436614419410_0031 failed with state FAILED due to: Application application_1436614419410_0031 failed 2 times due to AM Container for appattempt_1436614419410_0031_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://master:8088/cluster/app/application_1436614419410_0031Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1436614419410_0031_02_000001
Exit code: 1
Exception message: /bin/bash: line 0: fg: no job control

Stack trace: ExitCodeException exitCode=1: /bin/bash: line 0: fg: no job control

        at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
        at org.apache.hadoop.util.Shell.run(Shell.java:456)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
2015-07-15 16:21:32,779 INFO  [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 0
[/mw_shl_code]
回复

使用道具 举报

Alkaloid0515 发表于 2015-7-15 21:44:08
fzu3144 发表于 2015-7-15 16:24
[mw_shl_code=java,true]SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in  ...

应该是引用包重复了
回复

使用道具 举报

blue5whale 发表于 2016-4-28 21:58:50
quenlang 发表于 2014-9-28 23:01
楼主,我在eclipse中运行WordCount的实例程序时,没有任何报错信息,设置好输入路径和输出路径后,run,然 ...

我最近是刚接触hadoop,也碰到了跟你一模一样的问题,想问问你以前解决了这个问题了吗?
回复

使用道具 举报

when30 发表于 2016-4-28 22:01:58
blue5whale 发表于 2016-4-28 21:58
我最近是刚接触hadoop,也碰到了跟你一模一样的问题,想问问你以前解决了这个问题了吗?

最有可能环境不同造成的,上图看看
回复

使用道具 举报

blue5whale 发表于 2016-4-28 23:51:15
when30 发表于 2016-4-28 22:01
最有可能环境不同造成的,上图看看

这个图片是运行后的界面

这个图片是运行后的界面


回复

使用道具 举报

blue5whale 发表于 2016-4-28 23:57:06
when30 发表于 2016-4-28 22:01
最有可能环境不同造成的,上图看看

抱歉,对发图不怎么熟悉,不过我描述给你就知道了主要问题就是我在处理/user/Echizen/mytest下面的文件note.txt后,在mytest下面得到了out输出文件夹,但是里面没有内容,我每次刷新后out文件夹括号后面的数字就突然变成1然后又变回0了,总之就是没有输出结果,我感觉是log4j的配置问题,但是不知道如何配置
回复

使用道具 举报

when30 发表于 2016-4-29 07:28:31
blue5whale 发表于 2016-4-28 23:57
抱歉,对发图不怎么熟悉,不过我描述给你就知道了主要问题就是我在处理/user/Echizen/mytest下面的文件no ...

winutils.exe是否添加



详细参考:
零基础学习hadoop到上手工作线路指导(编程篇)
回复

使用道具 举报

blue5whale 发表于 2016-4-29 09:02:35
when30 发表于 2016-4-29 07:28
winutils.exe是否添加

都添加了,但是我只是添加了hadoop.dll和winutils.exe,我现在把其余几项也添加进去试试
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条