gobblin运行在yarn上报错

gobblin 跑YARN模式时，不知是啥没设置，如下：
WARN [NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
WARN [ZKHelixAdmin] Root directory exists.Cleaning the root directory:/GobblinYarn
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'admin'
        at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
        at com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:174)
        at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:285)
        at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:794)
ERROR [GobblinYarnAppLauncher] Failed to shutdown the GobblinYarnAppLauncher
java.io.IOException: Filesystem closed
        at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:857)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2118)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
        at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
        at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
        at gobblin.yarn.GobblinYarnAppLauncher.cleanUpAppWorkDirectory(GobblinYarnAppLauncher.java:729)
        at gobblin.yarn.GobblinYarnAppLauncher.stop(GobblinYarnAppLauncher.java:324)
        at gobblin.yarn.GobblinYarnAppLauncher$2.run(GobblinYarnAppLauncher.java:781)


不知哪位大神弄过，望解答，谢谢!

日志在
${gobblin.yarn.work.dir}/_applogs
这个目录下，可以看看日志

看到一篇文章，不错是国外的被墙，不过复制下来了。楼主可以参考

Gobblin on Yarn

可能是权限的原因

应该是两个问题一个是目录已经存在，楼主看看是否有这个目录，另外一个找不到配置，应该是都配置了， 对于admin可能是权限的问题

easthome001 发表于 2016-11-10 14:47
日志在
${gobblin.yarn.work.dir}/_applogs
这个目录下，可以看看日志

推荐文章是官网的，看过了，谢谢 Yarn中日志：2016-11-10 14:06:34 CST INFO  [LogCopier RUNNING] gobblin.util.ExecutorsUtils  - Successfully shutdown ExecutorService: java.util.concurrent.ScheduledThreadPoolExecutor@8954ee4[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]2016-11-10 14:06:34 CST INFO  [YarnService STARTING] org.apache.hadoop.yarn.client.RMProxy  - Connecting to ResourceManager at hadoop-offline-60/192.168.3.60:80302016-11-10 14:06:34 CST ERROR [LogCopier RUNNING] gobblin.runtime.app.ServiceBasedAppLauncher$1  - Service LogCopier has failed.java.io.FileNotFoundException: File file:/data/1/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001,/data/2/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001,/data/3/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001 does not exist        at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:590)        at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:803)        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:580)        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:425)        at gobblin.util.FileListUtils.listFilesRecursively(FileListUtils.java:60)        at gobblin.util.logs.LogCopier.checkSrcLogFiles(LogCopier.java:178)        at gobblin.util.logs.LogCopier.runOneIteration(LogCopier.java:166)        at com.google.common.util.concurrent.AbstractScheduledService$1$1.run(AbstractScheduledService.java:172)        at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)        at java.lang.Thread.run(Thread.java:745)2016-11-10 14:06:34 CST INFO  [LogCopier RUNNING] gobblin.runtime.app.ServiceBasedAppLauncher  - Shutting down the application2016-11-10 14:06:34 CST ERROR [GobblinHelixJobScheduler STARTING] gobblin.runtime.app.ServiceBasedAppLauncher$1  - Service GobblinHelixJobScheduler has failed.java.lang.IllegalStateException: Scheduler service is not running.        at gobblin.scheduler.JobScheduler.startUp(JobScheduler.java:172)        at gobblin.cluster.GobblinHelixJobScheduler.startUp(GobblinHelixJobScheduler.java:75)        at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)        at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)        at java.lang.Thread.run(Thread.java:745)

langke93 发表于 2016-11-10 15:44
应该是两个问题一个是目录已经存在，楼主看看是否有这个目录，另外一个找不到配置，应该是都配置了， 对于a ...

谢谢，
1）上文提到的/GobblinYarn为zookeeper中节点信息
2） admin用户含义是？意思是gobblin内部会内建admin用户？

Riordon 发表于 2016-11-10 15:46
推荐文章是官网的，看过了，谢谢 Yarn中日志：2016-11-10 14:06:34 CST INFO  [LogCopier RUNNING] gobbl ...

- Service LogCopier has failed.java.io.FileNotFoundException: File file:/data/1/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001,/data/2/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001,/data/3/yarn/container-logs/application_1478051839731_0042/container_1478051839731_0042_01_000001 does not exist     

上面是说LogCopier 这个工具类找下面的文件，但是没有找到。楼主看看文件是否存在。