spark-shell yarn-client模式启动异常问题

spark-shell yarn-client模式启动总是出现异常。

运行环境如下：
spark1.3
hadoop2.6 HA
jdk1.8.0_31

spark-env.sh

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/hadoop/spark
export SPARK_JAR=/hadoop/spark/lib/spark-assembly-1.3.0-hadoop2.4.0.jar
export PATH=$SPARK_HOME/bin:$PATH



错误日志：
15/03/31 09:00:43 INFO yarn.Client: Submitting application 1 to ResourceManager
15/03/31 09:00:43 INFO impl.YarnClientImpl: Submitted application application_1427763283312_0001
15/03/31 09:00:44 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:44 INFO yarn.Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1427763643558
         final status: UNDEFINED
         tracking URL: http://hadoop1:8088/proxy/application_1427763283312_0001/
         user: root
15/03/31 09:00:45 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:46 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:47 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:48 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:49 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:50 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:51 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:52 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:53 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:54 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:55 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:56 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:57 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:58 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:00:59 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:00 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:02 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:03 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:04 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:05 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:06 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:07 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:08 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:09 INFO yarn.Client: Application report for application_1427763283312_0001 (state: ACCEPTED)
15/03/31 09:01:10 INFO yarn.Client: Application report for application_1427763283312_0001 (state: FAILED)
15/03/31 09:01:10 INFO yarn.Client:
         client token: N/A
         diagnostics: Application application_1427763283312_0001 failed 2 times due to AM Container for appattempt_1427763283312_0001_000002 exited with  exitCode: -103
For more detailed output, check application tracking page:http://hadoop1:8088/proxy/application_1427763283312_0001/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=2990,containerID=container_1427763283312_0001_02_000001] is running beyond virtual memory limits. Current usage: 125.6 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1427763283312_0001_02_000001 :
        |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
        |- 2993 2990 2990 2990 (java) 419 122 2305159168 31860 /usr/java/jdk1.8.0_31/bin/java -server -Xmx512m -Djava.io.tmpdir=/hadoop/hadoop_tmp/nm-local-dir/usercache/root/appcache/application_1427763283312_0001/container_1427763283312_0001_02_000001/tmp -Dspark.fileserver.uri=http://192.168.161.128:52351 -Dspark.driver.port=43170 -Dspark.executor.id=<driver> -Dspark.app.name=Spark shell -Dspark.master=yarn-client -Dspark.driver.appUIAddress=http://hadoop1:4040 -Dspark.repl.class.uri=http://192.168.161.128:52415 -Dspark.jars= -Dspark.driver.host=hadoop1 -Dspark.tachyonStore.folderName=spark-941e53c2-79bb-4907-a88e-80ac76364d8c -Dspark.yarn.app.container.log.dir=/hadoop/hadoop/logs/userlogs/application_1427763283312_0001/container_1427763283312_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg hadoop1:43170 --executor-memory 1024m --executor-cores 1 --num-executors 2
        |- 2990 2988 2990 2990 (bash) 0 0 108650496 303 /bin/bash -c /usr/java/jdk1.8.0_31/bin/java -server -Xmx512m -Djava.io.tmpdir=/hadoop/hadoop_tmp/nm-local-dir/usercache/root/appcache/application_1427763283312_0001/container_1427763283312_0001_02_000001/tmp '-Dspark.fileserver.uri=http://192.168.161.128:52351' '-Dspark.driver.port=43170' '-Dspark.executor.id=<driver>' '-Dspark.app.name=Spark shell' '-Dspark.master=yarn-client' '-Dspark.driver.appUIAddress=http://hadoop1:4040' '-Dspark.repl.class.uri=http://192.168.161.128:52415' '-Dspark.jars=' '-Dspark.driver.host=hadoop1' '-Dspark.tachyonStore.folderName=spark-941e53c2-79bb-4907-a88e-80ac76364d8c' -Dspark.yarn.app.container.log.dir=/hadoop/hadoop/logs/userlogs/application_1427763283312_0001/container_1427763283312_0001_02_000001 org.apache.spark.deploy.yarn.ExecutorLauncher --arg 'hadoop1:43170' --executor-memory 1024m --executor-cores 1 --num-executors  2 1> /hadoop/hadoop/logs/userlogs/application_1427763283312_0001/container_1427763283312_0001_02_000001/stdout 2> /hadoop/hadoop/logs/userlogs/application_1427763283312_0001/container_1427763283312_0001_02_000001/stderr

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1427763643558
         final status: FAILED
         tracking URL: http://hadoop1:8088/cluster/app/application_1427763283312_0001
         user: root
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:113)
        at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
        at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:379)
        at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1016)
        at $iwC$$iwC.<init>(<console>:9)
        at $iwC.<init>(<console>:18)
        at <init>(<console>:20)
        at .<init>(<console>:24)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:123)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
        at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973)
        at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
        at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
        at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

java.lang.NullPointerException
        at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:141)
        at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:49)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
        at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1027)
        at $iwC$$iwC.<init>(<console>:9)
        at $iwC.<init>(<console>:18)
        at <init>(<console>:20)
        at .<init>(<console>:24)
        at .<clinit>(<console>)
        at .<init>(<console>:7)
        at .<clinit>(<console>)
        at $print(<console>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
        at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1338)
        at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
        at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
        at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:856)
        at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:901)
        at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:813)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:130)
        at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
        at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:122)
        at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:973)
        at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:157)
        at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:106)
        at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:990)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:944)
        at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
        at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:944)
        at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1058)
        at org.apache.spark.repl.Main$.main(Main.scala:31)
        at org.apache.spark.repl.Main.main(Main.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

<console>:10: error: not found: value sqlContext
       import sqlContext.implicits._
              ^
<console>:10: error: not found: value sqlContext
       import sqlContext.sql





resourcemanager上的stderr : Total file length is 1282 bytes.


SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop/hadoop_tmp/nm-local-dir/usercache/root/filecache/10/spark-assembly-1.3.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/31 09:01:05 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/03/31 09:01:07 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1427763283312_0001_000002
15/03/31 09:01:08 INFO spark.SecurityManager: Changing view acls to: root
15/03/31 09:01:08 INFO spark.SecurityManager: Changing modify acls to: root
15/03/31 09:01:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/03/31 09:01:08 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM
15/03/31 09:01:08 INFO yarn.ApplicationMaster: Final app status: UNDEFINED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
15/03/31 09:01:08 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before final status was reported.)
15/03/31 09:01:08 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1427763283312_0001


请问这是什么问题？以前安装hbase也出现这个问题，后来删除hbase的lib上的slf4j的jar就解决了，但是这个不知道怎么解决。求救

求救求救啊

cookeem 发表于 2015-3-31 10:14
求救求救啊

检查下yarn是否启动

yarn已经启动，http://hadoop:8088已经可以查看状态

cookeem 发表于 2015-3-31 11:34
yarn已经启动，http://hadoop:8088已经可以查看状态

增大下内存还有 virtual memory

楼主怎么解决的？我碰到了一模一样的问题，不知道什么问题。不过我是spark1.3.1+hadoop2.2+hive0.12+java1.7

根据官方说明，spark要支持yarn client是需要自行编译的

我也遇到了这样的问题，就是一直是Application report for application_12122114213431321,请问你解决了吗？

同样的问题，大家是怎么解决的呢？求告知，非常感谢

bingyuac 发表于 2016-9-22 11:45
同样的问题，大家是怎么解决的呢？求告知，非常感谢

尝试修改spark-default.conf
具体如下
spark.driver.extraJavaOption -Dhdp.version=current
spark.yarn.am.extraJavaOption -Dhdp.version=current
然后启动spark shell
bin/spark-shell --master yarn-cluent