分享

spark提交作业报错Failed to connect to master/172.16.97.213:7070

医生 2017-9-30 23:52:58 发表于 异常错误 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 4 20648
弄了一晚上了,网上找的都试过了就是不行
用的云服务器  172.16.97.213是master的内网

报错如下:

hduser@master:~/hadoop2.6/spark1.6/sbin$ spark-submit --master spark://master:7070 --deploy-mode cluster --name "wordcount" --class WordCount /home/hduser/files/wordcount.jar hdfs://master:9000/wordcount/input hdfs://master:9000/wordcount/output
Running Spark using the REST application submission protocol.
17/09/30 11:46:25 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://master:7070.
17/09/30 11:46:25 WARN rest.RestSubmissionClient: Unable to connect to server spark://master:7070.
Warning: Master endpoint spark://master:7070 was not a REST server. Falling back to legacy submission gateway instead.
17/09/30 11:46:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.io.IOException: Failed to connect to master/172.16.97.213:7070
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
        at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
        at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
        at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused: master/172.16.97.213:7070
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        ... 1 more



你们有遇到过类似的问题吗?怎么解决的?

spark-env.sh配置如下:
export HADOOP_HOME=/home/hduser/hadoop2.6
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SCALA_HOME=/home/hduser/hadoop2.6/scala2.10
export JAVA_HOME=/usr/jdk1.8
export SPARK_WORKER_MEMORY=2G
export SPARK_MASTER_PORT=7077
export STANDALONE_SPARK_MASTER_HOST=master
export SPARK_MASTER_IP=$STANDALONE_SPARK_MASTER_HOST


访问master7070端口显示如下:
hduser@node1:~/hadoop2.6/etc/hadoop$ nmap -p 7077 172.16.97.213

Starting Nmap 7.01 ( https://nmap.org ) at 2017-09-30 11:42 EDT
Nmap scan report for master (172.16.97.213)
Host is up (0.00095s latency).
PORT     STATE SERVICE
7077/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.03 seconds


host配置如下:
ff02::1 ip6-allnodes
#       The     following       lines   are     desirable       for     IPv6    capable hosts
ff02::2 ip6-allrouters
::1     localhost       ip6-localhost   ip6-loopback
172.16.97.213   master.cloud.mos        master
172.16.97.216   node2
172.16.97.215   node1
#127.0.0.1      localhost


spark菜鸟,有知道的朋友麻烦教教我是怎么回事?

已有(4)人评论

跳转到指定楼层
qcbb001 发表于 2017-10-3 11:07:57
在哪提交的作业。是哪个客户端ip下。
回复

使用道具 举报

医生 发表于 2017-10-3 22:51:33
qcbb001 发表于 2017-10-3 11:07
在哪提交的作业。是哪个客户端ip下。

master客户端
回复

使用道具 举报

zstu 发表于 2017-10-10 10:19:26
楼主解决了吗  试试将 --master spark://master:7070中master的hostname改成ip地址
回复

使用道具 举报

fengfengda 发表于 2017-11-2 18:43:59
楼主解决了吗,是什么原因呢?
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条