worker连不上master，已困4天。。。

SSH 可以互相无密码登陆，ping也能ping通。但是worker就是连不上master

vanxd@master:~/i/Spark/spark-1.3.1-bin-hadoop2.6/sbin> ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /home/vanxd/i/Spark/spark-1.3.1-bin-hadoop2.6/sbin/../logs/spark-vanxd-org.apache.spark.deploy.master.Master-1-master.out
Slave-1: starting org.apache.spark.deploy.worker.Worker, logging to /home/vanxd/i/Spark/spark-1.3.1-bin-hadoop2.6/sbin/../logs/spark-root-org.apache.spark.deploy.worker.Worker-1-Slave-1.out


这个是启动成功了吧?


但是为什么没有呢..?

我打开了worker的log:
Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
Retrying connection to master(attempt # 3)
Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
Retrying connection to master(attempt # 3)Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
Retrying connection to master(attempt # 3)
Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
Retrying connection to master(attempt # 3)

好像是连接不上master,但是我直接ssh master是可以无密码连接的啊.
那共sparkMaster是什么东东..?
[root@Slave2 bin]# ./spark-class org.apache.spark.deploy.worker.Worker spark://master:7077
15/06/01 16:36:19 INFO worker.Worker: Registered signal handlers for [TERM, HUP, INT]
15/06/01 16:36:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/01 16:36:21 INFO spark.SecurityManager: Changing view acls to: root
15/06/01 16:36:21 INFO spark.SecurityManager: Changing modify acls to: root
15/06/01 16:36:21 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/06/01 16:36:22 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/06/01 16:36:22 INFO Remoting: Starting remoting
15/06/01 16:36:22 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@Slave2:55552]
15/06/01 16:36:22 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkWorker@Slave2:55552]
15/06/01 16:36:22 INFO util.Utils: Successfully started service 'sparkWorker' on port 55552.
15/06/01 16:36:22 INFO worker.Worker: Starting Spark worker Slave2:55552 with 1 cores, 1024.0 MB RAM
15/06/01 16:36:22 INFO worker.Worker: Running Spark version 1.3.1
15/06/01 16:36:22 INFO worker.Worker: Spark home: /home/vanxd/i/Spark/spark-1.3.1-bin-hadoop2.6
15/06/01 16:36:23 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/06/01 16:36:23 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8081
15/06/01 16:36:23 INFO util.Utils: Successfully started service 'WorkerUI' on port 8081.
15/06/01 16:36:23 INFO ui.WorkerWebUI: Started WorkerWebUI at http://Slave2:8081
15/06/01 16:36:23 INFO worker.Worker: Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
15/06/01 16:36:34 INFO worker.Worker: Retrying connection to master (attempt # 1)
15/06/01 16:36:34 INFO worker.Worker: Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
15/06/01 16:36:45 INFO worker.Worker: Retrying connection to master (attempt # 2)
15/06/01 16:36:45 INFO worker.Worker: Connecting to master akka.tcp://sparkMaster@master:7077/user/Master...
15/06/01 16:36:56 INFO worker.Worker: Retrying connection to master (attempt # 3)
15/06/01 16:36:56 INFO worker.Worker: Connecting to master akka.tcp://sparkMaster@master:7077/u        

jps 看看进程是否在
可能没有启动起来

先看进程，在看日志。
界面中全是0

是不是你的worker进程已经启动了 但是worker向master注册的时候失败了 应该是worker结点的配置有问题

ding123lei 发表于 2015-6-2 19:32
是不是你的worker进程已经启动了 但是worker向master注册的时候失败了 应该是worker结点的配置有问题

先感谢下你的回复,谢谢!!

是的,worker进程已经启动,我看日志就是worker向master注册都时候,找不到master的地址,unreachable ip address.
我都是按照视频,博客各种资料上都试过都spark-env.sh都配置来的

master上配置,然后复制到worker上 .
这样就可以启动了呀.就出现那个问题了!

Alkaloid0515 发表于 2015-6-2 15:06
先看进程，在看日志。
界面中全是0

先感谢下你的回复,谢谢!

worker进程已经启动,就是不能注册到master.

ding123lei 发表于 2015-6-2 19:32
是不是你的worker进程已经启动了 但是worker向master注册的时候失败了 应该是worker结点的配置有问题

export SPARK_MASTER_IP=master
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_CORES=1
export SPARK_WORKER_INSTANCES=1
export SPARK_WORKER_MEMORY=1g
export MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}

这个是我的spark-env.sh

你的master和ip有没有加入到/etc/hosts里面
或者你的SPARK_MASTER_IP直接使用ip地址呢？

看看防火墙是否关闭，redhat、centos看看selinux，iptables

ding123lei 发表于 2015-6-3 09:46
你的master和ip有没有加入到/etc/hosts里面
或者你的SPARK_MASTER_IP直接使用ip地址呢？

可以直接ssh master, ssh slave1
hosts文件已经配号了..
直接写IP,也不行..