日志
Spark安装调试初探
热度 1 ||
Spark的安装其实是比较简单的，对于初学者来说，没必要看那么多复杂的资料，那些资料也不一定对，也许是过时的，所以，越简单越好，也有利于保持学习的美好心情
一、安装spark的前提准备
假设我已经安装好了hadoop, hadoop的用户名是hadoop,密码也是hadoop，共有2台虚拟机，1台是192.168.1.11,另外一台是192.168.1.12,两台机器都已经配置好了互信；
二、安装scala
在集群的每个节点上安装Scala，方法如下：
1、解压安装包
[hadoop@rhel6531 ~]$ tar xf ./setup/scala-2.11.4.tar
[hadoop@rhel6531 ~]$ ll
total 20
drwxrwxr-x 3 hadoop hadoop 4096 Mar 18 09:22 dfs
drwxr-xr-x 10 hadoop hadoop 4096 Mar 18 02:03 hadoop-2.4.0
drwxrwxr-x 6 hadoop hadoop 4096 Oct 24 07:00 scala-2.11.4
drwxrwxr-x 2 hadoop hadoop 4096 Mar 19 09:55 setup
drwxrwxr-x 3 hadoop hadoop 4096 Mar 18 09:23 tmp
[hadoop@rhel6531 ~]$ pwd
/home/hadoop
2、配置环境变量
在.bash_profile 中加入下面配置
export SCALA_HOME=/home/hadoop/scala-2.11.4
export PATH=$SCALA_HOME/bin:$PATH
4、验证安装
[hadoop@rhel6531 ~]$ . .bash_profile
[hadoop@rhel6531 ~]$ scala
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_45).
Type in expressions to have them evaluated.
Type :help for more information.
scala>
在使环境变量生效后，可运行scala 命令，进入scala 的Shell。
用:quit退出
二、安装Spark
1、解压安装包
下载spark1.3.0，并解压到安装目录下。为了配置方便，可以以解压后的目录更名。
[hadoop@rhel6531 ~]$ tar xzf ./setup/spark-1.3.0-bin-hadoop2.4.tgz
[hadoop@rhel6531 ~]$ ll
total 24
drwxrwxr-x 3 hadoop hadoop 4096 Mar 18 09:22 dfs
drwxr-xr-x 10 hadoop hadoop 4096 Mar 18 02:03 hadoop-2.4.0
drwxrwxr-x 6 hadoop hadoop 4096 Oct 24 07:00 scala-2.11.4
drwxrwxr-x 2 hadoop hadoop 4096 Mar 19 09:55 setup
drwxrwxr-x 10 hadoop hadoop 4096 Mar 6 08:31 spark-1.3.0-bin-hadoop2.4
drwxrwxr-x 3 hadoop hadoop 4096 Mar 18 09:23 tmp
[hadoop@rhel6531 ~]$ mv spark-1.3.0-bin-hadoop2.4 spark-1.3.0
[hadoop@rhel6531 ~]$ ls
dfs hadoop-2.4.0 scala-2.11.4 setup spark-1.3.0 tmp
[hadoop@rhel6531 ~]$
2、配置环境变量
export SPARK_HOME=/home/hadoop/spark-1.3.0
export PATH=$SPARK_HOME/bin:$PATH
3、配置Spark
Spark 的配置文件在其conf 目录中，如/home/hadoop/spark-1.3.0/conf。
创建spark-defaults.conf
[hadoop@rhel6531 conf]$ mv spark-defaults.conf.template spark-defaults.conf
[hadoop@rhel6531 conf]$ cat spark-defaults.conf
# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.
# Example:
# spark.master spark://master:7077
# spark.eventLog.enabled true
# spark.eventLog.dir hdfs://namenode:8021/directory
# spark.serializer org.apache.spark.serializer.KryoSerializer
# spark.driver.memory 5g
# spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value
-Dnumbers="one two three"
[hadoop@rhel6531 conf]$
4、创建slaves
[hadoop@rhel6531 conf]$ mv slaves.template slaves
[hadoop@rhel6531 conf]$ vi slaves
# A Spark Worker will be started on each of the machines listed below.
192.168.1.12
mv spark-env.sh.template spark-env.sh
vi spark-env.sh
HADOOP_CONF_DIR=/home/hadoop/hadoop-2.4.0/etc/hadoop
SPARK_EXECUTOR_INSTANCES=2
SPARK_EXECUTOR_CORES=1
SPARK_EXECUTOR_MEMORY=1G
SPARK_DRIVER_MEMORY=1G
SPARK_YARN_APP_NAME=GBase
SPARK_YARN_QUEUE=default
[hadoop@rhel6531 conf]$
5、同步各节点安装内容
同步环境变量[hadoop@rhel6531 ~]$ cpush .bash_profile ~
同步scala和spark 安装目录
三、运行spark
1、启动hadoop
2、启动spark，只需要运行脚本sh start-all.sh 即可
运行后，使用Jps查看进程启动情况
MASTER节点
5229 ResourceManager
5086 SecondaryNameNode
4755 Master
4907 Jps
4906 NameNode
SLAVE节点
30160 Worker
3371 NodeManager
30252 Jps
3266 DataNode
3、测试spark
1、测试spark的spark-shell
编辑test.txt并上传至/text.txt
user test
user test2
users test3
scala> val file=sc.textFile("hdfs://node1:9000/text.txt")
scala> users=file.filter(line => line.contain("user"))
scala> users.count
3
2、测试sparksql
spark-sql
create database testdb;
use testdb;
create table t1 (id1 int);
load data inpath '/t1.txt' into table t1;
select * from t1;
select id1 ,count(1) from t1 group by id1;
都能成功显示
10 1
至此，spark安装成功。
总结： spark只不过是处理大数据的一个工具而已，本身不需要太多的调测，如果遇到很多问题，那说明安装方法不对或者软件不对，不要浪费过多时间。抓紧时间问。
/2 
