日志
hadoop2.6.0+zookeeper3.4.5+hive2.0分布式环境搭建
热度 1 ||
hadoop2.6.0+zookeeper3.4.5 已搭建好，详见：http://www.aboutyun.com/home.php?mod=space&uid=40798&do=blog&id=2975
mysql使用5.7.9版本，已安装好，
mysql 5.7.9 安装在192.168.56.110上：
关闭防火墙和SELINUX，
[root@client ~]# /usr/local/mysql/bin/mysql -uroot -p123456
create user hive identified by 'hive';
flush privileges;
GRANT USAGE ON *.* TO 'hive'@'%' IDENTIFIED BY 'hive' WITH GRANT OPTION;
创建数据库，用hive用户创建失败，就用root用户创建了：
create database hive DEFAULT CHARACTER SET latin1;
用户已建好，如下：
mysql> select user, host from mysql.user;
+-----------+--------------+
| user | host |
+-----------+--------------+
| hive | % |
| mysql | % |
| root | % |
| repl | 192.168.56.% |
| mysql | localhost |
| mysql.sys | localhost |
| root | localhost |
+-----------+--------------+
7 rows in set (0.00 sec)
准备工作已完成，现安装hive2.0:
1.将hive2.0拷贝到/application/hadoop下，解压：
tar -zxvf apache-hive-2.0.0-bin.tar.gz
重命名：
mv apache-hive-2.0.0 hive
2.环境变量
vi /etc/profile.d/java.sh
export JAVA_HOME=/application/hadoop/jdk
export HADOOP_HOME=/application/hadoop/hadoop
export HADOOP_PREFIX=/application/hadoop/hadoop
export ZOOKEEPER_HOME=/application/hadoop/zookeeper
export HIVE_HOME=/application/hadoop/hive
export PATH=$JAVA_HOME/bin:$HAOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$HIVE_HOME/bin:$PATH
3. 检查下zookeeper和protobuf的jar包是否和hbase保持一致，如果不一致，
拷贝protobuf.**.jar和zookeeper-3.4.6.jar到hive/lib下。
经检查，hbase和hive下的这两个包是一样的，都为：protobuf-java-2.5.0.jar，zookeeper-3.4.6.jar
4.配置hive-env.sh文件
cp hive-env.sh.template hive-env.sh
vi hive-env.sh
HADOOP_HOME=/application/hadoop/hadoop
export HIVE_CONF_DIR=/application/hadoop/hive/conf
5.拷贝mysql驱动到hive/lib下
cp mysql-connector-java-5.1.38-bin.jar /application/hadoop/hive/lib
授权777：
cd /application/hadoop/hive/lib/
chmod 777 mysql-connector-java-5.1.38-bin.jar
查看hbase与hive的通信包是否存在：
find ./ -name hive-hbase-handler*
./hive-hbase-handler-2.0.0.jar
6.hdfs新建文件并授权：
hadoop fs -mkdir -p /hive/warehouse
hadoop fs -mkdir -p /hive/scratchdir
hadoop fs -chmod g+w /hive/warehouse
hadoop fs -chmod g+w /hive/scratchdir
7.配置hive-site.xml
hive-site.xml配置，服务端与客户端要分别配置，服务端配置为：
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://ns1/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://ns1/hive/scratchdir</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.56.110:3306/hive?createData baseIfNotExist=true&useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateColumns</name>
<value>true</value>
</property>
</configuration>
客户端配置：
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://ns1/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://ns1/hive/scratchdir</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.56.11:9083,thrift://192.168.56.12:9083</value>
</property>
</configuration>
8.修改hive-env.sh
cd /application/hadoop/hive/conf/
cp hive-env.sh.template hive-env.sh
vi hive-env.sh
export HADOOP_HOME=/application/hadoop/hadoop
export HIVE_CONF_DIR=/application/hadoop/hive/conf
export HIVE_AUX_JARS_PATH=/application/hadoop/hive/lib
9.修改$HIVE_HOME/bin的hive-config.sh，增加以下三行:
#
# processes --config option from command line
#
export JAVA_HOME=/application/hadoop/jdk
export HIVE_HOME=/application/hadoop/hive
export HADOOP_HOME=/application/hadoop/hadoop
10.同步hive和hadoop的jline版本
cd /application/hadoop/hive/lib/
cp jline-2.12.jar /application/hadoop/hadoop/share/hadoop/yarn/lib
删除低版本的jline-0.9.94.jar
11.复制jdk的tools.jar到hive/lib下
cp $JAVA_HOME/lib/tools.jar /application/hadoop/hive/lib
12.同步到其它服务器
scp -r /application/hadoop/hive hadoop@yun01-nn-02:/application/hadoop/
scp -r /application/hadoop/hive hadoop@yun01-dn-01:/application/hadoop/
scp -r /application/hadoop/hive hadoop@yun01-dn-02:/application/hadoop/
12.服务端启动元数据库
hive --service metastore
tar -zxvf apache-hive-2.0.0-bin.tar.gz
重命名：
mv apache-hive-2.0.0 hive
2.环境变量
vi /etc/profile.d/java.sh
export JAVA_HOME=/application/hadoop/jdk
export HADOOP_HOME=/application/hadoop/hadoop
export HADOOP_PREFIX=/application/hadoop/hadoop
export ZOOKEEPER_HOME=/application/hadoop/zookeeper
export HIVE_HOME=/application/hadoop/hive
export PATH=$JAVA_HOME/bin:$HAOOP_HOME/bin:$ZOOKEEPER_HOME/bin:$HIVE_HOME/bin:$PATH
3. 检查下zookeeper和protobuf的jar包是否和hbase保持一致，如果不一致，
拷贝protobuf.**.jar和zookeeper-3.4.6.jar到hive/lib下。
经检查，hbase和hive下的这两个包是一样的，都为：protobuf-java-2.5.0.jar，zookeeper-3.4.6.jar
4.配置hive-env.sh文件
cp hive-env.sh.template hive-env.sh
vi hive-env.sh
HADOOP_HOME=/application/hadoop/hadoop
export HIVE_CONF_DIR=/application/hadoop/hive/conf
5.拷贝mysql驱动到hive/lib下
cp mysql-connector-java-5.1.38-bin.jar /application/hadoop/hive/lib
授权777：
cd /application/hadoop/hive/lib/
chmod 777 mysql-connector-java-5.1.38-bin.jar
查看hbase与hive的通信包是否存在：
find ./ -name hive-hbase-handler*
./hive-hbase-handler-2.0.0.jar
6.hdfs新建文件并授权：
hadoop fs -mkdir -p /hive/warehouse
hadoop fs -mkdir -p /hive/scratchdir
hadoop fs -chmod g+w /hive/warehouse
hadoop fs -chmod g+w /hive/scratchdir
7.配置hive-site.xml
hive-site.xml配置，服务端与客户端要分别配置，服务端配置为：
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://ns1/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://ns1/hive/scratchdir</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.56.110:3306/hive?createData baseIfNotExist=true&useSSL=false</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hive</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hive</value>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateTables</name>
<value>true</value>
</property>
<property>
<name>datanucleus.autoCreateColumns</name>
<value>true</value>
</property>
</configuration>
客户端配置：
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://ns1/hive/warehouse</value>
</property>
<property>
<name>hive.exec.scratchdir</name>
<value>hdfs://ns1/hive/scratchdir</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>false</value>
</property>
<property>
<name>hive.metastore.uris</name>
<value>thrift://192.168.56.11:9083,thrift://192.168.56.12:9083</value>
</property>
</configuration>
8.修改hive-env.sh
cd /application/hadoop/hive/conf/
cp hive-env.sh.template hive-env.sh
vi hive-env.sh
export HADOOP_HOME=/application/hadoop/hadoop
export HIVE_CONF_DIR=/application/hadoop/hive/conf
export HIVE_AUX_JARS_PATH=/application/hadoop/hive/lib
9.修改$HIVE_HOME/bin的hive-config.sh，增加以下三行:
#
# processes --config option from command line
#
export JAVA_HOME=/application/hadoop/jdk
export HIVE_HOME=/application/hadoop/hive
export HADOOP_HOME=/application/hadoop/hadoop
10.同步hive和hadoop的jline版本
cd /application/hadoop/hive/lib/
cp jline-2.12.jar /application/hadoop/hadoop/share/hadoop/yarn/lib
删除低版本的jline-0.9.94.jar
11.复制jdk的tools.jar到hive/lib下
cp $JAVA_HOME/lib/tools.jar /application/hadoop/hive/lib
12.同步到其它服务器
scp -r /application/hadoop/hive hadoop@yun01-nn-02:/application/hadoop/
scp -r /application/hadoop/hive hadoop@yun01-dn-01:/application/hadoop/
scp -r /application/hadoop/hive hadoop@yun01-dn-02:/application/hadoop/
12.服务端启动元数据库
hive --service metastore
13.启动hive客户端
hive
执行如下：
[hadoop@yun01-dn-01 ~]$ hive
which: no hbase in (/usr/lib64/qt-3.3/bin:/application/hadoop/jdk/bin:/bin:/application/hadoop/hive/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hadoop/bin)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/application/hadoop/hive/lib/hive-jdbc-2.0.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/application/hadoop/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/application/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Logging initialized using configuration in file:/application/hadoop/hive/conf/hive-log4j2.properties
Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.
hive> show databases;
OK
default
/2 
