cloudera hue安装及Oozie的安装

本帖最后由 pig2 于 2014-2-9 19:27 编辑

思考：hue与Oozie之间的关系是什么？

下载hue 1.2
下载地址：
https://github.com/cloudera/hue/downloads/

HADOOP_HOME=/hadoop/hadoop0.20.2 PREFIX=/hadoop/hue make install
复制代码

以前叫cloudera desktop，使用django作为web框架

1. 安装

waiting .. .. ..
执行完会在PREFIX指定的路径下生成hue的服务程序

2.配置

在hue/desktop/conf配置hue.ini
[[database]]
engine=mysql
host=localhost
port=3306
user=peter
password=peter
name=hue
hadoop,hdfs,mapred配置
hadoop_home=/hadoop/hadoop0.20.2
hadoop_conf_dir=/hadoop/hadoop0.20.2/conf
hdfs_port=8020
http_port=50070
jobtracker_host=localhost

3.配置hive
hive_home=/hadoop/hive0.70
hive_conf_dir=/hadoop/hive0.70/conf
4.初始化

/hadoop/hue/build/env/bin/hue syncdb --noinput

5.启动

/hadoop/hue/build/env/bin/supervisor

---------------------------------------------------------------------------------------------------------------------------------------------------

Oozie的安装

Oozie是一种Java Web应用程序，它运行在Java servlet容器——即Tomcat——中，并使用数据库来存储以下内容：

工作流定义
当前运行的工作流实例，包括实例的状态和变量

Oozie工作流是放置在控制依赖DAG（有向无环图 Direct Acyclic Graph）中的一组动作（例如，Hadoop的Map/Reduce作业、Pig作业等），其中指定了动作执行的顺序。

Oozie翻译：驯象人，Hadoop得名来源于创始人Doug Cutting的孩子给一大象玩具取的名字，而oozie作为Hadoop上的工作流引擎，名字取得真的很贴切。
先获取oozie的版本，使用版本3.3.1.
http://mirror.bit.edu.cn/apache/oozie/3.3.1/
此版本需要自己build，需要先下载安装JDK, maven, hadoop, extjs libray 2.2.
我的环境:

Ubuntu
JDK 1.6.0_43
maven 3.0.7
Hadoop 1.0.4
extjs 2.2 (Oozie web-console需要）

1. Build：
./bin/mkdistro.sh -DskipTests
成功build后,会在目录里生成一个distribution.

hadoop安装时创建的hadoop组和hadoop用户对oozie目录是没有权限，所以最好一开始创建oozie组和oozie用户，同时赋予权限，不然会报错.偷懒用了root用户。新用户下记得export JAVA_HOME=<JDK安装路径>

2.安装oozie：
Create a libext/ directory in the directory where Oozie was expanded.If using a version of Hadoop bundled in Oozie hadooplibs/ , copy the corresponding Hadoop JARs from hadooplibs/ to the libext/ directory. If using a different version of Hadoop, copy the required Hadoop JARs from such version in the libext/ directory.
If using the ExtJS library copy the ZIP file to the libext/ directory.

根据官方文档在distibution的目录下（/usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1）创建libext，因为装的hadoop版本是1.0.4,所以拷贝$HADOOP_HOME/lib下的所有jar包到libext下，同时拷贝ext-2.2.zip到该目录下。
还要拷贝$HADOOP_HOME下的所有jar包到libext.

运行oozie-setup.sh创建oozie.war

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin 
./oozie-setup.sh
复制代码

成功运行后

New Oozie WAR file with added 'ExtJS library, JARs' at /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/oozie-server/webapps/oozie.war  
  
INFO: Oozie is ready to be started  
复制代码

拷贝oozie.war到tomcat部署目录下

root@ubuntu:/usr/local/oozie-3.3.1/webapp/src/main/webapp# cp /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/oozie.war ./  
复制代码

修改配置oozie-site.xml,将false改成true.vi /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/conf/oozie-site.xml

<property>
      <name>oozie.service.JPAService.create.db.schema</name>
      <value>true</value>
      <description>
         Creates Oozie DB.

         If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
         If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
      </description>
</property>
创建Oozie DB

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin  
./ooziedb.sh create -sqlfile oozie.sql -run  
复制代码

成功运行后

DONE  
Check DB schema does not exist  
DONE  
Check OOZIE_SYS table does not exist  
DONE  
Create SQL schema  
DONE  
DONE  
Create OOZIE_SYS table  
DONE  
Oozie DB has been created for Oozie version '3.3.1'   
复制代码

运行oozie

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin  
./oozie-start.sh  
复制代码

用浏览器访问http://localhost:11000/oozie/就能看到oozie web console.

来自群组: Hadoop技术组

pig2 · 发表于 2014-2-9 20:06:46

安装Hue和Oozie web console遇到的问题：
为了能够管理MapReduce Job和浏览HDFS文件系统，我们需要安装Hue服务，Hue依赖Oozie服务。部署完Hue和Oozie后，发现报错如下，现在给大家介绍一下解决方法。

To enable Oozie web console install the Ext JS library

下载和部署Ext JS

[root@nodeD oozie]# ll /usr/local/src/
total 6644
-rw-r--r-- 1 root root 6800612 Mar  7 09:44 ext-2.2.zip
[root@nodeD oozie]# unzip ext-2.2.zip
[root@nodeD oozie]# cd /var/lib/oozie/
[root@nodeD oozie]# ll
total 4
drwxr-xr-x 5 oozie oozie 4096 Mar  7 09:49 data
[root@nodeD oozie]# mv /usr/local/src/ext-2.2 .
复制代码

The deployment directory of Oozie workflows does not exist. Run “Setup Examples” on the Oozie workflow page.

在Hue界面上，点击Oozie Editor/Dashboard -> Workflow -> Setup Examples安装示例。

问题：

[06/Mar/2013 21:40:33 +0000] hive_site    CRITICAL Cannot understand remote metastore uri "thrift://nodeB:9083,thrift://nodeD:9083"
复制代码

尝试下面的方法有效：

[beeswax]
hive_conf_dir=/etc/hive/conf
复制代码


http://grokbase.com/t/cloudera/hue-user/132mwcgcm3/hive-settings-not-reflected-in-beeswax-ui

if you use CM you can't edit hue.ini as it will be overridden by CM.

If you want Hue to point to another hive-site.xml, you need to add the
property below to the "Hue Server Configuration Safety Valve for
hue_safety_valve_server.ini" property in the "Advanced" menu of the Hue
service.

[beeswax]

# Hive configuration directory, where hive-site.xml is located
hive_conf_dir=/etc/hive/conf

or click on the Beeswax service and insert your Hive properties in "Hive
Configuration Safety Valve", that way

<property>
<name>hive.security.authorization.enabled</name>
<value>true</value>
</property>
复制代码