本帖最后由 pig2 于 2014-7-16 21:24 编辑
问题导读:
1.编译 sqoop 1.4.4需要哪些软件?
2.如何在build.xml指定hadoop版本?
3.Compile failed; see the compiler error output for details.该如何解决?
![](static/image/hrline/4.gif)
-- 编译前准备:看了一下 README.txt文件,需要以下软件包:
Additionally, building the documentation requires these tools:
* asciidoc
* make
* python 2.5+
* xmlto
* tar
* gzip
yum -y install asciidoc
yum -y install make
yum -y install xmlto
yum -y install tar
yum -y install gzip
-- python 自己去安装
-----------------------------------------------------------------------------------------------------------------------------
-- 第一步:解压 sqoop-1.4.4.tar.gz 文件到 /opt/software目录下(在该目录下将生成 sqoop-1.4.4 文件夹)
cd /opt/software
tar -xvf sqoop-1.4.4.tar.gz
-----------------------------------------------------------------------------------------------------------------------------
-- 第二步:cd 到 sqoop-1.4.4 文件夹, 修改build.xml文件中指定的hadoop版本为2.4.1(其他不变,其他相关版本尝试修改好像有问题,提示下载不到相关版本的文件)
cd /opt/software/sqoop-1.4.4
vi build.xml
<elseif>
<equals arg1="${hadoopversion}" arg2="200" />
<then>
<property name="hadoop.version" value="2.4.1" />
<property name="hbase.version" value="0.94.2" />
<property name="zookeeper.version" value="3.4.2" />
<property name="hadoop.version.full" value="2.4.1" />
<property name="hcatalog.version" value="0.11.0" />
</then>
</elseif>
-----------------------------------------------------------------------------------------------------------------------------
-- 第三步:运行ant package
[root@funshion-hadoop194 sqoop-1.4.4]# ant package
...
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
BUILD FAILED
/opt/software/sqoop-1.4.4/build.xml:1282: impossible to resolve dependencies:
resolve failed - see output for details
Total time: 27 seconds
[ivy:resolve] com.google.protobuf#protobuf-java;2.4.1 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| hadoop200 | 154 | 59 | 58 | 37 || 120 | 48 |
---------------------------------------------------------------------
[ivy:resolve]
[ivy:resolve] :: problems summary ::
[ivy:resolve] :::: WARNINGS
[ivy:resolve] [FAILED ] org.mortbay.jetty#jetty;6.1.26!jetty.zip: (0ms)
[ivy:resolve] ==== fs: tried
[ivy:resolve] /root/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.zip
[ivy:resolve] ==== apache-snapshot: tried
[ivy:resolve] https://repository.apache.org/co ... 26/jetty-6.1.26.zip
[ivy:resolve] ==== datanucleus: tried
[ivy:resolve] http://www.datanucleus.org/downl ... 26/jetty-6.1.26.zip
[ivy:resolve] ==== cloudera-releases: tried
[ivy:resolve] https://repository.cloudera.com/ ... 26/jetty-6.1.26.zip
[ivy:resolve] ==== cloudera-staging: tried
[ivy:resolve] https://repository.cloudera.com/ ... 26/jetty-6.1.26.zip
[ivy:resolve] ==== maven2: tried
[ivy:resolve] http://repo1.maven.org/maven2/or ... 26/jetty-6.1.26.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: FAILED DOWNLOADS ::
[ivy:resolve] :: ^ see resolution messages for details ^ ::
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve] :: org.mortbay.jetty#jetty;6.1.26!jetty.zip
[ivy:resolve] ::::::::::::::::::::::::::::::::::::::::::::::
[ivy:resolve]
[ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
[ivy:resolve] io.netty#netty;3.4.0.Final by [io.netty#netty;3.6.2.Final] in [hadoop200test]
[ivy:resolve] asm#asm;[3.0, 4.0) by [asm#asm;3.1] in [hadoop200test]
[ivy:resolve] asm#asm;3.1 by [asm#asm;3.2] in [hadoop200test]
[ivy:resolve] com.google.protobuf#protobuf-java;2.4.1 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200test]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| hadoop200test | 156 | 0 | 0 | 38 || 121 | 0 |
---------------------------------------------------------------------
-- 错误1(如上)解决方法:单独下载 jetty-6.1.26.zip 文件到 /root/.m2/repository/org/mortbay/jetty/jetty/6.1.26/目录下,解决。
------------------------------------------------------------------------------
[ivy:resolve] com.google.protobuf#protobuf-java;2.4.1 by [com.google.protobuf#protobuf-java;2.5.0] in [hadoop200test]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| hadoop200test | 156 | 2 | 2 | 38 || 121 | 2 |
---------------------------------------------------------------------
ivy-retrieve-hadoop-test:
[ivy:retrieve] :: retrieving :: com.cloudera.sqoop#sqoop [sync]
[ivy:retrieve] confs: [hadoop200test]
[ivy:retrieve] 121 artifacts copied, 0 already retrieved (113206kB/376ms)
compile-test:
[mkdir] Created dir: /opt/software/sqoop-1.4.4/build/test/classes
[javac] /opt/software/sqoop-1.4.4/build.xml:433: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 130 source files to /opt/software/sqoop-1.4.4/build/test/classes
[javac] /opt/software/sqoop-1.4.4/src/test/org/apache/sqoop/TestExportUsingProcedure.java:244: repeat(java.lang.String,int) in org.apache.commons.lang.StringUtils cannot be applied to (java.lang.String,java.lang.String,int)
[javac] sql.append(StringUtils.repeat("?", ", ",
[javac] ^
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error
BUILD FAILED
/opt/software/sqoop-1.4.4/build.xml:433: Compile failed; see the compiler error output for details.
Total time: 1 minute 9 seconds
-- 错误2(如上),解决方法:
-------------
-- 第一步:
vi +433 /opt/software/sqoop-1.4.4/build.xml
debug="${javac.debug}">
-- 将第433行修改为如下两行:
debug="${javac.debug}"
includeantruntime="on">
-------------
-- 第二步:vi +244 /opt/software/sqoop-1.4.4/src/test/org/apache/sqoop/TestExportUsingProcedure.java
sql.append(StringUtils.repeat("?", ", ",
-- 将第244行修改为如下:
sql.append(StringUtils.repeat("?,",
-- 继续重新运行 ant package,最后我们将看到:BUILD SUCCESSFUL 字样,表示编译成功。
...
[move] Moving 1 file to /opt/software/sqoop-1.4.4
[mkdir] Created dir: /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/lib
[copy] Copying 10 files to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/lib
[copy] Copying 2 files to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/lib
[copy] Copying 115 files to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/docs
[copy] Copying 14 files to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/docs/man
[copy] Copying 13 files to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/bin
[copy] Copying 1 file to /opt/software/sqoop-1.4.4/build/sqoop-1.4.4.bin__hadoop-2.4.1/conf
BUILD SUCCESSFUL
-- 然后的/opt/software/sqoop-1.4.4/build目录下将生成 sqoop-1.4.4.bin__hadoop-2.4.1的文件夹,这就是我们的安装文件,将其压缩:
cd /opt/software/sqoop-1.4.4/build
tar -cvf sqoop-1.4.4.bin__hadoop-2.4.1.tar.gz ./sqoop-1.4.4.bin__hadoop-2.4.1
sqoop-1.4.4.bin__hadoop-2.4.1.tar.gz文件就是我们需要的sqoop安装包了。
-------------------------------------------
-- 安装很简单:
将sqoop-1.4.4.bin__hadoop-2.4.1.tar.gz解压并cp到/usr/local
tar -xvf sqoop-1.4.4.bin__hadoop-2.4.1.tar.gz
mv ./sqoop-1.4.4.bin__hadoop-2.4.1 /usr/local
chown -R hadoop.hadoop /usr/local/sqoop-1.4.4.bin__hadoop-2.4.1
ln -s /usr/local/sqoop-1.4.4.bin__hadoop-2.4.1 /usr/local/sqoop
vi ~/.bash_profile -- 添加如下两行:
export SQOOP_HOME=/usr/local/sqoop
export PATH=$PATH:${SQOOP_HOME}/bin
source ~/.bash_profile
cd /usr/local/sqoop/conf
cp sqoop-env-template.sh sqoop-env.sh -- 将sqoop-env-template.sh文件复制为sqoop-env.sh
vi sqoop-env.sh -- 编辑刚才复制的文件sqoop-env.sh,添加如下三行(如果没有安装hive的话,将最后一行屏蔽)
export HADOOP_COMMON_HOME=/usr/local/hadoop
export HADOOP_MAPRED_HOME=/usr/local/hadoop/share/hadoop/mapreduce
export HIVE_HOME=/usr/local/hive
-- 编译成功后,测试(看能否访问mysql,测试前记得将mysql的jdbc包放到 $SQOOP_HOME/lib目录下):
-- 我的mysql的jdbc包是:mysql-connector-java-5.1.30-bin.jar
sqoop list-databases --connect jdbc:mysql://funshion-hadoop32:3306/ --username hive --password bee56915
链接:http://pan.baidu.com/s/1sjuJv5r 密码:
此篇文章:由about云群371358502月光蚁族共享304466490
|