flume java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$Compr...

我在A机器上装有flume（A中并不在hadoop集群中），想要上传到装有CDH的B中（B中并没有装flume agent），运行报错如下：
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 12 more



/etc/profile配置如下：
export JAVA_HOME=/usr/lib/jvm/java-8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=${JAVA_HOME}/bin:$PATH
export FLUME_HOME=/panda/spark/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export PATH=.:$PATH::$FLUME_HOME/bin



flume-conf.properties文件如下：
a1.sources = r1
a1.sinks = k1
a1.channels = c1
#描述和配置sources
#第一步：配置数据源
a1.sources.r1.type = exec
a1.sources.r1.channels = c1
#配置需要监控的日志输出目录
a1.sources.r1.command = tail –F /panda/spark/data
#第二步：配置数据输出
a1.sinks.k1.type=hdfs
a1.sinks.k1.channel=c1
a1.sinks.k1.hdfs.useLocalTimeStamp=true
a1.sinks.k1.hdfs.path=hdfs://192.168.0.211:9000/flume/events/%Y/%m/%d/%H/%M
#定义上传到hdfs中文件的前缀
a1.sinks.k1.hdfs.filePrefix=panda-
a1.sinks.k1.hdfs.minBlockReplicas=1
#定义上传之后的文件类型为DataStream，这样能直接查看文件内容,默认为SequenceFile，无法直接查看
a1.sinks.k1.hdfs.fileType =DataStream
a1.sinks.k1.hdfs.writeFormat=Text
a1.sinks.k1.hdfs.rollInterval=60
a1.sinks.k1.hdfs.rollSize=0
a1.sinks.k1.hdfs.rollCount=0
a1.sinks.k1.hdfs.idleTimeout=0
#配置数据通道
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
#第四步：将三者级联
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1



flume.env.sh如下：
FLUME_CLASSPATH="/panda/spark/flume/lib/hadoop-hdfs-2.6.0-cdh5.7.0.jar"
export JAVA_HOME=/usr/lib/jvm/java-8/jre/


hadoop的相关jar已经拷贝到flume/jars中，怎么还是报错呢，求解。。。。

楼主使用的hdfs sink。所以最好在hadoop环境下安装，并且版本要支持。

加上flume jar的路径
FLUME_CLASSPATH="$FLUME_CLASSPATH:/usr/lib/flume-ng/lib/"  

jar包确定都加放进去了

nextuser 发表于 2016-6-27 22:27
加上flume jar的路径
FLUME_CLASSPATH="$FLUME_CLASSPATH:/usr/lib/flume-ng/lib/"

是在flume.env.sh里面加吗？我加了还是不对，报错：
2016-06-29 17:14:29,591 (hdfs-k1-call-runner-0) [WARN - org.apache.hadoop.util.NativeCodeLoader.<clinit>(NativeCodeLoader.java:62)] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-06-29 17:14:30,153 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed
java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder

pandatyut 发表于 2016-6-29 17:15
是在flume.env.sh里面加吗？我加了还是不对，报错：
2016-06-29 17:14:29,591 (hdfs-k1-call-runner-0)  ...

楼主为何会缺hbase包？！

这是hbase的jar包，可以把hbase下lib配置下。

hbase/lib/