大数据架构：flume-ng+Kafka+Storm+HDFS 实时系统组合

大赞楼主，谢谢啦！！

不错 感谢        

赞~               

楼主真真的大神，啥都懂啊

问个问题：
flume-ng+Kafka+Storm 的组合是否可以只用storm来实现，storm有自己的MQ机制（也可以集成JMS等），并且通过写spout来进行数据输入，不知道这样是否可以，而且实现更加简单？

你好啊！版主，我现在用到storm是0.9.3的版本，kafka是kafka_2.10-0.8.1.1的，现在遇到的问题是，环境是搭好的，但是用你的例子走一下，会报错，我知道，可能更新版本后代码也要相对更新，但是不知道怎么弄。一下时报的一场信息，请指点啊。

mjjian0 发表于 2015-3-13 10:21
你好啊！版主，我现在用到storm是0.9.3的版本，kafka是kafka_2.10-0.8.1.1的，现在遇到的问题是，环境是搭 ...

好啦，最终自己解决啦，是snakeyaml的包冲突导致的，最后我重新编译了一下 flume-kafka，storm-kafka的插件包，报了几个异常都是 程序的包的版本与环境里的storm下的包冲突导致的。

谢谢分享~O(∩_∩)O哈哈~

2015-05-07 17:59:51,612 (lifecycleSupervisor-1-1) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@4083633f counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoSuchMethodError: scala.Predef$.augmentString(Ljava/lang/String;)Ljava/lang/String;
        at kafka.utils.VerifiableProperties.getShortInRange(VerifiableProperties.scala:85)
        at kafka.producer.SyncProducerConfigShared$class.$init$(SyncProducerConfig.scala:53)
        at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:52)
        at kafka.producer.ProducerConfig.<init>(ProducerConfig.scala:56)
        at org.apache.flume.plugins.KafkaSink.start(KafkaSink.java:97)
        at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
        at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2015-05-07 17:59:51,615 (lifecycleSupervisor-1-1) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:264)] Unsuccessful attempt to shutdown component: {} due to missing dependencies. Please shutdown the agentor disable this component, or the agent will bein an undefined state.
java.lang.NullPointerException
        at org.apache.flume.plugins.KafkaSink.stop(KafkaSink.java:170)
        at org.apache.flume.sink.DefaultSinkProcessor.stop(DefaultSinkProcessor.java:53)
        at org.apache.flume.SinkRunner.stop(SinkRunner.java:115)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:259)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)


启动flume的时候报错 不知道是什么问题 麻烦大神能够帮帮忙 ！！！flume-1.5.2  kafka_2.11-0.8.2.1

./flume-ng agent -n producer -c ../conf/ -f ../conf/flume-conf-producer.properties -Dflume.root.logger=INFO,console &