求助 flume + kafka 异常 java.nio.BufferUnderflowException

查看数: 21379 | 评论数: 2 | 收藏 0
关灯 | 提示:支持键盘翻页<-左 右->
    组图打开中,请稍候......
发布时间: 2016-6-12 11:11

正文摘要:

小弟我这边搭建flume+kafka,将flume接收到的数据sink到kafka中,flume报异常,而kafka的消费端则不停的循环收到相同的数据。我用的flume版本是apache-flume-1.6.0-cdh5.7.0 异常截图: 大家知道这是什么 ...

回复

nextuser 发表于 2016-6-12 15:37:52
楼主应该缺少一些参数,这里摘自官网,楼主可以参考下:

Kafka Source


[mw_shl_code=bash,true]tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource
tier1.sources.source1.channels = channel1
tier1.sources.source1.zookeeperConnect = localhost:2181
tier1.sources.source1.topic = test1
tier1.sources.source1.groupId = flume
tier1.sources.source1.kafka.consumer.timeout.ms = 100[/mw_shl_code]

Kafka Sink
[mw_shl_code=bash,true]a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = mytopic
a1.sinks.k1.brokerList = localhost:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1[/mw_shl_code]

Kafka Channel
[mw_shl_code=bash,true]a1.channels.channel1.type   = org.apache.flume.channel.kafka.KafkaChannel
a1.channels.channel1.capacity = 10000
a1.channels.channel1.transactionCapacity = 1000
a1.channels.channel1.brokerList=kafka-2:9092,kafka-3:9092
a1.channels.channel1.topic=channel1
a1.channels.channel1.zookeeperConnect=kafka-1:2181[/mw_shl_code]







starrycheng 发表于 2016-6-12 14:49:13
错误应该还有的,提供的不够完整,需要提供完整一些。配置也是。

关闭

推荐上一条 /2 下一条