请教一下为什么flume每次都是运行一会就停了 - About云-梭伦科技

上传

点击文件名下载附件

作者: Hentai 时间: 2016-12-26 16:16
不管我是不是用excu source都是这样运行一会就停了

作者: Hentai 时间: 2016-12-26 16:46
本帖最后由 Hentai 于 2016-12-26 16:53 编辑

就算把memory调小都没有用！一样过了一会就自动停止了

作者: langke93 时间: 2016-12-26 19:24

Hentai 发表于 2016-12-26 16:46
就算把memory调小都没有用！一样过了一会就自动停止了

没遇到过，楼主需提供下面信息
1.用的哪个版本
2.去看下日志
3.具体如何配置的

作者: Hentai 时间: 2016-12-27 09:24

langke93 发表于 2016-12-26 19:24
没遇到过，楼主需提供下面信息
1.用的哪个版本
2.去看下日志

配置
hdfs.sinks.fs.channel=fc
hdfs.sources.fr.channels=fc

hdfs.sources.fr.type=exec
hdfs.sources.fr.command=tshark -i em2 -t ad

hdfs.sinks.fs.type=hdfs
hdfs.sinks.fs.hdfs.path=hdfs://10.0.40.2:8020/user/xiakang
#hdfs.sinks.fs.hdfs.filePrefix=Syslog
hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
hdfs.sinks.fs.hdfs.rollCount=0
hdfs.sinks.fs.hdfs.fileType=DataStream
hdfs.sinks.fs.hdfs.minBlockReplicas=1
hdfs.sinks.fs.hdfs.inUseSuffix=log

hdfs.channels.fc.type=SPILLABLEMEMORY
hdfs.channels.fc.checkpointDir = /mnt/flume/checkpoint
hdfs.channels.fc.dataDirs = /mnt/flume/data
版本 Flume 1.5.2.2.5.0.0-1245

错误
Failed while running command: tshark -i em2 -t ad
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at org.apache.flume.source.ExecSource$ExecRunnable.run(ExecSource.java:331)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
26 Dec 2016 16:23:41,039 INFO  [pool-5-thread-1] (org.apache.flume.source.ExecSource$ExecRu
nnable.run:370)  - Command [tshark -i em2 -t ad] exited with 141
26 Dec 2016 16:23:41,040 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:149)  - Component type: SOURCE, name: fr stopped
26 Dec 2016 16:23:41,040 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:155)  - Shutdown Metric for type: SOURCE, name: fr. source.start.tim
e == 1482740567800
26 Dec 2016 16:23:41,040 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:161)  - Shutdown Metric for type: SOURCE, name: fr. source.stop.time
== 1482740621040
26 Dec 2016 16:23:41,041 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.append-batch
.accepted == 0
26 Dec 2016 16:23:41,041 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.append-batch
.received == 0
26 Dec 2016 16:23:41,041 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.append.accep
ted == 0
26 Dec 2016 16:23:41,041 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.append.recei
ved == 0
26 Dec 2016 16:23:41,041 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.events.accep
ted == 514320
26 Dec 2016 16:23:41,042 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.events.recei
ved == 514334
26 Dec 2016 16:23:41,042 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.Moni
toredCounterGroup.stop:177)  - Shutdown Metric for type: SOURCE, name: fr. src.open-connect
ion.count == 0
26 Dec 2016 16:23:41,042 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.
flume.sink.hdfs.HDFSEventSink.process:466)  - process failed
java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.callTimeout
might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket
Writer.java:687)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
26 Dec 2016 16:23:41,043 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.
flume.SinkRunner$PollingRunner.run:160)  - Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: java.lang.InterruptedException: Timed out before H
DFS call was made. Your hdfs.callTimeout might be set too low or HDFS calls are taking too
long.
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:470)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.InterruptedException: Timed out before HDFS call was made. Your hdfs.c
allTimeout might be set too low or HDFS calls are taking too long.
at org.apache.flume.sink.hdfs.BucketWriter.checkAndThrowInterruptedException(Bucket
Writer.java:687)
at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:446)
at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:448)
... 3 more

作者: yaojiank 时间: 2016-12-27 09:54

Hentai 发表于 2016-12-27 09:24
配置
hdfs.sinks.fs.channel=fc
hdfs.sources.fr.channels=fc

hdfs.sinks.fs.hdfs.rollInterval=21600
hdfs.sinks.fs.hdfs.rollSize=8589934592
为何这两个设置这么大。超时出现问题正常的设置小一点，根据情况自己调试下。
建议初始值：
hdfs.sinks.fs.hdfs.rollInterval=10
hdfs.sinks.fs.hdfs.rollSize=0
依次：
20
40
80
200
等值。

欢迎光临 About云-梭伦科技 (https://www.aboutyun.com/)