本帖最后由 邓立辉 于 2016-3-7 17:43 编辑
报如下错误
16/03/07 14:59:27 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 2.2.2.30:53524 (size: 2.0 KB, free: 1060.0 MB)
16/03/07 14:59:27 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1457333965500 ms, 1457333966000 ms)
16/03/07 14:59:27 INFO InputInfoTracker: remove old batch metadata: 1457333965500 ms 1457333966000 ms
16/03/07 14:59:27 INFO BlockManagerInfo: Removed input-0-1457333966000 on 2.2.2.32:48519 in memory (size: 1024.0 B, free: 1060.0 MB)
16/03/07 14:59:27 INFO BlockManagerInfo: Removed input-0-1457333966000 on 2.2.2.30:53524 in memory (size: 1024.0 B, free: 1060.0 MB)
16/03/07 14:59:27 WARN TaskSetManager: Lost task 0.0 in stage 6.0 (TID 74, 2.2.2.30): java.lang.Exception: Could not compute split, block input-0-1457333966000 not found
at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:297)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:264)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
看日志是先删掉 Removed input-0-1457333966000。 然后有去读这个了, block input-0-1457333966000 not found。
这是为什么呢?我把集群的服务器时间都同步了一下,精确到2秒内了。
我看spark源码是InputInfoTracker的cleanup给删掉的。是不是要设置下这个参数"spark.streaming.internal.batchTime",我没查到这个参数是干什么的。
|