图文精华

About云-梭伦科技»专题 › 交流区 › 技术交流 › Spark › 通过eclipse 提交sc.textFile("XX").count() 报错 ...
分享

通过eclipse 提交sc.textFile("XX").count() 报错

taoxw 发表于 2017-7-31 17:09:09 [显示全部楼层] 回帖奖励 阅读模式 关闭右栏 11 8738
About云VIP帮助找工作和提升
2017-07-31 16:28:08,364 INFO org.apache.spark.executor.CoarseGrainedExecutorBackend: Got assigned task 5
2017-07-31 16:28:08,365 ERROR org.apache.spark.executor.Executor: Exception in task 0.2 in stage 0.0 (TID 4)
java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,365 INFO org.apache.spark.executor.Executor: Running task 1.2 in stage 0.0 (TID 5)
2017-07-31 16:28:08,369 ERROR org.apache.spark.util.Utils: Exception encountered
java.lang.IllegalArgumentException: Can not create a Path from an empty string
        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
        at org.apache.hadoop.fs.Path.<init>(Path.java:135)
        at org.apache.hadoop.mapreduce.lib.input.FileSplit.readFields(FileSplit.java:115)
        at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:103)
        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
        at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:45)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1206)
        at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:41)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,370 ERROR org.apache.spark.executor.Executor: Exception in task 1.2 in stage 0.0 (TID 5)
java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,381 INFO org.apache.spark.executor.CoarseGrainedExecutorBackend: Got assigned task 6
2017-07-31 16:28:08,381 INFO org.apache.spark.executor.Executor: Running task 0.3 in stage 0.0 (TID 6)
2017-07-31 16:28:08,385 ERROR org.apache.spark.util.Utils: Exception encountered
java.lang.IllegalArgumentException: Can not create a Path from an empty string
        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
        at org.apache.hadoop.fs.Path.<init>(Path.java:135)
        at org.apache.hadoop.mapreduce.lib.input.FileSplit.readFields(FileSplit.java:115)
        at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:103)
        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
        at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:45)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1206)
        at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:41)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,385 INFO org.apache.spark.executor.CoarseGrainedExecutorBackend: Got assigned task 7
2017-07-31 16:28:08,386 ERROR org.apache.spark.executor.Executor: Exception in task 0.3 in stage 0.0 (TID 6)
java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,386 INFO org.apache.spark.executor.Executor: Running task 1.3 in stage 0.0 (TID 7)
2017-07-31 16:28:08,390 ERROR org.apache.spark.util.Utils: Exception encountered
java.lang.IllegalArgumentException: Can not create a Path from an empty string
        at org.apache.hadoop.fs.Path.checkPathArg(Path.java:127)
        at org.apache.hadoop.fs.Path.<init>(Path.java:135)
        at org.apache.hadoop.mapreduce.lib.input.FileSplit.readFields(FileSplit.java:115)
        at org.apache.hadoop.mapred.FileSplit.readFields(FileSplit.java:103)
        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
        at org.apache.hadoop.io.ObjectWritable.readFields(ObjectWritable.java:77)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply$mcV$sp(SerializableWritable.scala:45)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.SerializableWritable$$anonfun$readObject$1.apply(SerializableWritable.scala:41)
        at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1206)
        at org.apache.spark.SerializableWritable.readObject(SerializableWritable.scala:41)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,391 ERROR org.apache.spark.executor.Executor: Exception in task 1.3 in stage 0.0 (TID 7)
java.lang.IllegalStateException: unread block data
        at java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
        at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
        at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:222)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2017-07-31 16:28:08,582 INFO org.apache.spark.executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
2017-07-31 16:28:08,585 INFO org.apache.spark.deploy.worker.Worker: Asked to kill executor app-20170731162802-0104/0
2017-07-31 16:28:08,585 INFO org.apache.spark.deploy.worker.ExecutorRunner: Runner thread for executor app-20170731162802-0104/0 interrupted
2017-07-31 16:28:08,585 INFO org.apache.spark.deploy.worker.ExecutorRunner: Killing process!
2017-07-31 16:28:08,586 ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL 15: SIGTERM
2017-07-31 16:28:08,590 INFO org.apache.spark.util.ShutdownHookManager: Shutdown hook called
2017-07-31 16:28:08,592 INFO org.apache.spark.util.ShutdownHookManager: Deleting directory /tmp/spark-2956dc1f-d385-4b42-8888-2f74f39c2411/executor-cb187d1e-4a0e-4199-bd8e-5df83cf1d5b6/spark-894f864f-d796-487e-a891-90e6a8ebadde








使用spark-shell运行不报错，我这里用的是 CDH版本的，请大神们帮忙看看






回复

使用道具 举报

自定义广告语

已有(11)人评论

跳转到指定楼层
正序浏览
langke93 发表于 2017-7-31 18:18:28
java.lang.IllegalArgumentException: Can not create a Path from an empty string
没有看到你的代码，但是应该是路径为空，没有读取到

回复

使用道具 举报

taoxw 发表于 2017-8-1 08:11:41
多谢楼上的，我也看到这个错误信息了，但是同样的代码在spark-shell里面就可以执行成功，我在IDE里面就不行，不知道是不是要设置什么参数？
回复

使用道具 举报

starrycheng 发表于 2017-8-1 08:15:41
taoxw 发表于 2017-8-1 08:11
多谢楼上的，我也看到这个错误信息了，但是同样的代码在spark-shell里面就可以执行成功，我在IDE里面就不行 ...

应该跟这帖子是一样的问题
提交spark执行scala程序报错unread block data ，求指教
http://www.aboutyun.com/forum.php?mod=viewthread&tid=22277

回复

使用道具 举报

taoxw 发表于 2017-8-1 08:22:51
谢谢楼上的，刚刚看了你提供的帖子，我加大了参数，但还是不行，还是报同样的错
回复

使用道具 举报

starrycheng 发表于 2017-8-1 08:24:56
taoxw 发表于 2017-8-1 08:22
谢谢楼上的，刚刚看了你提供的帖子，我加大了参数，但还是不行，还是报同样的错

修改还需要生效，否则可能读取不到。
回复

使用道具 举报

nextuser 发表于 2017-8-1 09:15:32
本地开发的版本与集群版本是否一致
回复

使用道具 举报

taoxw 发表于 2017-8-1 09:17:37
nextuser 发表于 2017-8-1 09:15
本地开发的版本与集群版本是否一致

一致的，不使用sc.textFile("XXX").count() 加载文件，只是其他的spark 编程就可以，一旦用了textFile()就报上面这个错
回复

使用道具 举报

nextuser 发表于 2017-8-1 09:28:26
taoxw 发表于 2017-8-1 09:17
一致的，不使用sc.textFile("XXX").count() 加载文件，只是其他的spark 编程就可以，一旦用了textFile() ...

把路径打印出来，看看是什么情况
回复

使用道具 举报

taoxw 发表于 2017-8-1 09:36:18
nextuser 发表于 2017-8-1 09:28
把路径打印出来，看看是什么情况

我的操作很简单的
val df = sc.textFile("hdfs://192.168.8.210:8020/a.txt")
    println(df.count())

回复

使用道具 举报

12下一页
高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
taoxw

新手上路

关注

  • 1

    主题

  • 6

    帖子

  • 3

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000