为啥 sparksql行而seq join就不行。。

各位大佬：

        DC_DIAGNOSISDF(263 MB)     DC_LAB_RESULT_MASTERDF(899.04 MB)   DC_ENCOUNTERDF(1.75 GB)  DC_VITAL_SIGNSDF(6.43 GB)

     这4个dataframe是我从sql server里拉出来的

当我用这sparksql 进行连接的时候能得到结果

DC_LAB_RESULT_MASTERDF.createOrReplaceTempView("DC_LAB_RESULT_MASTER")

DC_ENCOUNTERDF.createOrReplaceTempView("DC_ENCOUNTER")

DC_VITAL_SIGNSDF.createOrReplaceTempView("DC_VITAL_SIGNS")

DC_DIAGNOSISDF.createOrReplaceTempView("DC_DIAGNOSIS")


val res = spark.sql("select a.*,b.*,c.*,d.* from DC_LAB_RESULT_MASTER a join DC_ENCOUNTER b on a.PATIENT_ID=b.PATIENT_ID join DC_VITAL_SIGNS c on b.PATIENT_ID = c.PATIENT_ID join DC_DIAGNOSIS d on c.PATIENT_ID = d.PATIENT_ID ")   （请问这种是全连接吗？）



但是当我用 以下的方式进行join的时候就一直出错

val joinDF = DC_LAB_RESULT_MASTERDF.join(DC_ENCOUNTERDF, Seq("PERSON_ID","PATIENT_ID"), "full").join(DC_VITAL_SIGNSDF, Seq("PERSON_ID","PATIENT_ID"), "full").join(DC_DIAGNOSISDF, Seq("PERSON_ID","PATIENT_ID"), "full")
    joinDF.take(1)  或者joinDF.count()
异常如下：

      ent-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 2 on hadoop002: Executor heartbeat timed out after 129144 ms
3226 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 1 on hadoop003: Executor heartbeat timed out after 125932 ms
[Stage 13:>                                                      (1 + -1) / 200]68184 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 2 on hadoop002: Container container_1502873852630_0001_01_000003 exited from explicit termination request.
95358 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 1 on hadoop003: Container container_1502873852630_0001_01_000002 exited from explicit termination request.
[Stage 13:>                                                       (1 + 9) / 200]1274319 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 7 on hadoop002: Executor heartbeat timed out after 134390 ms
1304928 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost an executor 7 (already removed): Pending loss reason.
1304983 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost an executor 7 (already removed): Container container_1502873852630_0001_01_000008 exited from explicit termination request.
2004110 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 5 on hadoop002: Executor heartbeat timed out after 410257 ms
2008130 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 8 on hadoop002: Executor heartbeat timed out after 385546 ms
2008134 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 4 on hadoop003: Executor heartbeat timed out after 341075 ms
2008134 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 6 on hadoop002: Executor heartbeat timed out after 336506 ms
2008134 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 9 on hadoop003: Executor heartbeat timed out after 337133 ms
2008134 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 3 on hadoop003: Executor heartbeat timed out after 340119 ms
2014886 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state RUNNING for TID 231 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
2050206 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 4 on hadoop003: Container container_1502873852630_0001_01_000005 exited from explicit termination request.
2050795 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 9 on hadoop003: Container container_1502873852630_0001_01_000010 exited from explicit termination request.
2050798 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 3 on hadoop003: Container container_1502873852630_0001_01_000004 exited from explicit termination request.
2086620 [rpc-server-3-1] ERROR org.apache.spark.network.client.TransportResponseHandler  - Still have 1 requests outstanding when connection from /192.168.217.132:46530 is closed
2087950 [rpc-server-3-1] ERROR org.apache.spark.network.client.TransportResponseHandler  - Still have 1 requests outstanding when connection from /192.168.217.132:46531 is closed
2088591 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 5 on hadoop002: Container container_1502873852630_0001_01_000006 exited from explicit termination request.
2088595 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 6 on hadoop002: Container container_1502873852630_0001_01_000007 exited from explicit termination request.
2093186 [Spark Context Cleaner] ERROR org.apache.spark.ContextCleaner  - Error cleaning broadcast 14
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:152)
        at org.apache.spark.broadcast.TorrentBroadcast$.unpersist(TorrentBroadcast.scala:306)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.unbroadcast(TorrentBroadcastFactory.scala:45)
        at org.apache.spark.broadcast.BroadcastManager.unbroadcast(BroadcastManager.scala:60)
        at org.apache.spark.ContextCleaner.doCleanupBroadcast(ContextCleaner.scala:238)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:194)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1279)
        at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
        at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)
2110418 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 8 on hadoop002: Container container_1502873852630_0001_01_000009 exited from explicit termination request.
[Stage 11:=>                                                      (4 + 3) / 200]3602355 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 12 on hadoop003: Executor heartbeat timed out after 240061 ms
3602506 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 11 on hadoop002: Executor heartbeat timed out after 131830 ms
3602506 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 10 on hadoop003: Executor heartbeat timed out after 135100 ms
3602507 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 13 on hadoop003: Executor heartbeat timed out after 137986 ms
3604536 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FINISHED for TID 243 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
[Stage 11:==>                                                   (10 + -4) / 200]3657938 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 12 on hadoop003: Container container_1502873852630_0001_01_000013 exited from explicit termination request.
3663286 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 11 on hadoop002: Container container_1502873852630_0001_01_000012 exited from explicit termination request.
[Stage 11:==>                                                  (10 + -10) / 200]3681766 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 13 on hadoop003: Container container_1502873852630_0001_01_000014 exited from explicit termination request.
3681818 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 10 on hadoop003: Container container_1502873852630_0001_01_000011 exited from explicit termination request.
[Stage 11:>                                                       (0 + 6) / 200]5443736 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 15 on hadoop002: Executor heartbeat timed out after 199921 ms
5445364 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 18 on hadoop003: Executor heartbeat timed out after 173730 ms
5445819 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 17 on hadoop002: Executor heartbeat timed out after 172596 ms
5445819 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 20 on hadoop003: Executor heartbeat timed out after 189322 ms
5445819 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 14 on hadoop002: Executor heartbeat timed out after 180840 ms
5445819 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 16 on hadoop002: Executor heartbeat timed out after 175181 ms
5445819 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 19 on hadoop003: Executor heartbeat timed out after 187671 ms
[Stage 11:>                                                       (0 + 5) / 200]5446418 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 265 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
[Stage 11:>                                                       (0 + 1) / 200]5448759 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 264 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
[Stage 11:>                                                       (0 + 2) / 200]5458551 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 266 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
[Stage 11:>                                                       (0 + 2) / 200]5481676 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 269 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
5482299 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 268 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
5482301 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Ignoring update with state FAILED for TID 267 because its task set is gone (this is likely the result of receiving duplicate task finished status updates) or its executor has been marked as failed.
5545789 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 18 on hadoop003: Container container_1502873852630_0001_01_000020 exited from explicit termination request.
5567143 [Spark Context Cleaner] ERROR org.apache.spark.ContextCleaner  - Error cleaning broadcast 19
org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [120 seconds]. This timeout is controlled by spark.rpc.askTimeout
        at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$$createRpcTimeoutException(RpcTimeout.scala:47)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:62)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:58)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:76)
        at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:152)
        at org.apache.spark.broadcast.TorrentBroadcast$.unpersist(TorrentBroadcast.scala:306)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.unbroadcast(TorrentBroadcastFactory.scala:45)
        at org.apache.spark.broadcast.BroadcastManager.unbroadcast(BroadcastManager.scala:60)
        at org.apache.spark.ContextCleaner.doCleanupBroadcast(ContextCleaner.scala:238)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:194)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1279)
        at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
        at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
Caused by: java.util.concurrent.TimeoutException: Futures timed out after [120 seconds]
        at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
        at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        ... 12 more
5569449 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 15 on hadoop002: Container container_1502873852630_0001_01_000017 exited from explicit termination request.
5569977 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 19 on hadoop003: Container container_1502873852630_0001_01_000021 exited from explicit termination request.
5570166 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 20 on hadoop003: Container container_1502873852630_0001_01_000022 exited from explicit termination request.
5595547 [rpc-server-3-1] ERROR org.apache.spark.network.client.TransportResponseHandler  - Still have 1 requests outstanding when connection from /192.168.217.132:46563 is closed
5595547 [Spark Context Cleaner] ERROR org.apache.spark.ContextCleaner  - Error cleaning broadcast 18
org.apache.spark.SparkException: Exception thrown in awaitResult:
        at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
        at org.apache.spark.storage.BlockManagerMaster.removeBroadcast(BlockManagerMaster.scala:152)
        at org.apache.spark.broadcast.TorrentBroadcast$.unpersist(TorrentBroadcast.scala:306)
        at org.apache.spark.broadcast.TorrentBroadcastFactory.unbroadcast(TorrentBroadcastFactory.scala:45)
        at org.apache.spark.broadcast.BroadcastManager.unbroadcast(BroadcastManager.scala:60)
        at org.apache.spark.ContextCleaner.doCleanupBroadcast(ContextCleaner.scala:238)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:194)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1$$anonfun$apply$mcV$sp$1.apply(ContextCleaner.scala:185)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.ContextCleaner$$anonfun$org$apache$spark$ContextCleaner$$keepCleaning$1.apply$mcV$sp(ContextCleaner.scala:185)
        at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1279)
        at org.apache.spark.ContextCleaner.org$apache$spark$ContextCleaner$$keepCleaning(ContextCleaner.scala:178)
        at org.apache.spark.ContextCleaner$$anon$1.run(ContextCleaner.scala:73)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:221)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:899)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:275)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)
5595822 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 14 on hadoop002: Container container_1502873852630_0001_01_000016 exited from explicit termination request.
5598381 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 16 on hadoop002: Container container_1502873852630_0001_01_000018 exited from explicit termination request.
5600624 [dispatcher-event-loop-0] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 17 on hadoop002: Container container_1502873852630_0001_01_000019 exited from explicit termination request.
6194318 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 22 on hadoop003: Executor heartbeat timed out after 135915 ms
6195095 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.TaskSetManager  - Task 0 in stage 12.1 failed 4 times; aborting job
6251172 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 22 on hadoop003: Container container_1502873852630_0001_01_000024 exited from explicit termination request.
6254319 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 21 on hadoop003: Executor heartbeat timed out after 154039 ms
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 12.1 failed 4 times, most recent failure: Lost task 0.4 in stage 12.1 (TID 278, hadoop003, executor 22): ExecutorLostFailure (executor 22 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 135915 ms
Driver stacktrace:
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1499)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1487)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1486)
  at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1486)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:814)
  at scala.Option.foreach(Option.scala:257)
  at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:814)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1714)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
  at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
  at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
  at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:630)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2022)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2043)
  at org.apache.spark.SparkContext.runJob(SparkContext.scala:2062)
  at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:336)
  at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
  at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collectFromPlan(Dataset.scala:2853)
  at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2153)
  at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2153)
  at org.apache.spark.sql.Dataset$$anonfun$55.apply(Dataset.scala:2837)
  at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:2836)
  at org.apache.spark.sql.Dataset.head(Dataset.scala:2153)
  at org.apache.spark.sql.Dataset.take(Dataset.scala:2366)
  ... 48 elided

scala> 6266822 [dispatcher-event-loop-1] ERROR org.apache.spark.scheduler.cluster.YarnScheduler  - Lost executor 21 on hadoop003: Container container_1502873852630_0001_01_000023 exited from explicit termination request.

没见过这个形式的。楼主是从那地方看到的。
val joinDF = DC_LAB_RESULT_MASTERDF.join(DC_ENCOUNTERDF, Seq("PERSON_ID","PATIENT_ID"), "full").join(DC_VITAL_SIGNSDF, Seq("PERSON_ID","PATIENT_ID"), "full").join(DC_DIAGNOSISDF, Seq("PERSON_ID","PATIENT_ID"), "full")
一般见到下面的形式，楼主试试下面方式的
[mw_shl_code=sql,true]val joinDF = DC_LAB_RESULT_MASTERDF.join(DC_ENCOUNTERDF, Seq("PERSON_ID","PATIENT_ID")).join(DC_VITAL_SIGNSDF, Seq("PERSON_ID","PATIENT_ID")).join(DC_DIAGNOSISDF, Seq("PERSON_ID","PATIENT_ID"))[/mw_shl_code]

einhep 发表于 2017-8-17 14:20
没见过这个形式的。楼主是从那地方看到的。
val joinDF = DC_LAB_RESULT_MASTERDF.join(DC_ENCOUNTERDF, S ...

你这个是默认的内连接，  我加了 full是全连接。。

http://blog.csdn.net/anjingwunai/article/details/51934921

参考这个。

remarkzhao 发表于 2017-8-17 14:45
你这个是默认的内连接，  我加了 full是全连接。。

http://blog.csdn.net/anjingwunai/article/detail ...

join的话，可能内存要求比较高。增加下jvm和内存

一直没弄明白 jvm跟内存怎么加？？
jvm：spark.executor.extraJavaOptions=-XX:MaxPermSize=1024M？？？

内存： driver-memory？ 还是 executor-memory 2G？？

remarkzhao 发表于 2017-8-17 15:53
一直没弄明白 jvm跟内存怎么加？？
jvm：spark.executor.extraJavaOptions=-XX:MaxPermSize=1024M？？？
...

对的，并且jvm不能超过memory