分享

Spark0.9分布式运行MLlib的线性回归算法

问题导读
1、什么是线性回归算法?
2、可用线性回归算法有哪些?
3、如何对拟合优度进行评估?







1 什么是线性回归

线性回归是另一个传统的有监督机器学习算法。在这个问题中,每个实体与一个实数值的标签 (而不是一个像在二元分类的0,1标签),和我们想要预测标签尽可能给出数值代表实体特征。MLlib支持线性回归以及L2(ridge)和L1(lasso)正则化参数调整。Mllib还有一个回归算法,原始梯度下降(在下面描述),和上面描述的有相同的参数二元分类算法。

可用线性回归算法:

  • LinearRegressionWithSGD
  • RidgeRegressionWithSGD
  • LassoWithSGD。


注意:

(1)因为是线性回归,所以学习到的函数为线性函数,即直线函数;

(2)因为是单变量,因此只有一个x;

我们能够给出单变量线性回归的模型:
1.png


我们常称x为feature,h(x)为hypothesis;


2. Gradient Descent(梯度下降)
但是又一个问题引出了,虽然给定一个函数,我们能够根据cost function知道这个函数拟合的好不好,但是毕竟函数有这么多,总不可能一个一个试吧?

因此我们引出了梯度下降:能够找出cost function函数的最小值;

梯度下降原理:将函数比作一座山,我们站在某个山坡上,往四周看,从哪个方向向下走一小步,能够下降的最快;

当然解决问题的方法有很多,梯度下降只是其中一个,还有一种方法叫Normal Equation;

方法:
(1)先确定向下一步的步伐大小,我们称为Learning rate;
(2)任意给定一个初始值:;
(3)确定一个向下的方向,并向下走预先规定的步伐,并更新;
(4)当下降的高度小于某个定义的值,则停止下降;

算法:
1.png



特点:
(1)初始点不同,获得的最小值也不同,因此梯度下降求得的只是局部最小值;
(2)越接近最小值时,下降速度越慢;


3.线性回归代码

下面的示例演示如何加载训练数据,把它解析成 LabeledPoint的RDD对象(弹性分布式数据集)。这个例子然后使用LinearRegressionWithSGD构建一个简单的线性模型来预测标签值。最后我们计算均方误差对拟合优度进行评估。

执行结果如下:
  1. root@master scala]# sbt/sbt package run
  2. [info] Set current project to scala (in build file:/root/sample/scala/)
  3. [success] Total time: 2 s, completed Feb 17, 2014 9:53:53 PM
  4. [info] Running SimpleApp
  5. log4j:WARN No appenders could be found for logger (akka.event.slf4j.Slf4jLogger).
  6. log4j:WARN Please initialize the log4j system properly.
  7. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  8. 14/02/17 21:53:55 INFO SparkEnv: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
  9. 14/02/17 21:53:55 INFO SparkEnv: Registering BlockManagerMaster
  10. 14/02/17 21:53:55 INFO DiskBlockManager: Created local directory at /tmp/spark-local-20140217215355-b441
  11. 14/02/17 21:53:55 INFO MemoryStore: MemoryStore started with capacity 580.0 MB.
  12. 14/02/17 21:53:55 INFO ConnectionManager: Bound socket to port 45162 with id = ConnectionManagerId(master,45162)
  13. 14/02/17 21:53:55 INFO BlockManagerMaster: Trying to register BlockManager
  14. 14/02/17 21:53:55 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager master:45162 with 580.0 MB RAM
  15. 14/02/17 21:53:55 INFO BlockManagerMaster: Registered BlockManager
  16. 14/02/17 21:53:55 INFO HttpServer: Starting HTTP Server
  17. 14/02/17 21:53:56 INFO HttpBroadcast: Broadcast server started at http://192.168.159.129:54817
  18. 14/02/17 21:53:56 INFO SparkEnv: Registering MapOutputTracker
  19. 14/02/17 21:53:56 INFO HttpFileServer: HTTP File server directory is /tmp/spark-b1b6ca47-4f04-4a60-8cb5-4b7151c6e9a2
  20. 14/02/17 21:53:56 INFO HttpServer: Starting HTTP Server
  21. 14/02/17 21:53:56 INFO SparkUI: Started Spark Web UI at http://master:4040
  22. 14/02/17 21:53:57 WARN NativeCodeLoader: Unable to load native-Hadoop library for your platform... using builtin-java classes where applicable
  23. 14/02/17 21:53:57 INFO SparkContext: Added JAR target/scala-2.10/scala_2.10-0.1-SNAPSHOT.jar at http://192.168.159.129:47898/jars/scala_2.10-0.1-SNAPSHOT.jar with timestamp 1392645237384
  24. 14/02/17 21:53:57 INFO AppClient$ClientActor: Connecting to master spark://192.168.159.129:7077...
  25. 14/02/17 21:53:58 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes
  26. 14/02/17 21:53:58 INFO MemoryStore: ensureFreeSpace(132636) called with curMem=0, maxMem=608187187
  27. 14/02/17 21:53:58 INFO MemoryStore: Block broadcast_0 stored as values to memory (estimated size 129.5 KB, free 579.9 MB)
  28. 14/02/17 21:53:59 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20140217215359-0003
  29. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor added: app-20140217215359-0003/0 on worker-20140217214342-master-54909 (master:54909) with 1 cores
  30. 14/02/17 21:53:59 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140217215359-0003/0 on hostPort master:54909 with 1 cores, 512.0 MB RAM
  31. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor added: app-20140217215359-0003/1 on worker-20140217214339-slaver02-52414 (slaver02:52414) with 1 cores
  32. 14/02/17 21:53:59 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140217215359-0003/1 on hostPort slaver02:52414 with 1 cores, 512.0 MB RAM
  33. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor added: app-20140217215359-0003/2 on worker-20140217214341-slaver01-34119 (slaver01:34119) with 1 cores
  34. 14/02/17 21:53:59 INFO SparkDeploySchedulerBackend: Granted executor ID app-20140217215359-0003/2 on hostPort slaver01:34119 with 1 cores, 512.0 MB RAM
  35. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor updated: app-20140217215359-0003/1 is now RUNNING
  36. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor updated: app-20140217215359-0003/2 is now RUNNING
  37. 14/02/17 21:53:59 INFO AppClient$ClientActor: Executor updated: app-20140217215359-0003/0 is now RUNNING
  38. 14/02/17 21:54:02 INFO SparkDeploySchedulerBackend: Registered executor: Actor[akka.tcp://sparkExecutor@slaver02:42172/user/Executor#1991133218] with ID 1
  39. 14/02/17 21:54:03 INFO FileInputFormat: Total input paths to process : 1
  40. 14/02/17 21:54:03 INFO SparkContext: Starting job: first at GeneralizedLinearAlgorithm.scala:121
  41. 14/02/17 21:54:03 INFO DAGScheduler: Got job 0 (first at GeneralizedLinearAlgorithm.scala:121) with 1 output partitions (allowLocal=true)
  42. 14/02/17 21:54:03 INFO DAGScheduler: Final stage: Stage 0 (first at GeneralizedLinearAlgorithm.scala:121)
  43. 14/02/17 21:54:03 INFO DAGScheduler: Parents of final stage: List()
  44. 14/02/17 21:54:03 INFO DAGScheduler: Missing parents: List()
  45. 14/02/17 21:54:03 INFO DAGScheduler: Computing the requested partition locally
  46. 14/02/17 21:54:03 INFO HadoopRDD: Input split: hdfs://master:9000/mllib/lpsa.data:0+5197
  47. 14/02/17 21:54:03 INFO BlockManagerMasterActor$BlockManagerInfo: Registering block manager slaver02:33696 with 297.0 MB RAM
  48. 14/02/17 21:54:04 INFO SparkContext: Job finished: first at GeneralizedLinearAlgorithm.scala:121, took 0.703681387 s
  49. 14/02/17 21:54:04 INFO SparkContext: Starting job: count at GradientDescent.scala:137
  50. 14/02/17 21:54:04 INFO DAGScheduler: Got job 1 (count at GradientDescent.scala:137) with 2 output partitions (allowLocal=false)
  51. 14/02/17 21:54:04 INFO DAGScheduler: Final stage: Stage 1 (count at GradientDescent.scala:137)
  52. 14/02/17 21:54:04 INFO DAGScheduler: Parents of final stage: List()
  53. 14/02/17 21:54:04 INFO DAGScheduler: Missing parents: List()
  54. 14/02/17 21:54:04 INFO DAGScheduler: Submitting Stage 1 (MappedRDD[3] at map at GeneralizedLinearAlgorithm.scala:139), which has no missing parents
  55. 14/02/17 21:54:04 INFO DAGScheduler: Submitting 2 missing tasks from Stage 1 (MappedRDD[3] at map at GeneralizedLinearAlgorithm.scala:139)
  56. 14/02/17 21:54:04 INFO TaskSchedulerImpl: Adding task set 1.0 with 2 tasks
  57. 14/02/17 21:54:04 INFO TaskSetManager: Starting task 1.0:0 as TID 0 on executor 1: slaver02 (NODE_LOCAL)
  58. 14/02/17 21:54:04 INFO TaskSetManager: Serialized task 1.0:0 as 1749 bytes in 24 ms
  59. 14/02/17 21:54:32 INFO DAGScheduler: Got job 14 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  60. 14/02/17 21:54:32 INFO DAGScheduler: Final stage: Stage 14 (reduce at GradientDescent.scala:150)
  61. 14/02/17 21:54:32 INFO DAGScheduler: Parents of final stage: List()
  62. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  63. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 14 (MappedRDD[29] at map at GradientDescent.scala:145), which has no missing parents
  64. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 14 (MappedRDD[29] at map at GradientDescent.scala:145)
  65. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 14.0 with 2 tasks
  66. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 14.0:0 as TID 26 on executor 2: slaver01 (NODE_LOCAL)
  67. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 14.0:0 as 2419 bytes in 0 ms
  68. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 14.0:1 as TID 27 on executor 1: slaver02 (NODE_LOCAL)
  69. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 14.0:1 as 2419 bytes in 0 ms
  70. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 26 in 35 ms on slaver01 (progress: 0/2)
  71. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(14, 0)
  72. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 27 in 85 ms on slaver02 (progress: 1/2)
  73. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 14.0 from pool
  74. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(14, 1)
  75. 14/02/17 21:54:33 INFO DAGScheduler: Stage 14 (reduce at GradientDescent.scala:150) finished in 0.082 s
  76. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.098827167 s
  77. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  78. 14/02/17 21:54:33 INFO DAGScheduler: Got job 15 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  79. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 15 (reduce at GradientDescent.scala:150)
  80. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  81. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  82. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 15 (MappedRDD[31] at map at GradientDescent.scala:145), which has no missing parents
  83. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 15 (MappedRDD[31] at map at GradientDescent.scala:145)
  84. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 15.0 with 2 tasks
  85. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 15.0:0 as TID 28 on executor 2: slaver01 (NODE_LOCAL)
  86. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 15.0:0 as 2421 bytes in 1 ms
  87. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 15.0:1 as TID 29 on executor 1: slaver02 (NODE_LOCAL)
  88. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 15.0:1 as 2421 bytes in 0 ms
  89. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 28 in 68 ms on slaver01 (progress: 0/2)
  90. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(15, 0)
  91. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 29 in 80 ms on slaver02 (progress: 1/2)
  92. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 15.0 from pool
  93. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(15, 1)
  94. 14/02/17 21:54:33 INFO DAGScheduler: Stage 15 (reduce at GradientDescent.scala:150) finished in 0.082 s
  95. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.317842176 s
  96. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  97. 14/02/17 21:54:33 INFO DAGScheduler: Got job 16 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  98. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 16 (reduce at GradientDescent.scala:150)
  99. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  100. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  101. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 16 (MappedRDD[33] at map at GradientDescent.scala:145), which has no missing parents
  102. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 16 (MappedRDD[33] at map at GradientDescent.scala:145)
  103. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 16.0 with 2 tasks
  104. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 16.0:0 as TID 30 on executor 2: slaver01 (NODE_LOCAL)
  105. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 16.0:0 as 2420 bytes in 0 ms
  106. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 16.0:1 as TID 31 on executor 1: slaver02 (NODE_LOCAL)
  107. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 16.0:1 as 2420 bytes in 0 ms
  108. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 31 in 52 ms on slaver02 (progress: 0/2)
  109. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(16, 1)
  110. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 30 in 60 ms on slaver01 (progress: 1/2)
  111. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 16.0 from pool
  112. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(16, 0)
  113. 14/02/17 21:54:33 INFO DAGScheduler: Stage 16 (reduce at GradientDescent.scala:150) finished in 0.050 s
  114. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.071822529 s
  115. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  116. 14/02/17 21:54:33 INFO DAGScheduler: Got job 17 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  117. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 17 (reduce at GradientDescent.scala:150)
  118. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  119. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  120. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 17 (MappedRDD[35] at map at GradientDescent.scala:145), which has no missing parents
  121. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 17 (MappedRDD[35] at map at GradientDescent.scala:145)
  122. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 17.0 with 2 tasks
  123. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 17.0:0 as TID 32 on executor 2: slaver01 (NODE_LOCAL)
  124. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 17.0:0 as 2417 bytes in 0 ms
  125. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 17.0:1 as TID 33 on executor 1: slaver02 (NODE_LOCAL)
  126. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 17.0:1 as 2417 bytes in 1 ms
  127. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 33 in 45 ms on slaver02 (progress: 0/2)
  128. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(17, 1)
  129. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 32 in 58 ms on slaver01 (progress: 1/2)
  130. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 17.0 from pool
  131. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(17, 0)
  132. 14/02/17 21:54:33 INFO DAGScheduler: Stage 17 (reduce at GradientDescent.scala:150) finished in 0.055 s
  133. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.067749084 s
  134. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  135. 14/02/17 21:54:33 INFO DAGScheduler: Got job 18 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  136. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 18 (reduce at GradientDescent.scala:150)
  137. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  138. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  139. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 18 (MappedRDD[37] at map at GradientDescent.scala:145), which has no missing parents
  140. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 18 (MappedRDD[37] at map at GradientDescent.scala:145)
  141. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 18.0 with 2 tasks
  142. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 18.0:0 as TID 34 on executor 2: slaver01 (NODE_LOCAL)
  143. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 18.0:0 as 2419 bytes in 0 ms
  144. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 18.0:1 as TID 35 on executor 1: slaver02 (NODE_LOCAL)
  145. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 18.0:1 as 2419 bytes in 0 ms
  146. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 35 in 40 ms on slaver02 (progress: 0/2)
  147. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(18, 1)
  148. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 34 in 97 ms on slaver01 (progress: 1/2)
  149. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 18.0 from pool
  150. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(18, 0)
  151. 14/02/17 21:54:33 INFO DAGScheduler: Stage 18 (reduce at GradientDescent.scala:150) finished in 0.092 s
  152. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.105965063 s
  153. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  154. 14/02/17 21:54:33 INFO DAGScheduler: Got job 19 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  155. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 19 (reduce at GradientDescent.scala:150)
  156. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  157. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  158. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 19 (MappedRDD[39] at map at GradientDescent.scala:145), which has no missing parents
  159. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 19 (MappedRDD[39] at map at GradientDescent.scala:145)
  160. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 19.0 with 2 tasks
  161. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 19.0:0 as TID 36 on executor 2: slaver01 (NODE_LOCAL)
  162. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 19.0:0 as 2418 bytes in 1 ms
  163. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 19.0:1 as TID 37 on executor 1: slaver02 (NODE_LOCAL)
  164. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 19.0:1 as 2418 bytes in 0 ms
  165. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 36 in 39 ms on slaver01 (progress: 0/2)
  166. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(19, 0)
  167. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 37 in 52 ms on slaver02 (progress: 1/2)
  168. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 19.0 from pool
  169. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(19, 1)
  170. 14/02/17 21:54:33 INFO DAGScheduler: Stage 19 (reduce at GradientDescent.scala:150) finished in 0.042 s
  171. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.060941515 s
  172. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  173. 14/02/17 21:54:33 INFO DAGScheduler: Got job 20 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  174. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 20 (reduce at GradientDescent.scala:150)
  175. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  176. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  177. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 20 (MappedRDD[41] at map at GradientDescent.scala:145), which has no missing parents
  178. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 20 (MappedRDD[41] at map at GradientDescent.scala:145)
  179. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 20.0 with 2 tasks
  180. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 20.0:0 as TID 38 on executor 2: slaver01 (NODE_LOCAL)
  181. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 20.0:0 as 2418 bytes in 0 ms
  182. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 20.0:1 as TID 39 on executor 1: slaver02 (NODE_LOCAL)
  183. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 20.0:1 as 2418 bytes in 0 ms
  184. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 38 in 33 ms on slaver01 (progress: 0/2)
  185. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(20, 0)
  186. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 39 in 71 ms on slaver02 (progress: 1/2)
  187. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(20, 1)
  188. 14/02/17 21:54:33 INFO DAGScheduler: Stage 20 (reduce at GradientDescent.scala:150) finished in 0.064 s
  189. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.080835519 s
  190. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 20.0 from pool
  191. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at GradientDescent.scala:150
  192. 14/02/17 21:54:33 INFO DAGScheduler: Got job 21 (reduce at GradientDescent.scala:150) with 2 output partitions (allowLocal=false)
  193. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 21 (reduce at GradientDescent.scala:150)
  194. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  195. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  196. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 21 (MappedRDD[43] at map at GradientDescent.scala:145), which has no missing parents
  197. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 21 (MappedRDD[43] at map at GradientDescent.scala:145)
  198. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 21.0 with 2 tasks
  199. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 21.0:0 as TID 40 on executor 2: slaver01 (NODE_LOCAL)
  200. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 21.0:0 as 2422 bytes in 0 ms
  201. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 21.0:1 as TID 41 on executor 1: slaver02 (NODE_LOCAL)
  202. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 21.0:1 as 2422 bytes in 0 ms
  203. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 40 in 40 ms on slaver01 (progress: 0/2)
  204. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(21, 0)
  205. 14/02/17 21:54:33 INFO TaskSetManager: Finished TID 41 in 45 ms on slaver02 (progress: 1/2)
  206. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Remove TaskSet 21.0 from pool
  207. 14/02/17 21:54:33 INFO DAGScheduler: Completed ResultTask(21, 1)
  208. 14/02/17 21:54:33 INFO DAGScheduler: Stage 21 (reduce at GradientDescent.scala:150) finished in 0.041 s
  209. 14/02/17 21:54:33 INFO SparkContext: Job finished: reduce at GradientDescent.scala:150, took 0.051875321 s
  210. 14/02/17 21:54:33 INFO GradientDescent: GradientDescent finished. Last 10 stochastic losses 0.22493248687186032, 0.2241836511724591, 0.22358630434392676, 0.22309440787976811, 0.22268441631265215, 0.22233909585390685, 0.22204555434717815, 0.2217939816090842, 0.22157679929323662, 0.22138807401560764
  211. 14/02/17 21:54:33 INFO LinearRegressionWithSGD: Final model weights 0.6226986501625317,0.26562471165823115,-0.13304380020663167,0.21671917665388107,0.3037175607477254,-0.2007533914066441,0.013953499241049204,0.20270603251011174
  212. 14/02/17 21:54:33 INFO LinearRegressionWithSGD: Final model intercept 2.46997566387807
  213. 14/02/17 21:54:33 INFO SparkContext: Starting job: reduce at SimpleApp.scala:24
  214. 14/02/17 21:54:33 INFO DAGScheduler: Got job 22 (reduce at SimpleApp.scala:24) with 2 output partitions (allowLocal=false)
  215. 14/02/17 21:54:33 INFO DAGScheduler: Final stage: Stage 22 (reduce at SimpleApp.scala:24)
  216. 14/02/17 21:54:33 INFO DAGScheduler: Parents of final stage: List()
  217. 14/02/17 21:54:33 INFO DAGScheduler: Missing parents: List()
  218. 14/02/17 21:54:33 INFO DAGScheduler: Submitting Stage 22 (MappedRDD[45] at map at SimpleApp.scala:24), which has no missing parents
  219. 14/02/17 21:54:33 INFO DAGScheduler: Submitting 2 missing tasks from Stage 22 (MappedRDD[45] at map at SimpleApp.scala:24)
  220. 14/02/17 21:54:33 INFO TaskSchedulerImpl: Adding task set 22.0 with 2 tasks
  221. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 22.0:0 as TID 42 on executor 2: slaver01 (NODE_LOCAL)
  222. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 22.0:0 as 2122 bytes in 1 ms
  223. 14/02/17 21:54:33 INFO TaskSetManager: Starting task 22.0:1 as TID 43 on executor 1: slaver02 (NODE_LOCAL)
  224. 14/02/17 21:54:33 INFO TaskSetManager: Serialized task 22.0:1 as 2122 bytes in 0 ms
  225. 14/02/17 21:54:34 INFO TaskSetManager: Finished TID 42 in 58 ms on slaver01 (progress: 0/2)
  226. 14/02/17 21:54:34 INFO DAGScheduler: Completed ResultTask(22, 0)
  227. 14/02/17 21:54:34 INFO TaskSetManager: Finished TID 43 in 60 ms on slaver02 (progress: 1/2)
  228. 14/02/17 21:54:34 INFO TaskSchedulerImpl: Remove TaskSet 22.0 from pool
  229. 14/02/17 21:54:34 INFO DAGScheduler: Completed ResultTask(22, 1)
  230. 14/02/17 21:54:34 INFO DAGScheduler: Stage 22 (reduce at SimpleApp.scala:24) finished in 0.057 s
  231. 14/02/17 21:54:34 INFO SparkContext: Job finished: reduce at SimpleApp.scala:24, took 0.081040964 s
  232. 14/02/17 21:54:34 INFO SparkContext: Starting job: count at SimpleApp.scala:24
  233. 14/02/17 21:54:34 INFO DAGScheduler: Got job 23 (count at SimpleApp.scala:24) with 2 output partitions (allowLocal=false)
  234. 14/02/17 21:54:34 INFO DAGScheduler: Final stage: Stage 23 (count at SimpleApp.scala:24)
  235. 14/02/17 21:54:34 INFO DAGScheduler: Parents of final stage: List()
  236. 14/02/17 21:54:34 INFO DAGScheduler: Missing parents: List()
  237. 14/02/17 21:54:34 INFO DAGScheduler: Submitting Stage 23 (MappedRDD[44] at map at SimpleApp.scala:20), which has no missing parents
  238. 14/02/17 21:54:34 INFO DAGScheduler: Submitting 2 missing tasks from Stage 23 (MappedRDD[44] at map at SimpleApp.scala:20)
  239. 14/02/17 21:54:34 INFO TaskSchedulerImpl: Adding task set 23.0 with 2 tasks
  240. 14/02/17 21:54:34 INFO TaskSetManager: Starting task 23.0:0 as TID 44 on executor 2: slaver01 (NODE_LOCAL)
  241. 14/02/17 21:54:34 INFO TaskSetManager: Serialized task 23.0:0 as 2011 bytes in 0 ms
  242. 14/02/17 21:54:34 INFO TaskSetManager: Starting task 23.0:1 as TID 45 on executor 1: slaver02 (NODE_LOCAL)
  243. 14/02/17 21:54:34 INFO TaskSetManager: Serialized task 23.0:1 as 2011 bytes in 0 ms
  244. 14/02/17 21:54:34 INFO TaskSetManager: Finished TID 45 in 44 ms on slaver02 (progress: 0/2)
  245. 14/02/17 21:54:34 INFO DAGScheduler: Completed ResultTask(23, 1)
  246. 14/02/17 21:54:34 INFO TaskSetManager: Finished TID 44 in 51 ms on slaver01 (progress: 1/2)
  247. 14/02/17 21:54:34 INFO TaskSchedulerImpl: Remove TaskSet 23.0 from pool
  248. 14/02/17 21:54:34 INFO DAGScheduler: Completed ResultTask(23, 0)
  249. 14/02/17 21:54:34 INFO DAGScheduler: Stage 23 (count at SimpleApp.scala:24) finished in 0.025 s
  250. 14/02/17 21:54:34 INFO SparkContext: Job finished: count at SimpleApp.scala:24, took 0.0633455 s
  251. training Mean Squared Error = 0.4424462080486391
  252. 14/02/17 21:54:34 INFO ConnectionManager: Selector thread was interrupted!
  253. [success] Total time: 41 s, completed Feb 17, 2014 9:54:34 PM
  254. [root@master scala]#
复制代码

Spark 的下载地址:请点这里

没找到任何评论,期待你打破沉寂

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条