about云开发-活到老 学到老

 找回密码
 立即注册

QQ登录

只需一步,快速开始

扫一扫,访问微社区

查看: 349|回复: 9

[疑问解答] spring和sparksql

[复制链接]

24

主题

5

听众

0

收听

中级会员

Rank: 3Rank: 3

积分
822
发表于 2018-6-13 16:17:42 | 显示全部楼层 |阅读模式
大家有用过spring和sparksql一块用吗?我从github上找了一个demo(https://github.com/Zhuinden/spring-spark-example),打成jar包后在local模式下是可以,但设置master.uri=yarn设置后出错了,在SparkSession那加.enableHiveSupport()后,hive也访问不了。
[AppleScript] 纯文本查看 复制代码
3404 [main] ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: Unable to load YARN support
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:390)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:385)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:385)
        at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:410)
        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2330)
        at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:107)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:349)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at com.zhuinden.sparkexperiment.ApplicationConfig.javaSparkContext(ApplicationConfig.java:45)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.CGLIB$javaSparkContext$1(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999$$FastClassBySpringCGLIB$$b0f16f12.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.javaSparkContext(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.obtainBeanInstanceFromFactory(ConfigurationClassEnhancer.java:389)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.javaSparkContext(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig.sparkSession(ApplicationConfig.java:52)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.CGLIB$sparkSession$2(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999$$FastClassBySpringCGLIB$$b0f16f12.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.sparkSession(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151)
        at com.zhuinden.sparkexperiment.SparkExperimentApplication.main(SparkExperimentApplication.java:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:386)
        ... 97 more
3413 [main] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext

67

主题

7

听众

8

收听

高级会员

Rank: 4

积分
1483
发表于 2018-6-14 05:51:34 | 显示全部楼层
猜测版本冲突spark2使用SparkSessionSparkContext被封装了。
这是常规的初始化方式:
SparkSession.builder
    .master("local")
    .appName("Word Count")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()

Java版 api:http://spark.apache.org/docs/2.2 ... l/SparkSession.html

欢迎加入about云群371358502、39327136,云计算爱好者群

24

主题

5

听众

0

收听

中级会员

Rank: 3Rank: 3

积分
822
 楼主| 发表于 2018-6-14 09:10:34 | 显示全部楼层
NEOGX 发表于 2018-6-14 05:51
猜测版本冲突spark2使用SparkSessionSparkContext被封装了。
这是常规的初始化方式:
SparkSession.build ...

集群是spark2.1.0。本地的jar是跟集群一致的,
[AppleScript] 纯文本查看 复制代码
return SparkSession
                .builder()
//                .sparkContext(javaSparkContext().sc())
                .master("yarn")
                .appName("Java Spark SQL basic example")
                .enableHiveSupport()
                .getOrCreate();
改成这样是一样的

6

主题

5

听众

5

收听

中级会员

Rank: 3Rank: 3

积分
725
发表于 2018-6-14 10:14:52 | 显示全部楼层
zstu 发表于 2018-6-14 09:10
集群是spark2.1.0。本地的jar是跟集群一致的,[mw_shl_code=applescript,true]return SparkSession
     ...

是不是不支持Yarn。standlone模式试下。

24

主题

5

听众

0

收听

中级会员

Rank: 3Rank: 3

积分
822
 楼主| 发表于 2018-6-14 10:28:11 | 显示全部楼层
hello2018 发表于 2018-6-14 10:14
是不是不支持Yarn。standlone模式试下。

standlone是可以的,但读不了hive数据。

171

主题

33

听众

36

收听

高级会员

Rank: 4

积分
3967

最佳新人活跃会员热心会员

发表于 2018-6-14 10:32:55 | 显示全部楼层
zstu 发表于 2018-6-14 10:28
standlone是可以的,但读不了hive数据。

试试添加下面:
SparkConf conf = new SparkConf();
conf.set("spark.master", "yarn-client");
conf.set("spark.local.ip","IP_OF_SANDBOX") ;
conf.set("spark.driver.host","IP_OF_MY_LOCAL_WINDOWS_MACHINE");
conf.set("spark.sql.hive.metastore.jars", "builtin");
conf.setAppName("Application name");
this.sparkSession = SparkSession.builder().config(conf).getOrCreate();
System.setProperty("HADOOP_USER_NAME", "root");
System.setProperty("SPARK_YARN_MODE", "yarn");

欢迎加入about云群425860289432264021 ,云计算爱好者群,关注about云腾讯认证空间

24

主题

5

听众

0

收听

中级会员

Rank: 3Rank: 3

积分
822
 楼主| 发表于 2018-6-14 11:07:34 | 显示全部楼层
langke93 发表于 2018-6-14 10:32
试试添加下面:
SparkConf conf = new SparkConf();
conf.set("spark.master", "yarn-client");

@Bean
    public SparkConf sparkConf() {
        /*SparkConf sparkConf = new SparkConf()
                .setAppName(appName)
                .setSparkHome(sparkHome)
                .setMaster(masterUri);*/
        SparkConf sparkConf = new SparkConf();
        sparkConf.set("spark.master", "yarn-client");
        sparkConf.set("spark.local.ip","node1") ;
        sparkConf.set("spark.driver.host","node1");
        sparkConf.set("spark.sql.hive.metastore.jars", "builtin");
        sparkConf.setAppName("Application name");

        return sparkConf;
    }

    @Bean
    public JavaSparkContext javaSparkContext() {
        return new JavaSparkContext(sparkConf());
    }

    @Bean
    public SparkSession sparkSession() {
        /*return SparkSession
                .builder()
//                .sparkContext(javaSparkContext().sc())
                .master("yarn")
                .appName("Java Spark SQL basic example")
                .enableHiveSupport()
                .getOrCreate();*/

        System.setProperty("HADOOP_USER_NAME", "root");
        System.setProperty("SPARK_YARN_MODE", "yarn");
        return SparkSession.builder().config(sparkConf()).enableHiveSupport().getOrCreate();

    }

改成这样,还是那个错

24

主题

5

听众

0

收听

中级会员

Rank: 3Rank: 3

积分
822
 楼主| 发表于 2018-6-14 14:40:42 | 显示全部楼层

感觉是spring这边加载不了yarn集群的环境

6

主题

5

听众

5

收听

中级会员

Rank: 3Rank: 3

积分
725
发表于 4 天前 | 显示全部楼层
zstu 发表于 2018-6-14 14:40
感觉是spring这边加载不了yarn集群的环境

spring与hadoop是否集成了。
推荐参考:
http://spring.io/projects/spring-hadoop
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

QQ|小黑屋|about云开发-学问论坛|社区-大数据云技术学习分享平台 ( 京ICP备12023829号

GMT+8, 2018-6-23 06:27 , Processed in 0.399938 second(s), 30 queries , Gzip On.

Powered by Discuz! X3.2 Licensed

© 2018 Comsenz Inc.Designed by u179

快速回复 返回顶部 返回列表