spring和sparksql

大家有用过spring和sparksql一块用吗？我从github上找了一个demo（https://github.com/Zhuinden/spring-spark-example），打成jar包后在local模式下是可以，但设置master.uri=yarn设置后出错了，在SparkSession那加.enableHiveSupport()后，hive也访问不了。[mw_shl_code=applescript,true]3404 [main] ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
org.apache.spark.SparkException: Unable to load YARN support
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:390)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn$lzycompute(SparkHadoopUtil.scala:385)
        at org.apache.spark.deploy.SparkHadoopUtil$.yarn(SparkHadoopUtil.scala:385)
        at org.apache.spark.deploy.SparkHadoopUtil$.get(SparkHadoopUtil.scala:410)
        at org.apache.spark.util.Utils$.getSparkOrYarnConfig(Utils.scala:2330)
        at org.apache.spark.storage.BlockManager.<init>(BlockManager.scala:107)
        at org.apache.spark.SparkEnv$.create(SparkEnv.scala:349)
        at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:174)
        at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
        at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
        at com.zhuinden.sparkexperiment.ApplicationConfig.javaSparkContext(ApplicationConfig.java:45)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.CGLIB$javaSparkContext$1(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999$$FastClassBySpringCGLIB$$b0f16f12.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.javaSparkContext(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.obtainBeanInstanceFromFactory(ConfigurationClassEnhancer.java:389)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:361)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.javaSparkContext(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig.sparkSession(ApplicationConfig.java:52)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.CGLIB$sparkSession$2(<generated>)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999$$FastClassBySpringCGLIB$$b0f16f12.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:358)
        at com.zhuinden.sparkexperiment.ApplicationConfig$$EnhancerBySpringCGLIB$$2b068999.sparkSession(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:737)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:370)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:314)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1162)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1151)
        at com.zhuinden.sparkexperiment.SparkExperimentApplication.main(SparkExperimentApplication.java:10)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:94)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
        at org.apache.spark.deploy.SparkHadoopUtil$.liftedTree1$1(SparkHadoopUtil.scala:386)
        ... 97 more
3413 [main] INFO org.apache.spark.SparkContext - Successfully stopped SparkContext[/mw_shl_code]

猜测版本冲突spark2使用SparkSessionSparkContext被封装了。
这是常规的初始化方式：
SparkSession.builder
    .master("local")
    .appName("Word Count")
    .config("spark.some.config.option", "some-value")
    .getOrCreate()

Java版 api：http://spark.apache.org/docs/2.2 ... l/SparkSession.html

NEOGX 发表于 2018-6-14 05:51
猜测版本冲突spark2使用SparkSessionSparkContext被封装了。
这是常规的初始化方式：
SparkSession.build ...

集群是spark2.1.0。本地的jar是跟集群一致的，[mw_shl_code=applescript,true]return SparkSession
                .builder()
//                .sparkContext(javaSparkContext().sc())
                .master("yarn")
                .appName("Java Spark SQL basic example")
                .enableHiveSupport()
                .getOrCreate();[/mw_shl_code]改成这样是一样的

zstu 发表于 2018-6-14 09:10
集群是spark2.1.0。本地的jar是跟集群一致的，[mw_shl_code=applescript,true]return SparkSession
     ...

是不是不支持Yarn。standlone模式试下。

hello2018 发表于 2018-6-14 10:14
是不是不支持Yarn。standlone模式试下。

standlone是可以的，但读不了hive数据。

zstu 发表于 2018-6-14 10:28
standlone是可以的，但读不了hive数据。

试试添加下面：
SparkConf conf = new SparkConf();
conf.set("spark.master", "yarn-client");
conf.set("spark.local.ip","IP_OF_SANDBOX") ;
conf.set("spark.driver.host","IP_OF_MY_LOCAL_WINDOWS_MACHINE");
conf.set("spark.sql.hive.metastore.jars", "builtin");
conf.setAppName("Application name");
this.sparkSession = SparkSession.builder().config(conf).getOrCreate();
System.setProperty("HADOOP_USER_NAME", "root");
System.setProperty("SPARK_YARN_MODE", "yarn");

langke93 发表于 2018-6-14 10:32
试试添加下面：
SparkConf conf = new SparkConf();
conf.set("spark.master", "yarn-client");

@Bean
    public SparkConf sparkConf() {
        /*SparkConf sparkConf = new SparkConf()
                .setAppName(appName)
                .setSparkHome(sparkHome)
                .setMaster(masterUri);*/
        SparkConf sparkConf = new SparkConf();
        sparkConf.set("spark.master", "yarn-client");
        sparkConf.set("spark.local.ip","node1") ;
        sparkConf.set("spark.driver.host","node1");
        sparkConf.set("spark.sql.hive.metastore.jars", "builtin");
        sparkConf.setAppName("Application name");

        return sparkConf;
    }

    @Bean
    public JavaSparkContext javaSparkContext() {
        return new JavaSparkContext(sparkConf());
    }

    @Bean
    public SparkSession sparkSession() {
        /*return SparkSession
                .builder()
//                .sparkContext(javaSparkContext().sc())
                .master("yarn")
                .appName("Java Spark SQL basic example")
                .enableHiveSupport()
                .getOrCreate();*/

        System.setProperty("HADOOP_USER_NAME", "root");
        System.setProperty("SPARK_YARN_MODE", "yarn");
        return SparkSession.builder().config(sparkConf()).enableHiveSupport().getOrCreate();

    }

改成这样，还是那个错

缺少jar

fly2015 发表于 2018-6-14 11:44
缺少jar

感觉是spring这边加载不了yarn集群的环境

zstu 发表于 2018-6-14 14:40
感觉是spring这边加载不了yarn集群的环境

spring与hadoop是否集成了。
推荐参考：
http://spring.io/projects/spring-hadoop