spark SQL Running the Thrift JDBC/ODBC server

本帖最后由 breaking 于 2016-3-31 13:48 编辑

问题导读：
1.Saprk怎么Running在Thrift JDBC Server上？
2.java JDBC怎么操作？

Running the Thrift JDBC/ODBC server
1:运行

[mw_shl_code=bash,true]
[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10000 --hiveconf hive.server2.thrift.bind.host=feng02 --master spark://feng02:7077 --driver-class-path /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/lib/mysql-connector-java-5.1.32-bin.jar
starting org.apache.spark.sql.hive.thriftserver.HiveThriftServer2, logging to /home/jifeng/hadoop/spark-1.2.0-bin-2.4.1/sbin/../logs/spark-jifeng-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-feng02.out[/mw_shl_code]

2:运行beeline

Now you can use beeline to test the Thrift JDBC/ODBC server:

[mw_shl_code=bash,true]./bin/beeline[/mw_shl_code]

[mw_shl_code=bash,true]
[jifeng@feng02 spark-1.2.0-bin-2.4.1]$ ./bin/beeline
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Beeline version ??? by Apache Hive[/mw_shl_code]

3:连接server
参考：https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineExample

[mw_shl_code=bash,true]
beeline> !connect jdbc:hive2://feng02:10000 jifeng jifeng org.apache.hive.jdbc.HiveDriver
Connecting to jdbc:hive2://feng02:10000
log4j:WARN No appenders could be found for logger (org.apache.thrift.transport.TSaslTransport).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Connected to: Spark SQL (version 1.2.0)
Driver: null (version null)
Transaction isolation: TRANSACTION_REPEATABLE_READ[/mw_shl_code]
4：查询

[mw_shl_code=bash,true]
0: jdbc:hive2://feng02:10000> show tables;
+----------------+
|    result    |
+----------------+
| course       |
| hbase_table_1  |
| pokes       |
| student       |
+----------------+
4 rows selected (2.723 seconds)[/mw_shl_code]
[mw_shl_code=bash,true]
0: jdbc:hive2://feng02:10000> select * from student;
+-----+----------+------+
| id  | name | age  |
+-----+----------+------+
| 1 | nick    | 24 |
| 2 | doping | 25 |
| 3 | caizhi | 26 |
| 4 | liaozhi  | 27 |
| 5 | wind    | 30 |
+-----+----------+------+
5 rows selected (10.554 seconds)
0: jdbc:hive2://feng02:10000> select a.*,b.* from student a  join course b where a.id=b.id ;
+-----+----------+------+-----+-----+-----+-----+-----+
| id  | name | age  | id  | c1  | c2  | c3  | c4  |
+-----+----------+------+-----+-----+-----+-----+-----+
| 1 | nick    | 24 | 1 | 英语  | 中文  | 法文  | 日文  |
| 2 | doping | 25 | 2 | 中文  | 法文  |    |    |
| 3 | caizhi | 26 | 3 | 中文  | 法文  | 日文  |    |
| 4 | liaozhi  | 27 | 4 | 中文  | 法文  | 拉丁  |    |
| 5 | wind    | 30 | 5 | 中文  | 法文  | 德文  |    |
+-----+----------+------+-----+-----+-----+-----+-----+
5 rows selected (2.33 seconds)[/mw_shl_code]

5:Java JDBC连接

[mw_shl_code=java,true]package demo.test;

import java.sql.*;

public class Pretest {

      public static void main( String args[] )
         throws SQLException , ClassNotFoundException {
              String jdbcdriver="org.apache.hive.jdbc.HiveDriver";
              String jdbcurl="jdbc:hive2://feng02:10000";
              String username="scott";
              String password="tiger";
              Class.forName(jdbcdriver);
              Connection c = DriverManager.getConnection(jdbcurl,username,password);
         Statement st = c.createStatement();
         print( "num should be 1 " , st.executeQuery("select * from student"));

         // TODO indexing
      }
   static void print( String name , ResultSet res )
         throws SQLException {
         System.out.println( name);
         ResultSetMetaData meta=res.getMetaData();
                  //System.out.println( "\t"+res.getRow()+"条记录");
                  String str="";
                  for(int i=1;i<=meta.getColumnCount();i++){
                    str+=meta.getColumnName(i)+" ";
                    //System.out.println( meta.getColumnName(i)+" ");
                  }
                  System.out.println("\t"+str);
                  str="";
         while ( res.next() ){
            for(int i=1;i<=meta.getColumnCount();i++){
            str+= res.getString(i)+" ";             }
            System.out.println("\t"+str);
            str="";
         }
      }
}[/mw_shl_code]

上面是运行参数
结果显示：

[mw_shl_code=bash,true]
num should be 1
id name age
1 nick 24
2 doping 25
3 caizhi 26
4 liaozhi 27
5 wind 30 [/mw_shl_code]

原文链接：http://blog.csdn.net/wind520/article/details/44061563