Hive Server 2 安装部署测试

时间：2023-03-09 08:40:09
Hive Server 2 安装部署测试

Hive 0.11 包含了Hive Server 1 和 Hive Server 2，还包含1的原因是为了做到向下兼容性。从长远来看都会以Hive Server 2作为首选

1. 配置hive server监听端口和Host 
<property>

  <name>hive.server2.thrift.port</name>

  <value>10000</value>

</property>

<property>

  <name>hive.server2.thrift.bind.host</name>

  <value>test84.hadoop</value>

</property>

2. 配置kerberos认证，这样thrift client与hive server 2, hive server 2和hdfs 都由kerberos作认证

<property>

  <name>hive.server2.authentication</name>

  <value>KERBEROS</value>

  <description>

    Client authentication types.

       NONE: no authentication check

       LDAP: LDAP/AD based authentication

       KERBEROS: Kerberos/GSSAPI authentication

       CUSTOM: Custom authentication provider

               (Use with property hive.server2.custom.authentication.class)

  </description>

</property>

<property>

  <name>hive.server2.authentication.kerberos.principal</name>

  <value>hadoop/_HOST@DIANPING.COM</value>

</property>

<property>

  <name>hive.server2.authentication.kerberos.keytab</name>

  <value>/etc/hadoop.keytab</value>

</property>

3. 设置impersonation，这样hive server会以提交用户的身份去执行语句，如果设置为false，则会以起hive server daemon的admin user来执行语句

<property>

  <name>hive.server2.enable.doAs</name>

  <value>true</value>

</property>

执行命令$HIVE_HOME/bin/hive --service hiveserver2 会调用org.apache.hive.service.server.HiveServer2的main方法

hive log中输出日志信息如下：
2013-09-17 14:59:21,081 INFO  server.HiveServer2 (HiveStringUtils.java:startupShutdownMessage(604)) - STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting HiveServer2

STARTUP_MSG:   host = test84.hadoop/10.1.77.84

STARTUP_MSG:   args = []

STARTUP_MSG:   version = 0.11.0

STARTUP_MSG:   classpath = 略.................

2013-09-17 14:59:21,957 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/test84.hadoop@DIANPING.COM using keytab file /etc/hadoop.keytab

2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:OperationManager is inited.

2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:SessionManager is inited.

2013-09-17 14:59:21,958 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:CLIService is inited.

2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:ThriftCLIService is inited.

2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:init(89)) - Service:HiveServer2 is inited.

2013-09-17 14:59:21,959 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:OperationManager is started.

2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:SessionManager is started.

2013-09-17 14:59:21,960 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:CLIService is started.

2013-09-17 14:59:22,007 INFO  metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(409)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

2013-09-17 14:59:22,032 INFO  metastore.ObjectStore (ObjectStore.java:initialize(222)) - ObjectStore, initialize called

2013-09-17 14:59:22,955 INFO  metastore.ObjectStore (ObjectStore.java:getPMF(267)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"

2013-09-17 14:59:23,000 INFO  metastore.ObjectStore (ObjectStore.java:setConf(205)) - Initialized ObjectStore

2013-09-17 14:59:23,909 INFO  metastore.HiveMetaStore (HiveMetaStore.java:logInfo(452)) - 0: get_databases: default

2013-09-17 14:59:23,912 INFO  HiveMetaStore.audit (HiveMetaStore.java:logAuditEvent(238)) - ugi=hadoop/test84.hadoop@DIANPING.COM ip=unknown-ip-addr cmd=get_databases: default

2013-09-17 14:59:23,933 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:ThriftCLIService is started.

2013-09-17 14:59:23,948 INFO  service.AbstractService (AbstractService.java:start(104)) - Service:HiveServer2 is started.

2013-09-17 14:59:24,025 INFO  security.UserGroupInformation (UserGroupInformation.java:loginUserFromKeytab(633)) - Login successful for user hadoop/test84.hadoop@DIANPING.COM using keytab file /etc/hadoop.keytab

2013-09-17 14:59:24,047 INFO  thrift.ThriftCLIService (ThriftCLIService.java:run(435)) - ThriftCLIService listening on test84.hadoop/10.1.77.84:10000

可以看到在HiveServer2已经变成一个CompisiteService了，它包含了一组service，包括OperationManager，SessionManager，CLIService，ThriftCLIService。并且在初始化的时候会建立HiveMetaStore连接，并调用get_databases命令来测试。最后启动thrift server，监听在test84.hadoop/10.1.77.84:10000端口上

1. Beeline访问hive server 2
Beeline是hive 0.11引入的新的交互式CLI，它基于SQLLine，可以作为Hive JDBC Client端访问Hive Server 2，启动一个beeline就是维护了一个session。
由于采用了kerberos认证方式，所以需要在本地有kerberos ticket，并且在connection url中指定hive server 2的service principal，此处为principal=hadoop/test84.hadoop@DIANPING.COM，另外用户名和密码可以不用填写，之后的语句会以当前ticket cache中principal的用户身份来执行。 
-dpsh-3.2$ bin/beeline

Beeline version 0.11.0 by Apache Hive

beeline> !connect jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM

scan complete in 2ms

Connecting to jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM

Enter username for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM:

Enter password for jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM:

Connected to: Hive (version 0.11.0)

Driver: Hive (version 0.11.0)

Transaction isolation: TRANSACTION_REPEATABLE_READ

0: jdbc:hive2://test84.hadoop:10000/default> select count(1) from abc;

+------+

| _c0  |

+------+

| 0    |

+------+

1 row selected (29.277 seconds)

0: jdbc:hive2://test84.hadoop:10000/default> !q

Closing: org.apache.hive.jdbc.HiveConnection
thrift client和server会建立一个session handler，有唯一的HandleIdentifier，由SessionManager Service管理，这也就是Hive server 2支持concurrency的方式。每次操作(会有不同的opType,比如EXECUTE_STATEMEN)会生成独立的OperationHandle，也有各自的HandleIdentifier。用户在beeline中输入"!q"会销毁该session，并且销毁相应的资源
ps : 用下来有点不太爽的是执行mapreduce job时候没有执行过程信息
，如果是一个执行时间很长
的语句，会等很久而没有任何信息反馈
。

2. JDBC方式
hive server 1的driver classname是org.apache.hadoop.hive.jdbc.HiveDriver，Hive Server 2的是
org.apache.hive.jdbc.HiveDriver，这两个容易混淆。
另外可以在connectionUrl中指定HiveConf param和变量，params之间用‘;’分割，params和variables用'#'来隔开。这些都是session级别的，hive在建立完session后，会首先执行set hiveconf key value语句。
比如
jdbc:hive2://test84.hadoop:10000/default?hive.cli.conf.printheader=true#stab=salesTable;icol=customerID
jdbc:hive2://test84.hadoop:10000/default;user=foo;password=bar

示例代码 

import java.sql.Connection;

import java.sql.DriverManager;

import java.sql.ResultSet;

import java.sql.ResultSetMetaData;

import java.sql.SQLException;

import java.sql.Statement;

public class HiveTest {

	public static void main(String[] args) throws SQLException {

		try {

			Class.forName("org.apache.hive.jdbc.HiveDriver");

		} catch (ClassNotFoundException e) {

			e.printStackTrace();

		}

		Connection conn = DriverManager

				.getConnection(

						"jdbc:hive2://test84.hadoop:10000/default;principal=hadoop/test84.hadoop@DIANPING.COM",

						"", "");

		Statement stmt = conn.createStatement();

		String sql = "select * from abc";

		System.out.println("Running: " + sql);

		ResultSet res = stmt.executeQuery(sql);

		ResultSetMetaData rsmd = res.getMetaData();

		int columnCount = rsmd.getColumnCount();

		for (int i = 1; i <= columnCount; i++) {

			System.out.println(rsmd.getColumnTypeName(i) + ":"

					+ rsmd.getColumnName(i));

		}

		while (res.next()) {

			System.out.println(String.valueOf(res.getInt(1)) + "\t"

					+ res.getString(2));

		}

	}

}

如果kerberos认证有问题的话，可以在起client jvm时候增加JVM option "
-Dsun.security.krb5.debug=true"来查看详细信息

参考链接:


相关文章