Presto-[5]-Presto Running in IDEA

时间:2021-12-14 21:50:16

准备

已经成功安装单机presto(没有安装也可以),配置过hive.catalog文件

参考:Presto-[4]-单机配置查询Hive

git clone master分支

gihub:https://github.com/prestodb/presto

1-Building Presto

在根目录执行,会下载依赖至 (~/.m2/repository),这会一并执行测试部分,时间长些

./mvnw clean install

若想跳过其中的test 测试代码,也可选择执行

./mvnw clean install -DskipTests 
或者./mvn clean install -DskipTests -Dcheckstyle.skip=true (当IDEA启动presto server 有问题时,重新执行一遍

2-Running Presto in your IDE

推荐使用IntelliJ IDEA,通过Open from the File menu and select the root pom.xml file 打开工程,确定Java SDK 配置正确:

  • Open the File menu and select Project Structure
  • In the SDKs section, ensure that a 1.8 JDK is selected (create one if none exist)
  • In the Project section, ensure the Project language level is set to 8.0 as Presto makes use of several Java 8 language features

配置运行环境:

  • Main Class: com.facebook.presto.server.PrestoServer
  • VM Options: -ea -XX:+UseG1GC -XX:G1HeapRegionSize=32M -XX:+UseGCOverheadLimit -XX:+ExplicitGCInvokesConcurrent -Xmx2G -Dconfig=etc/config.properties -Dlog.levels-file=etc/log.properties 
    -Dhive.metastore.uri=thrift://localhost:9083
  • Working directory: $MODULE_DIR$
  • Use classpath of module: presto-main

The working directory should be the presto-main subdirectory. In IntelliJ, using $MODULE_DIR$ accomplishes this automatically.

备注:对于hive plugin必须配置Hive metastore Thrift service的location,在VM options的列表中添加下面的参数 把 localhost:9083 改成你的实际参数(如果你没有hive  metastore 直接使用下面的参数):

-Dhive.metastore.uri=thrift://localhost:9083

3-启动hive-meta和hive-server2 

hive --service hiveserver2 & 

hive --service metastore &

如果没有安装presto 跳过4到7,直接按8中说明IDEA中完全执行

4-启动Presto server

启动mysql、启动hadoop

Presto-[5]-Presto Running in IDEA


/Documents/software/Presto/presto-server-0.191$ bin/launcher start
Started as 56523

5-Running the CLI

IDEA中Start the CLI to connect to the server and run SQL queries:

presto-cli/target/presto-cli-*-executable.jar

6-IDEA查询hive

presto> SELECT * FROM hive.hive_test.student;
SELECT * FROM hive.hive_test.student;
id | name 
------+----------
1001 | zhangsan 
1002 | lisi 
(2 rows)

Query 20171220_031213_00003_8z8an, FINISHED, 1 node
Splits: 17 total, 17 done (100.00%)
0:00 [2 rows, 23B] [26 rows/s, 300B/s]

presto>

Presto-[5]-Presto Running in IDEA

7-UI

localhost:8080

Presto-[5]-Presto Running in IDEA

以上是利用IDEA中源码中的presto-cli代码部分提交statement(sql),可以debug看presto-cli部分代码的执行流程。

然后,通过restful接口调用本地的单机presto server (“4-启动Presto server”)执行查询,查询本地单机hive中的数据

其中主入口为:

Presto-[5]-Presto Running in IDEA

Console::runConsole方法中等待Console中用户输入的statement,之后
LineReader接收到输入,转换成sql
process(queryRunner, split.statement(), outputFormat, tableNameCompleter::populateCache, true);

然后,通过restful接口调用本地的单机presto server (“4-启动Presto server”)执行查询,查询本地单机hive中的数据


8-Debug 模式学习源码

 要debug执行代码,  则需要本地IDEA中运行persto Server ,步骤主要是:

关闭 “4-启动Presto server”中启动的本地的单机presto server (否则已有一个sever idea中再启用一个会出错),命令行jps 查询pid kill掉

Presto-[5]-Presto Running in IDEA


(1)如“ 2-Running Presto in your IDE” 中配置一个run application 如命名为persto

Presto-[5]-Presto Running in IDEA


然后run 或 debug presto,同时run CLI 如 “5-Running the CLI”所示,IDEA中启动了persto Server

Presto-[5]-Presto Running in IDEA

Presto-[5]-Presto Running in IDEA


Presto-[5]-Presto Running in IDEA

这时,就可从IEDA中的cli中提交sql,同时debug 在IDEA 中看server中具体执行,server的具体代码解析后续学习后 再整理

备注debug 过程中CLI 中提示timeout ,这时可以将配置参数调的大些

Presto-[5]-Presto Running in IDEA

Presto-[5]-Presto Running in IDEA


Presto-[5]-Presto Running in IDEA