Hadoop-Impala学习笔记之SQL参考

时间:2023-03-08 20:44:11

参考:Apache Impala Guide--Impala SQL Language Reference。

Impala使用和Hive一样的元数据存储,Impala可以访问使用原生Impala CREATE TABLE创建的表和Hive DDL创建的表;

Impala支持和HiveQL类似的DML;

支持TRUNCATE(至少在5.5及之前,不支持Kudu表)

UPDATE/UPSERT(5.10开始支持Kudu表)

Impala提供了很多和HiveQL等价的内置函数;

支持大多数子句如WITH, JOIN, AGGREGATE,DISTINCT, UNION ALL, ORDER BY, LIMIT(相关子查询除外);

支持和Hive一样的数据类型,如STRING, TINYINT,SMALLINT, INT, BIGINT, FLOAT, DOUBLE, BOOLEAN, STRING, TIMESTAMP(需要注意Kudu和Parquet文件格式的限制);

impala支持分区表、外部表;

impala支持绝大部分SQL 92语句;

查看各种上下文信息,可以使用SHOW XXX;如:

SHOW DATABASES [[LIKE] 'pattern']
SHOW SCHEMAS [[LIKE] 'pattern'] - an alias for SHOW DATABASES
SHOW TABLES [IN database_name] [[LIKE] 'pattern']
SHOW [AGGREGATE | ANALYTIC] FUNCTIONS [IN database_name] [[LIKE] 'pattern']
SHOW CREATE TABLE [database_name].table_name
SHOW CREATE VIEW [database_name].view_name
SHOW TABLE STATS [database_name.]table_name
SHOW COLUMN STATS [database_name.]table_name
SHOW [RANGE] PARTITIONS [database_name.]table_name
SHOW FILES IN [database_name.]table_name [PARTITION (key_col_expression [,
key_col_expression]]
SHOW ROLES
SHOW CURRENT ROLES
SHOW ROLE GRANT GROUP group_name
SHOW GRANT ROLE role_name

和RDBMS一样,支持统计信息收集COMPUTE [INCREMENTAL] STATS [db_name.]table_name(不同于Hive的ANALYZE TABLE需要分别为列和表收集统计信息,会一次性收集);增量统计信息收集适合于分区表(在ETL的最后应该收集统计信息)。下列语句可用来查看表的统计信息:

[quickstart.cloudera:] > show table stats t1;
Query: show table stats t1
+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------+
| #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location |
+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------+
| - | | 512B | NOT CACHED | NOT CACHED | TEXT | false | hdfs://quickstart.cloudera:8020/user/hive/warehouse/t1 |
+-------+--------+------+--------------+-------------------+--------+-------------------+--------------------------------------------------------+
Fetched row(s) in .17s
[quickstart.cloudera:] > show column stats t1;
Query: show column stats t1
+--------+------+------------------+--------+----------+----------+
| Column | Type | #Distinct Values | #Nulls | Max Size | Avg Size |
+--------+------+------------------+--------+----------+----------+
| x | INT | - | - | | |
+--------+------+------------------+--------+----------+----------+
Fetched row(s) in .11s

支持优化器提示(但是比较弱,要掌握优化器提示,必须先掌握执行计划)

有好几种格式(有点乱):

SELECT STRAIGHT_JOIN select_list FROM
join_left_hand_table
JOIN [{ /* +BROADCAST */ | /* +SHUFFLE */ }]
join_right_hand_table
remainder_of_query; INSERT insert_clauses
[{ /* +SHUFFLE */ | /* +NOSHUFFLE */ }]
[/* +CLUSTERED */]
SELECT remainder_of_query; SELECT select_list FROM
table_ref
/* +{SCHEDULE_CACHE_LOCAL | SCHEDULE_DISK_LOCAL | SCHEDULE_REMOTE}
[,RANDOM_REPLICA] */
remainder_of_query; -- 最近的优化器提示支持下列格式:
SELECT select_list FROM
join_left_hand_table
JOIN -- +BROADCAST|SHUFFLE
join_right_hand_table
remainder_of_query; INSERT insert_clauses
/* +SHUFFLE|NOSHUFFLE */
SELECT remainder_of_query;

有针对Impala执行本身的、也有针对控制HDFS调度的。

支持的各种hint可以参考Query Hints in Impala SELECT Statements

数据移动操作,LOAD DATA语句可以用来将HDFS目录的数据移动到Impala数据目录(目前不支持从本地文件系统移动,真正的移动、不是复制)可以直接映射到HDFS文件,那什么时候需要这个操作??(其实和外部表性质类似,只不过移动后文件为Impala管辖,否则为HDFS管辖)。

LOAD DATA INPATH 'hdfs_file_or_directory_path' [OVERWRITE] INTO TABLE tablename [PARTITION (partcol1=val1, partcol2=val2 ...)]

会话上下文设置:

它原先是一个impala-shell命令,后来被扩展为SQL语句,主要用于JDBC和ODBC API中。

不带任何参数可输出所有可设置的上下文选项列表,如下:

[quickstart.cloudera:21000] > set
> ;
Query options (defaults shown in []):
ABORT_ON_DEFAULT_LIMIT_EXCEEDED: []
ABORT_ON_ERROR: []
ALLOW_UNSUPPORTED_FORMATS: []
APPX_COUNT_DISTINCT: []
BATCH_SIZE: []
BUFFER_POOL_LIMIT: []
COMPRESSION_CODEC: [NONE]
DEBUG_ACTION: []
DECIMAL_V2: []
DEFAULT_JOIN_DISTRIBUTION_MODE: []
DEFAULT_ORDER_BY_LIMIT: [-1]
DEFAULT_SPILLABLE_BUFFER_SIZE: []
DISABLE_CACHED_READS: []
DISABLE_CODEGEN: []
DISABLE_CODEGEN_ROWS_THRESHOLD: []
DISABLE_OUTERMOST_TOPN: []
DISABLE_ROW_RUNTIME_FILTERING: []
DISABLE_STREAMING_PREAGGREGATIONS: []
DISABLE_UNSAFE_SPILLS: []
ENABLE_EXPR_REWRITES: []
EXEC_SINGLE_NODE_ROWS_THRESHOLD: []
EXPLAIN_LEVEL: []
HBASE_CACHE_BLOCKS: []
HBASE_CACHING: []
MAX_ERRORS: []
MAX_IO_BUFFERS: []
MAX_NUM_RUNTIME_FILTERS: []
MAX_ROW_SIZE: []
MAX_SCAN_RANGE_LENGTH: []
MEM_LIMIT: []
MIN_SPILLABLE_BUFFER_SIZE: []
MT_DOP: []
NUM_NODES: []
NUM_SCANNER_THREADS: []
OPTIMIZE_PARTITION_KEY_SCANS: []
PARQUET_ANNOTATE_STRINGS_UTF8: []
PARQUET_ARRAY_RESOLUTION: []
PARQUET_DICTIONARY_FILTERING: []
PARQUET_FALLBACK_SCHEMA_RESOLUTION: []
PARQUET_FILE_SIZE: []
PARQUET_READ_STATISTICS: []
PREFETCH_MODE: []
QUERY_TIMEOUT_S: []
REPLICA_PREFERENCE: []
REQUEST_POOL: []
RESERVATION_REQUEST_TIMEOUT: []
RM_INITIAL_MEM: []
RUNTIME_BLOOM_FILTER_SIZE: []
RUNTIME_FILTER_MAX_SIZE: []
RUNTIME_FILTER_MIN_SIZE: []
RUNTIME_FILTER_MODE: []
RUNTIME_FILTER_WAIT_TIME_MS: []
S3_SKIP_INSERT_STAGING: []
SCAN_NODE_CODEGEN_THRESHOLD: []
SCHEDULE_RANDOM_REPLICA: []
SCRATCH_LIMIT: [-1]
SEQ_COMPRESSION_MODE: []
STRICT_MODE: []
SUPPORT_START_OVER: [false]
SYNC_DDL: []
V_CPU_CORES: [] Shell Options
LIVE_PROGRESS: False
LIVE_SUMMARY: False Variables:
No variables defined.

作为命令行参数值不需要带引号,否则需要引号。所有可用的选项可以参考“Query Options for the SET Statement”。