kudu是cloudera开源的运行在hadoop平台上的列式存储系统,拥有Hadoop生态系统应用的常见技术特性，运行在一般的商用硬件上，支持水平扩展,高可用,集成impala后，支持标准sql语句，相对于hbase易用性强，详细介绍。
impala是Cloudera公司主导开发的新型查询系统，它提供SQL语义，能查询存储在Hadoop的HDFS和HBase中的PB级大数据。已有的Hive系统虽然也提供了SQL语义，但由于Hive底层执行使用的是MapReduce引擎，仍然是一个批处理过程，难以满足查询的交互性。相比之下，Impala的最大特点也是最大卖点就是它的快速，导入数据实测可达30+W/s,详细介绍。
导入流程：准备数据--》上传hdfs--》导入impala临时表--》导入kudu表
1.准备数据
app@hadoop01:/software/develop/pujh>cat genBiData.sh
#!/usr/bash date
echo ''>data.txt
chmod data.txt for((i=;i<=20593279;i++))
do
echo "$i|aa$i|aa$i$i|aa$i$i$i" >>data.txt;
done; date app@hadoop01:/software/develop/pujh> sed 's/|/,/g' data.txt > temp.csv
app@hadoop01:/software/develop/pujh>chmod 777 tmp.csv
2.上传到hdfs
su - root
su - hdfs
hadoop dfs -mkdir /input/data/pujh
hadoop dfs -chmod -R /input/data/pujh
hadoop dfs -put /software/develop/pujh /input/data/pujh
hadoop dfs -ls /input/data/pujh
hdfs@hadoop01:>./hadoop dfs -ls /input/data/pujh
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it. Found items
-rwxrwxrwx hdfs supergroup -- : /input/data/pujh/aa.txt
-rwxrwxrwx hdfs supergroup -- : /input/data/pujh/data.txt
-rwxrwxrwx hdfs supergroup -- : /input/data/pujh/data2kw.csv
-rwxrwxrwx hdfs supergroup -- : /input/data/pujh/data_2kw.txt
-rwxrwxrwx hdfs supergroup -- : /input/data/pujh/genBiData.sh
3.导入impala临时表
创建impala临时表
employee_temp
create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';hdfs@hadoop02>./impala-shell
Starting Impala Shell without Kerberos authentication
Connected to hadoop02:
Server version: impala version 2.8.-cdh5.11.2 RELEASE (build f89269c4b96da14a841e94bdf6d4d48821b0d658)
***********************************************************************************
Welcome to the Impala shell.
(Impala Shell v2.8.0-cdh5.11.2 (f89269c) built on Fri Aug :: PDT ) The HISTORY command lists all shell commands in chronological order.
***********************************************************************************
[hadoop02:] > show databases;
Query: show databases
+------------------+----------------------------------------------+
| name | comment |
+------------------+----------------------------------------------+
| _impala_builtins | System database for Impala builtin functions |
| default | Default Hive database |
| td_test | |
+------------------+----------------------------------------------+
Fetched row(s) in .01s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name |
+----------------+
| employee |
| my_first_table |
+----------------+
Fetched row(s) in .00s [hadoop02:] > create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
Query: create table employee_temp ( eid int, name String,salary String, destination String) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' Fetched row(s) in .32s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name |
+----------------+
| employee |
| employee_temp |
| my_first_table |
+----------------+
Fetched row(s) in .01s
将hadoop上的文件导入impala临时表
load data inpath '/input/data/pujh/temp.csv' into table employee_temp;
[hadoop02:] > load data inpath '/input/data/pujh/temp.csv' into table employee_temp;
Query: load data inpath '/input/data/pujh/temp.csv' into table employee_temp
ERROR: AnalysisException: Unable to LOAD DATA from hdfs://hadoop01:8020/input/data/pujh/temp.csv because Impala does not have WRITE permissions on its parent directory hdfs://hadoop01:8020/input/data/pujh [hadoop02:] > load data inpath '/input/data/pujh/temp.csv' into table employee_temp;
Query: load data inpath '/input/data/pujh/temp.csv' into table employee_temp
+----------------------------------------------------------+
| summary |
+----------------------------------------------------------+
| Loaded file(s). Total files in destination location: |
+----------------------------------------------------------+
Fetched row(s) in .44s
[hadoop02:] > select * from employee_temp limit ;
Query: select * from employee_temp limit
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=4246eaa38a3d8bbb:953ce4d300000000
+------+------+--------+-------------+
| eid | name | salary | destination |
+------+------+--------+-------------+
| NULL | NULL | | |
| | aa1 | aa11 | aa111 |
+------+------+--------+-------------+
Fetched row(s) in .19s
[hadoop02:] > select * from employee_temp limit ;
Query: select * from employee_temp limit
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=cb4c3cf5d647c97a:75d2985f00000000
+------+------+--------+-------------+
| eid | name | salary | destination |
+------+------+--------+-------------+
| NULL | NULL | | |
| | aa1 | aa11 | aa111 |
| | aa2 | aa22 | aa222 |
| | aa3 | aa33 | aa333 |
| | aa4 | aa44 | aa444 |
| | aa5 | aa55 | aa555 |
| | aa6 | aa66 | aa666 |
| | aa7 | aa77 | aa777 |
| | aa8 | aa88 | aa888 |
| | aa9 | aa99 | aa999 |
+------+------+--------+-------------+
Fetched row(s) in .02s
[hadoop02:] > select count(*) from employee_temp;
Query: select count(*) from employee_temp
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=5a4c1107de118395:bfe96a1600000000
+----------+
| count(*) |
+----------+
| |
+----------+
Fetched row(s) in .65s
3.从impala临时表employee_temp 导入kudu表employee_kudu
创建kudu表
create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS 16 STORED AS KUDU;
[hadoop02:] > create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS STORED AS KUDU;
Query: create table employee_kudu ( eid int, name String,salary String, destination String,PRIMARY KEY(eid)) PARTITION BY HASH PARTITIONS STORED AS KUDU Fetched row(s) in .94s
[hadoop02:] > show tables;
Query: show tables
+----------------+
| name |
+----------------+
| employee |
| employee_kudu |
| employee_temp |
| my_first_table |
界面查看是否创建成功
从impala临时表employee_temp 导入kudu表employee_kudu
[hadoop02:] > insert into employee_kudu select * from employee_temp;
Query: insert into employee_kudu select * from employee_temp
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=2e47536cc5c82392:ef4d552600000000
WARNINGS: Row with null value violates nullability constraint on table 'impala::default.employee_kudu'. Modified row(s), row error(s) in .75s
[hadoop02:] > select count(*) from employee_kudu;
Query: select count(*) from employee_kudu
Query submitted at: -- :: (Coordinator: http://hadoop02:25000)
Query progress can be monitored at: http://hadoop02:25000/query_plan?query_id=6d4bad44a980f229:fd7878d00000000
+----------+
| count(*) |
+----------+
| |
+----------+
Fetched row(s) in .18s