hadoop hdfs 命令

时间:2021-09-02 20:25:41

hdfs命令常用操作:

hdfs帮助

-help [cmd] 显示命令的帮助信息

[hadoop@hadoop-01 ~]$ hdfs dfs -help ls

递归显示当前目录下的所有文件:

[hadoop@hadoop-01 ~]$ hdfs dfs -ls -h /
Found 1 items
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp [hadoop@hadoop-01 ~]$ hdfs dfs -ls -h -R /
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp/hadoop-yarn
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp/hadoop-yarn/staging
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp/hadoop-yarn/staging/history
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /tmp/hadoop-yarn/staging/history/done
drwxrwxrwt - hadoop supergroup 0 2017-11-23 13:09 /tmp/hadoop-yarn/staging/history/done_intermediate

-du 显示目录中所有文件大小:

[hadoop@hadoop-01 ~]$ hdfs dfs -du -s -h /tmp/
0 /tmp
[hadoop@hadoop-01 ~]$

-count计算路径下的目录、文件和字节数

[hadoop@hadoop-01 ~]$ hdfs dfs -count -q -h /tmp/
none inf none inf 6 0 0 /tmp
[hadoop@hadoop-01 ~]$

-mkdir 在指定位置创建一个hdfs目录.

[hadoop@hadoop-01 ~]$ hdfs dfs -mkdir testdhadoop

递归创建目录:

[hadoop@hadoop-01 bin]$ ./hdfs dfs -mkdir -p /test1/test2/test3
#递归查看当前目录下所有文件.
[hadoop@hadoop-01 bin]$ ./hdfs dfs -ls -R /test1
drwxr-xr-x - hadoop supergroup 0 2017-11-23 15:17 /test1/test2
drwxr-xr-x - hadoop supergroup 0 2017-11-23 15:17 /test1/test2/test3

-mv  移动多个文件目录到目标目录,(移动的文件也需要是hdfs目录中存在的文件.)

[hadoop@hadoop-01 ~]$ hdfs dfs -mv /tmp/hadoop-yarn /user/hadoop/testdhadoop

-cp 复制多个dhfs文件到目标目录

[hadoop@hadoop-01 ~]$ hdfs dfs -cp /user/hadoop/testdhadoop /tmp/hadoop-yarn

-put 本地文件复制到hdfs

[hadoop@hadoop-01 ~]$ hdfs dfs -put /etc/passwd /user/hadoop/testdhadoop

-copyFromLocal 与- put 命令相同.

[hadoop@hadoop-01 ~]$ hdfs dfs -copyFromLocal /etc/yum.conf /user/Hadoop
[hadoop@hadoop-01 ~]$ hdfs dfs -ls -R /user/hadoop
drwxr-xr-x - hadoop supergroup 0 2017-11-23 14:37 /user/hadoop/testdhadoop
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /user/hadoop/testdhadoop/hadoop-yarn
drwxrwx--- - hadoop supergroup 0 2017-11-23 13:09 /user/hadoop/testdhadoop/passwd
-rw-r--r-- 2 hadoop supergroup 969 2017-11-23 14:41 /user/hadoop/yum.conf

-moveFromLocal 本地文件移动到 hdfs.

[hadoop@hadoop-01 ~]$ hdfs dfs -mkdir /logs
[hadoop@hadoop-01 ~]$ hdfs dfs -ls -d /logs
drwxr-xr-x - hadoop supergroup 0 2017-11-23 14:47 /logs
[hadoop@hadoop-01 ~]$ hdfs dfs -moveFromLocal test.txt /logs
[hadoop@hadoop-01 ~]$ hdfs dfs -ls -h /logs
Found 1 items
-rw-r--r-- 2 hadoop supergroup 12 2017-11-23 14:49 /logs/test.txt

get [-ignoreCrc] 复制hdfs文件到本地,可以忽略crc校验.

[hadoop@hadoop-01 ~]$ hdfs dfs -get /logs/test.txt /tmp/

- copyToLocal 与- get命令相同 复制dhfs文件到本地.

[hadoop@hadoop-01 bin]$ ./hdfs dfs -copyToLocal /logs/test.txt /home/hadoop/
[hadoop@hadoop-01 ~]$ ls -lh /home/hadoop/
total 16K
drwxrwxr-x. 4 hadoop hadoop 4.0K Nov 23 12:12 dfs
drwxr-xr-x. 11 hadoop hadoop 4.0K Nov 23 12:47 hadoop
-rw-r--r--. 1 hadoop hadoop 12 Nov 23 15:05 test.txt
drwxrwxr-x. 3 hadoop hadoop 4.0K Nov 23 12:48 tmp

- cat 在终端显示文件内容

[hadoop@hadoop-01 /]$ hdfs dfs -cat /logs/test.txt
hello world
[hadoop@hadoop-01 /]$

- text 在终端显示文件内容,将源文件输出为文本格式。允许的格式是zip和TextRecordInputStream.  

[hadoop@hadoop-01 bin]$ ./hdfs dfs -text /logs/test.txt
hello world
[hadoop@hadoop-01 /]$ hdfs dfs -tail /logs/part-00000 (查看文件的最后一千行)
[hadoop@hadoop-01 /]$ hdfs dfs -cat /logs/part-00000 | head

- touchz 创建一个hdfs空文件.

[hadoop@hadoop-01 bin]$ ./hdfs dfs -touchz /test1/1.txt
[hadoop@hadoop-01 bin]$ ./hdfs dfs -ls -R /test1
-rw-r--r-- 2 hadoop supergroup 0 2017-11-23 15:20 /test1/1.txt
drwxr-xr-x - hadoop supergroup 0 2017-11-23 15:17 /test1/test2
drwxr-xr-x - hadoop supergroup 0 2017-11-23 15:17 /test1/test2/test3

- getmerge [addnl] 将hdfs源目录中的所有文件排序合并到一个本地文件中,接受一个源目录和一个目标文件作为输入,并且将源目录中所有的文件连接成本地目标文件。addnl是可选的,用于指定在每个文件结尾添加一个换行符.

#将hdfs上的/logs/* 下的所有文件合并下载到本地的/tmp/hello文件中.
[hadoop@hadoop-01 bin]$ ./hdfs dfs -getmerge /logs/* /tmp/hello
[hadoop@hadoop-01 bin]$ cat /tmp/hello
111111111111111111111111
hello world
[hadoop@hadoop-01 bin]$

- grep 从hdfs上过滤包含某个字符的行内容

[hadoop@hadoop-01 bin]$ ./hdfs dfs -cat /logs/* | grep 过滤字段

 

参考文档:http://blog.csdn.net/zhaojw_420/article/details/53161624