hadoop常见错误集锦

时间:2024-01-15 08:56:20

1  Hadoop namenode -format后Incompatible namespaceIDS 错误解决办法

重新格式话namenode后,会重新分配一个namespaceId, 这个namespaceID就会和存储在datanode上的namespaceID不一致,

解决办法:

a  修改datanode中的namespaceID, 在“dfs.data.dir”/current/VERSION中进行修改,新的namespaceID可以在log中寻找;

b  删除datanode,  dfs.data.dir中的文件,或者更改dfs.data.dir目录为新目录  

至于如何恢复系统的数据可以参考该文章: http://www.cnblogs.com/Richardzhu/p/3435989.html

2   SafeModeException

error: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /home/feixiang/hadoop-0.20.2/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /home/feixiang/hadoop-0.20.2/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. Safe mode will be turned off automatically.

原因:namenode每次启动的时候,会进入安全模式,接受datanode上报块数,如果上报的块数损失率达到某个数值(默认是1-0.9990),会一直处于只读不能写的安全模式。namenode每次启动的时候为什么要删除mapred/system文件???

有两种方式离开安全模式:

a  在hdfs-site.xml中, 修改dfs.safemode.threshold.pct的值,但是具体为多少,要看log里面的ratio of reported blocks

b  强制离开。命令: hadoop dfsadmin -safemode leave

参考:

http://www.linuxidc.com/Linux/2012-12/76782.htm

http://blog.csdn.net/rzhzhz/article/details/7056793

 

3

2014-03-28 21:15:26,118 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9003, call addBlock(/home/kanglishan/hadoop/hadoop-1.2.1/mapred/system/jobtracker.info, DFSClient_NONMAPREDUCE_-690631423_1, null) from 127.0.0.1:36367: error: java.io.IOException: File /home/kanglishan/hadoop/hadoop-1.2.1/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
java.io.IOException: File /home/kanglishan/hadoop/hadoop-1.2.1/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

mapred文件夹下没有建立system目录,namenode -format再启动就会出现这个问题,建立这个目录后,重新格式话,再启动就ok,不知道什么原因。