基于zookeeper的高可用Hadoop HA集群安装

时间：2023-03-09 03:32:48
基于zookeeper的高可用Hadoop HA集群安装
(1)hadoop2.7.1源码编译 http://aperise.iteye.com/blog/2246856
(2)hadoop2.7.1安装准备 http://aperise.iteye.com/blog/2253544
(3)1.x和2.x都支持的集群安装 http://aperise.iteye.com/blog/2245547
(4)hbase安装准备 http://aperise.iteye.com/blog/2254451
(5)hbase安装 http://aperise.iteye.com/blog/2254460
(6)snappy安装 http://aperise.iteye.com/blog/2254487
(7)hbase性能优化 http://aperise.iteye.com/blog/2282670
(8)雅虎YCSBC测试hbase性能测试 http://aperise.iteye.com/blog/2248863
(9)spring-hadoop实战 http://aperise.iteye.com/blog/2254491
(10)基于ZK的Hadoop HA集群安装  http://aperise.iteye.com/blog/2305809

1.Hadoop集群方式介绍

1.1 hadoop1.x和hadoop2.x都支持的namenode+secondarynamenode方式

基于zookeeper的高可用Hadoop HA集群安装
         优点：搭建环境简单，适合开发者模式下调试程序

缺点：namenode作为很重要的服务，存在单点故障，如果namenode出问题，会导致整个集群不可用

1.2.仅hadoop2.x支持的active namenode+standby namenode方式

基于zookeeper的高可用Hadoop HA集群安装
       优点：为解决1.x中namenode单节点故障而生，充分保障Hadoop集群的高可用

缺点：需要zookeeper最少3台，需要journalnode最少三台，目前最多支持2台namenode，不过节点可以复用，但是不建议

1.3 Hadoop官网关于集群方式介绍

1）单机Hadoop环境搭建

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html

2）集群方式

集群方式一(hadoop1.x和hadoop2.x都支持的namenode+secondarynamenode方式)

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/ClusterSetup.html

集群方式二(仅hadoop2.x支持的active namenode+standby namenode方式，也叫HADOOP HA方式)，这种方式又分为HDFS的HA和YARN的HA单独分开讲解。

HDFS HA(zookeeper+journalnode)http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html

HDFS HA(zookeeper+NFS)http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailability

YARN HA(zookeeper)http://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html

生产环境多采用HDFS(zookeeper+journalnode)(active NameNode+standby NameNode+JournalNode+DFSZKFailoverController+DataNode)+YARN(zookeeper)(active ResourceManager+standby ResourceManager+NodeManager)方式，这里我讲解的是仅hadoop2.x支持基于zookeeper的Hadoop HA集群方式，这种方式主要适用于生产环境。

2.基于zookeeper的Hadoop HA集群安装

2.1 安装环境介绍

基于zookeeper的高可用Hadoop HA集群安装

2.2 安装前准备工作

1）关闭防火墙

centos7防火墙操作介绍 
#centos7启动firewall
systemctl start firewalld.service
#centos7重启firewall
systemctl restart firewalld.service
#centos7停止firewall
systemctl stop firewalld.service 
#centos7禁止firewall开机启动
systemctl disable firewalld.service 
#centos7查看防火墙状态
firewall-cmd --state
#开放防火墙端口
vi /etc/sysconfig/iptables-config
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 6379 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 6380 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 6381 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 16379 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 16380 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 16381 -j ACCEPT

这里我关闭防火墙，root下执行如下命令：

systemctl stop firewalld.service 
systemctl disable firewalld.service

2）优化selinux

作用：Hadoop主节点管理子节点是通过SSH实现的， SELinux不关闭的情况下无法实现，会限制ssh免密码登录。

编辑/etc/selinux/config，修改前：

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected. 
# mls - Multi Level Security protection.
SELINUXTYPE=targeted

修改后：

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
#SELINUX=enforcing
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected. 
# mls - Multi Level Security protection.
#SELINUXTYPE=targeted

执行以下命令使selinux 修改立即生效：

setenforce 0

3）机器名配置

作用：Hadoop集群中机器IP可能变化导致集群间服务中断，所以在Hadoop中最好以机器名进行配置。

修改各机器上文件/etc/hostname，配置主机名称如下：

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.185.31 hadoop31
192.168.185.32 hadoop32
192.168.185.33 hadoop33
192.168.185.34 hadoop34
192.168.185.35 hadoop35

而centos7下各个机器的主机名设置文件为/etc/hostname，以hadoop31节点主机配置为例，配置如下：

#localdomain
hadoop31

4)创建hadoop用户和组

作用：后续单独以用户hadoop来管理Hadoop集群,防止其他用户误操作关闭Hadoop 集群

#以root用户创建hadoop用户和组创建hadoop用户和组 
groupadd hadoop 
useradd -g hadoop hadoop 
#修改用户密码
passwd hadoop

5）用户hadoop免秘钥登录

作用：Hadoop中主节点管理从节点是通过SSH协议登录到从节点实现的，而一般的SSH登录，都是需要输入密码验证的，为了Hadoop主节点方便管理成千上百的从节点，这里将主节点公钥拷贝到从节点，实现SSH协议免秘钥登录，我这里做的是所有主从节点之间机器免秘钥登录

#首先切换到上面的hadoop用户,这里我是在hadoop31机器上操作 
ssh hadoop31
su hadoop 
#生成非对称公钥和私钥，这个在集群中所有节点机器都必须执行,一直回车就行 
ssh-keygen -t rsa 
#通过ssh登录远程机器时，本机会默认将当前用户目录下的.ssh/authorized_keys带到远程机器进行验证，这里是/home/hadoop/.ssh/authorized_keys中公钥(来自其他机器上的/home/hadoop/.ssh/id_rsa.pub.pub),以下代码只在主节点执行就可以做到主从节点之间SSH免密码登录 
cd /home/hadoop/.ssh/ 
#首先将Master节点的公钥添加到authorized_keys 
cat id_rsa.pub>>authorized_keys 
#其次将Slaves节点的公钥添加到authorized_keys，这里我是在Hadoop31机器上操作的 
ssh hadoop@192.168.185.32 cat /home/hadoop/.ssh/id_rsa.pub>> authorized_keys 
ssh hadoop@192.168.185.33 cat /home/hadoop/.ssh/id_rsa.pub>> authorized_keys 
ssh hadoop@192.168.185.34 cat /home/hadoop/.ssh/id_rsa.pub>> authorized_keys 
ssh hadoop@192.168.185.35 cat /home/hadoop/.ssh/id_rsa.pub>> authorized_keys 
#必须设置修改/home/hadoop/.ssh/authorized_keys权限 
chmod 600 /home/hadoop/.ssh/authorized_keys 
#这里将Master节点的authorized_keys分发到其他slaves节点 
scp -r /home/hadoop/.ssh/authorized_keys hadoop@192.168.185.32:/home/hadoop/.ssh/ 
scp -r /home/hadoop/.ssh/authorized_keys hadoop@192.168.185.33:/home/hadoop/.ssh/ 
scp -r /home/hadoop/.ssh/authorized_keys hadoop@192.168.185.34:/home/hadoop/.ssh/ 
scp -r /home/hadoop/.ssh/authorized_keys hadoop@192.168.185.35:/home/hadoop/.ssh/

6）JDK安装

作用：Hadoop需要java环境支撑，而Hadoop2.7.1最少需要java版本1.7，安装如下：

#登录到到到hadoop用户下
su hadoop
#下载jdk-7u65-linux-x64.gz放置于/home/hadoop/java并解压
cd /home/hadoop/java
tar -zxvf jdk-7u65-linux-x64.gz
#编辑vi /home/hadoop/.bashrc,在文件末尾追加如下内容
export JAVA_HOME=/home/hadoop/java/jdk1.7.0_65 
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar 
export PATH=$PATH:$JAVA_HOME/bin 
#使得/home/hadoop/.bashrc配置生效
source /home/hadoop/.bashrc

很多人是配置linux全局/etc/profile，这里不建议这么做，一旦有人在里面降级了java环境或者删除了java环境，就会出问题，建议的是在管理Hadoop集群的用户下面修改其.bashrc单独配置该用户环境变量

7）zookeeper安装

#1登录hadoop用户并下载并解压zookeeper3.4.6
su hadoop
cd /home/hadoop 
tar -zxvf zookeeper-3.4.6.tar.gz

#2在集群中各个节点中配置/etc/hosts，内容如下：
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.185.31 hadoop31 
192.168.185.32 hadoop32 
192.168.185.33 hadoop33 
192.168.185.34 hadoop34 
192.168.185.35 hadoop35

#3在集群中各个节点中创建zookeeper数据文件
ssh hadoop31
cd /home/hadoop 
#zookeeper数据存放位置
mkdir -p /opt/hadoop/zookeeper 
ssh hadoop32
cd /home/hadoop 
#zookeeper数据存放位置
mkdir -p /opt/hadoop/zookeeper 
ssh hadoop33
cd /home/hadoop 
#zookeeper数据存放位置
mkdir -p /opt/hadoop/zookeeper 
ssh hadoop34
cd /home/hadoop 
#zookeeper数据存放位置
mkdir -p /opt/hadoop/zookeeper 
ssh hadoop35
cd /home/hadoop 
#zookeeper数据存放位置
mkdir -p /opt/hadoop/zookeeper

#4配置zoo.cfg
ssh hadoop31
cd /home/hadoop/zookeeper-3.4.6/conf
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
#内容如下
initLimit=10 
syncLimit=5 
dataDir=/opt/hadoop/zookeeper 
clientPort=2181 

#数据文件保存最近的3个快照，默认是都保存，时间长的话会占用很大磁盘空间
autopurge.snapRetainCount=3
#单位为小时，每小时清理一次快照数据
autopurge.purgeInterval=1
server.1=hadoop31:2888:3888 
server.2=hadoop32:2888:3888 
server.3=hadoop33:2888:3888
server.4=hadoop34:2888:3888 
server.5=hadoop35:2888:3888 
#5在hadoop31上远程复制分发安装文件
scp -r /home/hadoop/zookeeper-3.4.6 hadoop@hadoop32:/home/hadoop/ 
scp -r /home/hadoop/zookeeper-3.4.6 hadoop@hadoop33:/home/hadoop/ 
scp -r /home/hadoop/zookeeper-3.4.6 hadoop@hadoop34:/home/hadoop/ 
scp -r /home/hadoop/zookeeper-3.4.6 hadoop@hadoop35:/home/hadoop/

#6在集群中各个节点设置myid必须为数字 
ssh hadoop31 
echo "1" > /opt/hadoop/zookeeper/myid 
ssh hadoop32 
echo "2" > /opt/hadoop/zookeeper/myid 
ssh hadoop33 
echo "3" > /opt/hadoop/zookeeper/myid

#7.各个节点如何启动zookeeper
ssh hadoop31
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start

#8.各个节点如何关闭zookeeper
ssh hadoop31
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh stop

#9.各个节点如何查看zookeeper状态
ssh hadoop31
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh status

#10.各个节点如何通过客户端访问zookeeper上目录数据
ssh hadoop31
/home/hadoop/zookeeper-3.4.6/bin/zkCli.sh -server hadoop31:2181,hadoop32:2181,hadoop33:2181,hadoop34:2181,hadoop35:2181

2.3 Hadoop HA安装

1）hadoop-2.7.1.tar.gz

#下载hadoop-2.7.1.tar.gz放置于/home/hadoop下并解压,这里我在hadoop31操作
ssh hadoop31
su hadoop
cd /home/hadoop
tar –zxvf hadoop-2.7.1.tar.gz

2）core-site.xml

修改配置文件/home/hadoop/hadoop-2.7.1/etc/hadoop/core-site.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4. <!-- 开启垃圾回收站功能,HDFS文件删除后先进入垃圾回收站，垃圾回收站最长保留数据时间为1天，超过一天后就删除 -->
  5. <property>
  6. <name>fs.trash.interval</name>
  7. <value>1440</value>
  8. </property>
  9. <!-- Hadoop HA部署方式下namenode访问地址，bigdatacluster-ha是名字可自定义，后面hdfs-site.xml会用到 -->
  10. <property>
  11. <name>fs.defaultFS</name>
  12. <value>hdfs:// bigdatacluster-ha</value>
  13. </property>
  14. <!--hadoop访问文件的IO操作都需要通过代码库。因此，在很多情况下，io.file.buffer.size都被用来设置SequenceFile中用到的读/写缓存大小。不论是对硬盘或者是网络操作来讲，较大的缓存都可以提供更高的数据传输，但这也就意味着更大的内存消耗和延迟。这个参数要设置为系统页面大小的倍数，以byte为单位，默认值是4KB，一般情况下，可以设置为64KB（65536byte）,这里设置128K-->
  15. <property>
  16. <name>io.file.buffer.size</name>
  17. <value>131072</value>
  18. </property>
  19. <!-- 指定hadoop临时目录 -->
  20. <property>
  21. <name>hadoop.tmp.dir</name>
  22. <value>/opt/hadoop/tmp</value>
  23. </property>
  24. <!-- 指定zookeeper地址 -->
  25. <property>
  26. <name>ha.zookeeper.quorum</name>
  27. <value>hadoop31:2181,hadoop32:2181,hadoop33:2181,hadoop34:2181,hadoop35:2181</value>
  28. </property>
  29. <property>
  30. <name>ha.zookeeper.session-timeout.ms</name>
  31. <value>300000</value>
  32. </property>
  33. <!-- 指定Hadoop压缩格式，Apache官网下载的安装包不支持snappy，需要自己编译安装，如何编译安装包我在博客http://aperise.iteye.com/blog/2254487有讲解,不适用snappy的话可以不配置 -->
  34. <property>
  35. <name>io.compression.codecs</name>
  36. <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  37. </property>
  38. </configuration>

3）hdfs-site.xml

修改配置文件/home/hadoop/hadoop-2.7.1/etc/hadoop/hdfs-site.xml

  1. <?xml version="1.0" encoding="UTF-8"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4. <!--指定hdfs的nameservice为bigdatacluster-ha，需要和core-site.xml中的保持一致 -->
  5. <property>
  6. <name>dfs.nameservices</name>
  7. <value>bigdatacluster-ha</value>
  8. </property>
  9. <!—指定磁盘预留多少空间，防止磁盘被撑满用完，单位为bytes -->
  10. <property>
  11. <name>dfs.datanode.du.reserved</name>
  12. <value>107374182400</value>
  13. </property>
  14. <!-- bigdatacluster-ha下面有两个NameNode，分别是namenode1，namenode2 -->
  15. <property>
  16. <name>dfs.ha.namenodes.bigdatacluster-ha</name>
  17. <value>namenode1,namenode2</value>
  18. </property>
  19. <!-- namenode1的RPC通信地址,这里端口要和core-site.xml中fs.defaultFS保持一致 -->
  20. <property>
  21. <name>dfs.namenode.rpc-address.bigdatacluster-ha.namenode1</name>
  22. <value>hadoop31:9000</value>
  23. </property>
  24. <!-- namenode1的http通信地址 -->
  25. <property>
  26. <name>dfs.namenode.http-address.bigdatacluster-ha.namenode1</name>
  27. <value>hadoop31:50070</value>
  28. </property>
  29. <!-- namenode2的RPC通信地址,这里端口要和core-site.xml中fs.defaultFS保持一致 -->
  30. <property>
  31. <name>dfs.namenode.rpc-address.bigdatacluster-ha.namenode2</name>
  32. <value>hadoop32:9000</value>
  33. </property>
  34. <!-- namenode2的http通信地址 -->
  35. <property>
  36. <name>dfs.namenode.http-address.bigdatacluster-ha.namenode2</name>
  37. <value>hadoop32:50070</value>
  38. </property>
  39. <!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
  40. <property>
  41. <name>dfs.namenode.shared.edits.dir</name>
  42. <value>qjournal://hadoop31:8485;hadoop32:8485;hadoop33:8485;hadoop34:8485;hadoop35:8485/bigdatacluster-ha</value>
  43. </property>
  44. <!-- 配置失败自动切换实现方式 -->
  45. <property>
  46. <name>dfs.client.failover.proxy.provider.bigdatacluster-ha</name>
  47. <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
  48. </property>
  49. <!-- 配置隔离机制,主要用户远程管理监听其他机器相关服务 -->
  50. <property>
  51. <name>dfs.ha.fencing.methods</name>
  52. <value>sshfence</value>
  53. </property>
  54. <!-- 使用隔离机制时需要ssh免密码登陆 -->
  55. <property>
  56. <name>dfs.ha.fencing.ssh.private-key-files</name>
  57. <value>/home/hadoop/.ssh/id_rsa</value>
  58. </property>
  59. <!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
  60. <property>
  61. <name>dfs.journalnode.edits.dir</name>
  62. <value>/opt/hadoop/journal</value>
  63. </property>
  64. <!--指定支持高可用自动切换机制-->
  65. <property>
  66. <name>dfs.ha.automatic-failover.enabled</name>
  67. <value>true</value>
  68. </property>
  69. <!--指定namenode名称空间的存储地址-->
  70. <property>
  71. <name>dfs.namenode.name.dir</name>
  72. <value>file:/opt/hadoop/hdfs/name</value>
  73. </property>
  74. <!--指定datanode数据存储地址-->
  75. <property>
  76. <name>dfs.datanode.data.dir</name>
  77. <value>file:/opt/hadoop/hdfs/data</value>
  78. </property>
  79. <!--指定数据冗余份数-->
  80. <property>
  81. <name>dfs.replication</name>
  82. <value>3</value>
  83. </property>
  84. <!--指定可以通过web访问hdfs目录-->
  85. <property>
  86. <name>dfs.webhdfs.enabled</name>
  87. <value>true</value>
  88. </property>
  89. <property>
  90. <name>ha.zookeeper.quorum</name>
  91. <value>hadoop31:2181,hadoop32:2181,hadoop33:2181,hadoop34:2181,hadoop35:2181</value>
  92. </property>
  93. <property>
  94. <name>dfs.namenode.handler.count</name>
  95. <value>600</value>
  96. <description>The number of server threads for the namenode.</description>
  97. </property>
  98. <property>
  99. <name>dfs.datanode.handler.count</name>
  100. <value>600</value>
  101. <description>The number of server threads for the datanode.</description>
  102. </property>
  103. <property>
  104. <name>dfs.client.socket-timeout</name>
  105. <value>600000</value>
  106. </property>
  107. <property>
  108. <!--这里设置Hadoop允许打开最大文件数，默认4096，不设置的话会提示xcievers exceeded错误-->
  109. <name>dfs.datanode.max.transfer.threads</name>
  110. <value>409600</value>
  111. </property>
  112. </configuration>

4）mapred-site.xml

修改配置文件/home/hadoop/hadoop-2.7.1/etc/hadoop/mapred-site.xml

  1. <?xml version="1.0"?>
  2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
  3. <configuration>
  4. <!-- 配置MapReduce运行于yarn中 -->
  5. <property>
  6. <name>mapreduce.framework.name</name>
  7. <value>yarn</value>
  8. </property>
  9. <property>
  10. <name>mapreduce.job.maps</name>
  11. <value>12</value>
  12. </property>
  13. <property>
  14. <name>mapreduce.job.reduces</name>
  15. <value>12</value>
  16. </property>
  17. <!-- 指定Hadoop压缩格式，Apache官网下载的安装包不支持snappy，需要自己编译安装，如何编译安装包我在博客http://aperise.iteye.com/blog/2254487有讲解,不适用snappy的话可以不配置 -->
  18. <property>
  19. <name>mapreduce.output.fileoutputformat.compress</name>
  20. <value>true</value>
  21. <description>Should the job outputs be compressed?
  22. </description>
  23. </property>
  24. <property>
  25. <name>mapreduce.output.fileoutputformat.compress.type</name>
  26. <value>RECORD</value>
  27. <description>If the job outputs are to compressed as SequenceFiles, how should
  28. they be compressed? Should be one of NONE, RECORD or BLOCK.
  29. </description>
  30. </property>
  31. <property>
  32. <name>mapreduce.output.fileoutputformat.compress.codec</name>
  33. <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  34. <description>If the job outputs are compressed, how should they be compressed?
  35. </description>
  36. </property>
  37. <property>
  38. <name>mapreduce.map.output.compress</name>
  39. <value>true</value>
  40. <description>Should the outputs of the maps be compressed before being
  41. sent across the network. Uses SequenceFile compression.
  42. </description>
  43. </property>
  44. <property>
  45. <name>mapreduce.map.output.compress.codec</name>
  46. <value>org.apache.hadoop.io.compress.SnappyCodec</value>
  47. <description>If the map outputs are compressed, how should they be
  48. compressed?
  49. </description>
  50. </property>
  51. </configuration>

5）yarn-site.xml

修改配置文件/home/hadoop/hadoop-2.7.1/etc/hadoop/yarn-site.xml

  1. <?xml version="1.0"?>
  2. <configuration>
  3. <!--日志聚合功能yarn.log start------------------------------------------------------------------------>
  4. <property>
  5. <name>yarn.log-aggregation-enable</name>
  6. <value>true</value>
  7. </property>
  8. <!--在HDFS上聚合的日志最长保留多少秒。3天-->
  9. <property>
  10. <name>yarn.log-aggregation.retain-seconds</name>
  11. <value>259200</value>
  12. </property>
  13. <!--日志聚合功能yarn.log end-------------------------------------------------------------------------->
  14. <!--resourcemanager失联后重新链接的时间-->
  15. <property>
  16. <name>yarn.resourcemanager.connect.retry-interval.ms</name>
  17. <value>2000</value>
  18. </property>
  19. <!--配置resourcemanager start------------------------------------------------------------------------->
  20. <property>
  21. <name>yarn.resourcemanager.zk-address</name>
  22. <value>hadoop31:2181,hadoop32:2181,hadoop33:2181,hadoop34:2181,hadoop35:2181</value>
  23. </property>
  24. <property>
  25. <name>yarn.resourcemanager.cluster-id</name>
  26. <value>besttonecluster-yarn</value>
  27. </property>
  28. <!--开启resourcemanager HA,默认为false-->
  29. <property>
  30. <name>yarn.resourcemanager.ha.enabled</name>
  31. <value>true</value>
  32. </property>
  33. <property>
  34. <name>yarn.resourcemanager.ha.rm-ids</name>
  35. <value>rm1,rm2</value>
  36. </property>
  37. <property>
  38. <name>yarn.resourcemanager.hostname.rm1</name>
  39. <value>hadoop31</value>
  40. </property>
  41. <property>
  42. <name>yarn.resourcemanager.hostname.rm2</name>
  43. <value>hadoop32</value>
  44. </property>
  45. <!--配置rm1-->
  46. <property>
  47. <name>yarn.resourcemanager.webapp.address.rm1</name>
  48. <value>hadoop31:8088</value>
  49. </property>
  50. <!--配置rm2-->
  51. <property>
  52. <name>yarn.resourcemanager.webapp.address.rm2</name>
  53. <value>hadoop32:8088</value>
  54. </property>
  55. <!--开启故障自动切换-->
  56. <property>
  57. <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
  58. <value>true</value>
  59. </property>
  60. <property>
  61. <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
  62. <value>true</value>
  63. </property>
  64. <property>
  65. <name>yarn.resourcemanager.ha.automatic-failover.zk-base-path</name>
  66. <value>/yarn-leader-election</value>
  67. </property>
  68. <!--开启自动恢复功能-->
  69. <property>
  70. <name>yarn.resourcemanager.recovery.enabled</name>
  71. <value>true</value>
  72. </property>
  73. <property>
  74. <name>yarn.resourcemanager.store.class</name>
  75. <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
  76. </property>
  77. <!--配置resourcemanager end--------------------------------------------------------------------------->
  78. <!--配置nodemanager start----------------------------------------------------------------------------->
  79. <property>
  80. <name>yarn.nodemanager.aux-services</name>
  81. <value>mapreduce_shuffle</value>
  82. </property>
  83. <property>
  84. <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
  85. <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  86. </property>
  87. <!--配置nodemanager end------------------------------------------------------------------------------->
  88. </configuration>

6）slaves

修改配置文件/home/hadoop/hadoop-2.7.1/etc/hadoop/slaves

Hadoop31
Hadoop32
Hadoop33
Hadoop34
Hadoop35

7）hadoop-env.sh和yarn-env.sh

在/home/hadoop/hadoop-2.7.1/etc/hadoop/hadoop-env.sh和/home/hadoop/hadoop-2.7.1/etc/hadoop/yarn-env.sh中配置JAVA_HOME

export JAVA_HOME=/home/hadoop/java/jdk1.7.0_65

8）bashrc

当前用户hadoop生效，在用户目录下/home/hadoop/.bashrc增加如下配置

export HADOOP_HOME=/home/hadoop/hadoop2.7.1
export PATH=${HADOOP_HOME}/bin:${PATH}

9）分发安装文件到其他机器

#这里我是在hadoop31上操作
scp -r /home/hadoop/hadoop-2.7.1 hadoop@hadoop32:/home/hadoop/
scp -r /home/hadoop/hadoop-2.7.1 hadoop@ hadoop33:/home/hadoop/
scp -r /home/hadoop/hadoop-2.7.1 hadoop@ hadoop34:/home/hadoop/ 
scp -r /home/hadoop/hadoop-2.7.1 hadoop@ hadoop35:/home/hadoop/

2.4 Hadoop HA初次启动

1）启动zookeeper

ssh hadoop31
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start
ssh hadoop32
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start
ssh hadoop33
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start
ssh hadoop34
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start
ssh hadoop35
/home/hadoop/zookeeper-3.4.6/bin/zkServer.sh start

#jps查看是否有QuorumPeerMain 进程

#/home/hadoop/zookeeper-3.4.6/ bin/zkServer.sh status查看zookeeper状态

#/home/hadoop/zookeeper-3.4.6/ bin/zkServer.sh stop关闭zookeeper

2）格式化zookeeper上hadoop-ha目录

/home/hadoop/hadoop-2.7.1/bin/hdfs zkfc –formatZK
#可以通过如下方法检查zookeeper上是否已经有Hadoop HA目录
# /home/hadoop/zookeeper-3.4.6/bin/zkCli.sh -server hadoop31:2181,hadoop32:2181,hadoop33:2181,hadoop34:2181,hadoop35:2181 
#ls /

3）启动namenode日志同步服务journalnode

ssh hadoop31
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start journalnode 
ssh hadoop32
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start journalnode 
ssh hadoop33
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start journalnode 
ssh hadoop34
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start journalnode 
ssh hadoop35
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start journalnode

4）格式化namenode

#这步操作只能在namenode服务节点hadoop31或者hadoop32执行中一台上执行
ssh hadoop31
/home/hadoop/hadoop-2.7.1/bin/hdfs namenode -format

5）启动namenode、同步备用namenode、启动备用namenode

#启动namenode
ssh hadoop31
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start namenode 
#同步备用namenode、启动备用namenode
ssh hadoop32
/home/hadoop/hadoop-2.7.1/bin/hdfs namenode -bootstrapStandby 
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start namenode

6)启动DFSZKFailoverController

ssh hadoop31
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start zkfc 
ssh hadoop32
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemon.sh start zkfc

7）启动datanode

#注意hadoop-daemons.sh datanode是启动所有datanode，而hadoop-daemon.sh datanode是启动单个datanode
ssh hadoop31
/home/hadoop/hadoop-2.7.1/sbin/hadoop-daemons.sh start datanode

8）启动yarn

#在hadoop31上启动resouremanager,在hadoop31，hadoop32，hadoop33，hadoop34，hadoop35上启动nodemanager
ssh hadoop31
/home/hadoop/hadoop-2.7.1/sbin/start-yarn.sh 
#在hadoop31上启动备用resouremanager
ssh hadoop32
/home/hadoop/hadoop-2.7.1/sbin/yarn-daemon.sh start resourcemanager

至此，Hadoop 基于zookeeper的高可用集群就安装成功，并且启动了。

相关文章