http://blog.itpub.net/26230597/cid-87082-list-2/ 1，简介
.1mha简介
MHA，即MasterHigh Availability Manager and Tools for MySQL，是日本的一位MySQL专家采用Perl语言编写的一个脚本管理工具，该工具仅适用于MySQLReplication（二层）环境，目的在于维持Master主库的高可用性。 MHA(Master High Availability)是自动的master故障转移和Slave提升的软件包.它是基于标准的MySQL复制(异步/半同步).
MHA有两部分组成:MHA Manager(管理节点)和MHA Node(数据节点).
MHA Manager可以单独部署在一*立机器上管理多个master-slave集群,也可以部署在一台slave上.MHA Manager探测集群的node节点,当发现master出现故障的时候,它可以自动将具有最新数据的slave提升为新的master,然后将所有其它的slave导向新的master上.整个故障转移过程对应用程序是透明的。
MHA node运行在每台MySQL服务器上(master/slave/manager),它通过监控具备解析和清理logs功能的脚本来加快故障转移的。 1.2，mha特点
（）. -30s实现master failover（-12s可以检测到主机故障，-10s可以关闭主机避免SB，在用很短的时间应用差异日志） （） 部署简单，无需对现有M-S结构做任何改动（至少3台，保证切换后仍保持M-S结构） （）. 支持手动在线切换（主机硬件维护），downtime几乎很短0.-2s （）. 保证故障切换后多从库数据的一致性 （）. 完全自动化的failover及快速复制架构恢复方案（一主多从） （）. 恢复过程包括：选择新主库、确认从库间relaylog差异、新主库应用必要语句、其他从库同步差异语句、重新建立复制连接 ，工作原理 相较于其它HA软件，MHA的目的在于维持MySQL Replication中Master库的高可用性，其最大特点是可以修复多个Slave之间的差异日志，最终使所有Slave保持数据一致，然后从中选择一个充当新的Master，并将其它Slave指向它。
-从宕机崩溃的master保存二进制日志事件(binlogevents)。
-识别含有最新更新的slave。
-应用差异的中继日志(relay log)到其它slave。
-应用从master保存的二进制日志事件(binlogevents)。
-提升一个slave为新master。
-使其它的slave连接新的master进行复制。 ，mha工具包
()、 Manager工具:
- masterha_check_ssh : 检查MHA的SSH配置。
- masterha_check_repl : 检查MySQL复制。
- masterha_manager : 启动MHA。
- masterha_check_status : 检测当前MHA运行状态。
- masterha_master_monitor : 监测master是否宕机。
- masterha_master_switch : 控制故障转移(自动或手动)。
- masterha_conf_host : 添加或删除配置的server信息。 ()、 Node工具(这些工具通常由MHAManager的脚本触发,无需人手操作)。
- save_binary_logs : 保存和复制master的二进制日志。
- apply_diff_relay_logs : 识别差异的中继日志事件并应用于其它slave。
- filter_mysqlbinlog : 去除不必要的ROLLBACK事件(MHA已不再使用这个工具)。
- purge_relay_logs : 清除中继日志(不会阻塞SQL线程)。 ，主机部署
Ip地址
mha状况
Mysql状况
192.168.52.129
Node
Master
192.168.52.130
Node
Slave
192.168.52.131
Manager
Slave ，生成ssh无密钥证书
5.1 先在192.168.52.129上面生成密钥
ssh-keygen -t dsa -P '' -f id_dsa
Id_dsa.pub为公钥，id_dsa为私钥，紧接着将公钥文件复制成authorized_keys文件，这个步骤是必须的，过程如下： cat id_dsa.pub >> authorized_keys 5.2 在192.168.52.130上面生产密钥
ssh-keygen -t dsa -P '' -f id_dsa
cat id_dsa.pub >> authorized_keys 5.3 在192.168.52.131上面生产密钥
ssh-keygen -t dsa -P '' -f id_dsa
cat id_dsa.pub >> authorized_keys 5.4 构造3个通用的authorized_keys
在192.168.52.129上面操作：
cd /root/.ssh #copy130和131上面的密钥过来
scp 192.168.52.130:/root/.ssh/id_dsa.pub./id_dsa.pub.
scp 192.168.52.131:/root/.ssh/id_dsa.pub./id_dsa.pub.
cat id_dsa.pub. >> authorized_keys
cat id_dsa.pub. >> authorized_keys 查看生成的通用密钥
[root@data01 .ssh]# cat authorized_keys
ssh-dssAAAAB3NzaC1kc3MAAACBAKe9oTz+hQ3vAUE+x7s2EIT6RSrlPoD2VHuSsDo+ZmAyv6+DD6/eVhVXrCyqzjQPJa6UI0PYjlPMk2r2wqdvC/YqQaLhFuQmsBQwrVA2xNHwhB3ic+Om44GVoiZFM7+bGAtfhQ9DLK2+sjfaa/oQfuDvTJ2SI/f0oG3wDGmokgdLAAAAFQC/O4R1yX1FxW7+dCKHhV+LQHWWHQAAAIADR5yqZGZCx8VB8Q6kAYg3cuUCCo3gF/hA5bHY+2xO5CcGAD1mq/l5v55QPUlGAH7btdmfip1tiaQ+V3N+Ektf2psM0uap/LKvbV2jQYKc2UZrpfF6R0lG+x9rpkxWMce1TJ4yANGOasjNjUV6Lg0RVDxLiMT4Ja4+edQVduYt2AAAAIBPNfJlyglZ5CwsI+v753tD8WT4GaH8ssNLpIKrH9qJU6OuT9MmniKE1RqZr+e5MCvkUAAXHFPq0DhHQlPKWqhIpVlu0E8Zsn9a5tv728JpIYz1umB5VLo2J5fhpobefD3AhjEHSyaubJgQG4Gu+jdwsR0H21xLNx0VoP8YPbYkAQ==root@data01
ssh-dssAAAAB3NzaC1kc3MAAACBAPtU+mTL9an88U1wfSwxdlHBg9n8eB9l218sXGLHwJyxNJN0pq4iPCLMfRxhM6T30HxUnyhghxBF2XvkAkMEjZ+IuCoA0mwRi1CcYSMh72SXqfRdgE2GpRBZDRRhlCIE5feNswWZdC7fIDmgNHtK5CFmJLcl+9Bkb3kfGqu8JOxbAAAAFQDo2YRMd5ZsfBRvPZcCYWcOsuy2oQAAAIEA4pGH2w7luN9JhJ0V6sDUUySg4R488yloKR/k4yD33qPXTejCiDUKUbanMLs6obQqxpbVvlhRT6cyo/le7PO6H8IzRHjFy65EPL0omn7J06PmHBUMqCn4jXo27EGXlRLavnonUf3tFeaVo7GxXerj71NdBKkQX7e/bgzD4d5v0PMAAACBAIhx1X50hlYzqPEZEDXTJmntLRXsMB20DvngvUcQLlgLozwfaNdJAQuYSGqklXYTquSmsyaTNJsxj8EkKG4XbM/7hHtNbV8KuAMJGT4Dm7jEdiKClyle2tIvVtonYaL41KeZVdn6Lk4lRYIFxpDoQHKXXr+XEFhLjoniT8urPISlroot@data02
ssh-dss AAAAB3NzaC1kc3MAAACBAJtC3j4Gq+zR7adyKFco/1hELblR65Af+Cgi81mfL+sJAFuOqPcFvDANhymBQ9ltH1N2/eDq1CrD0U9tRMxSwBvgiWZW9brkMpl5ix6oJSCBHdyqL6iuREk7CZ3V/y7P2V+TDCc+am5onMWDG1Af9o6CeA7CP92CHaUoltcXd7L7AAAAFQCqpeVqjbEs/lC/J1utfWrpGDxt8QAAAIB1aeB6C3S9t0dU3bEZs06DaooO46foBFMOI7u0w7uEWvj48UQF7W5Y++vjsiARxr6clk2g2T70n0fJmZCtMMiHqD6vyy8OOy8PzdUcQVAUW2GZQ8hn7M1U2GOz2KPo6uUbPiKkXilEfh9YRsyZyxMdmC4raPjPea8sj6favK8RbgAAAIAima6hWfBFsFcE1oh02xk7ydavHGJbHAlxeH9g9glE0FPmzPgWMFkYQnLsyV2z+ouMPFmERUPYzg1k/8Dhoxjp9j4JB6bIcPNtKdlS660NcFLxRtFAhrnzvLOTzXYzeIuZOlE0WcjeQGNpx8JHAef/3WzpHnVmnhbmlkBrZ8X/OQ==root@oraclem1
[root@data01 .ssh]# 看到authorized_keys文件里面有3行记录，分别代表了访问data01（192.168.52.129），oraclem1（192.168.52.131），data02（192.168.52.130）的公用密钥。然后把这个authorized_keys公钥文件copy到oraclem1（192.168.52.131）和data02（192.168.52.130）上面同一个目录下。Scp命令如下：
scp authorized_keys192.168.52.:/root/.ssh/
scp authorized_keys192.168.52.:/root/.ssh/ 5.5 开始check验证如下：
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr :: from data01
[root@data02 ~]# ssh 192.168.52.131
Last login: Tue Apr :: from 192.168.52.131
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr :: from 192.168.52.131
[root@data02 ~]# ssh 192.168.52.129
Last login: Tue Apr :: from 192.168.52.131
[root@data01 ~]# ssh 192.168.52.131
Last login: Tue Apr :: from 192.168.52.130
[root@oraclem1 ~]# ssh 192.168.52.129
Last login: Tue Apr :: from data02
[root@data01 ~]# ssh 192.168.52.131
Last login: Tue Apr :: from 192.168.52.129
[root@oraclem1 ~]# ssh 192.168.52.130
Last login: Tue Apr :: from 192.168.52.131
[root@data02 ~]# ssh 192.168.52.129
Last login: Tue Apr :: from 192.168.52.131
[root@data01 ~]
OK，3台服务器已经能实现两两互相ssh通了，不需要输入密码即可。
PS：如果不能实现任何两台主机互相之间可以无密码登录，后面的环节可能会有问题。 5.6 实现主机名hostname登录
在3台服务器上，编辑/etc/hosts,追加入以下内容保存退出
192.168.52.129 data01
192.168.52.130 data02
192.168.52.131 oraclem1 验证主机名登录：
[root@data02 ~]# ssh oraclem1
The authenticity of host 'oraclem1(192.168.52.131)' can't be established.
RSA key fingerprint ise5:f0:ae:e3:::2f::1f::dd::c3:1a:e1:.
Are you sure you want to continueconnecting (yes/no)? yes
Warning: Permanently added 'oraclem1' (RSA)to the list of known hosts.
Last login: Tue Apr :: from data01
[root@oraclem1 ~]# ssh data02
Last login: Tue Apr :: from data01
[root@data02 ~]# ssh oraclem1
Last login: Tue Apr :: from data02
[root@oraclem1 ~]# vim /etc/hosts
[root@oraclem1 ~]# ssh data01
Last login: Tue Apr :: from data02
[root@data01 ~]# ssh data02
Last login: Tue Apr :: from oraclem1
[root@data02 ~]# ssh data01
Last login: Tue Apr :: from oraclem1
[root@data01 ~]#
OK，看到可以实现通过主机名来无密码登录了。 ，准备好mysql主从环境 具体搭建过程，参考以前的blog：http://blog.csdn.net/mchdba/article/details/44734597
架构如下，一主二从的架构
Master主库à192.168.52.，slave从库à192.168.52.
Master 主库à192.168.52.，slave从库à192.168.52. 创建用户mha管理的账号，在所有mysql服务器上都需要执行：
GRANT SUPER,RELOAD,REPLICATIONCLIENT,SELECT ON *.* TO manager@'192.168.52.%' IDENTIFIED BY 'manager_1234';
GRANT CREATE,INSERT,UPDATE,DELETE,DROP ON*.* TO manager@'192.168.52.%'; 创建主从账号，在所有mysql服务器上都需要执行：
GRANT RELOAD, SUPER, REPLICATION SLAVE ON*.* TO 'repl'@'192.168.52.%' IDENTIFIED BY 'repl_1234'; ，开始安装mha
mha包括manager节点和data节点，data节点包括原有的MySQL复制结构中的主机，至少3台，即1主2从，当masterfailover后，还能保证主从结构；只需安装node包。manager server：运行监控脚本，负责monitoring 和 auto-failover；需要安装node包和manager包。 为了节省机器，可以从现有复制架构中选一台“闲置”从库作为manager server，比如：某台从库不对外提供读的服务，只是作为候选主库，或是专门用于备份。
7.1，在数据节点上安装mha
首先安装yum -y install perl-DBD-MySQL
tar -zxvpf mha4mysql-node-0.56.tar.gz
perl Makefile.PL
make && make install 7.2 在管理节点上安装mha
首先安装perl的mysql包：
yum install -y perl-DBD-MySQL
yum install -y perl-Config-Tiny
yum install -y perl-Log-Dispatch
yum install -y perl-Parallel-ForkManager
yum install -y perl-Config-IniFiles
一些安装失败，可以直接下载rpm包安装：
wgetftp://ftp.muug.mb.ca/mirror/centos/5.10/os/x86_64/CentOS/perl-5.8.8-41.el5.x86_64.rpm
wgetftp://ftp.muug.mb.ca/mirror/centos/6.5/os/x86_64/Packages/compat-db43-4.3.29-15.el6.x86_64.rpm
wgethttp://downloads.naulinux.ru/pub/NauLinux/6x/i386/sites/School/RPMS/perl-Log-Dispatch-2.27-1.el6.noarch.rpm wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Parallel-ForkManager-0.7.9-1.el6.noarch.rpm wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Mail-Sender-0.8.16-3.el6.noarch.rpm wget http://dl.fedoraproject.org/pub/epel/6/i386/perl-Mail-Sendmail-0.79-12.el6.noarch.rpm wget http://mirror.centos.org/centos/6/os/x86_64/Packages/perl-Time-HiRes-1.9721-136.el6.x86_64.rpm 下载完后，一个个rpm安装好
如果最后还是安装不好，可以尝试一下perl CPAN的方式：
perl -MCPAN -e shell
cpan[]> install Log::Dispatch 然后通过perlMakefile.PL检查mha的perl安装环境，如下所示：
[root@oraclem1 mha4mysql-manager-0.56]#perl Makefile.PL
*** Module::AutoInstall version 1.03
*** Checking for Perl dependencies...
[Core Features]
- DBI ...loaded. (1.609)
- DBD::mysql ...loaded. (4.013)
- Time::HiRes ...loaded. (1.9726)
- Config::Tiny ...loaded. (2.12)
- Log::Dispatch ...loaded. (2.44)
- Parallel::ForkManager ...loaded. (0.7.)
- MHA::NodeConst ...loaded. (0.56)
*** Module::AutoInstall configurationfinished.
Generating a Unix-style Makefile
Writing Makefile for mha4mysql::manager
Writing MYMETA.yml and MYMETA.json
[root@oraclem1 mha4mysql-manager-0.56]# 然后解压缩安装：
tar -xvf mha4mysql-manager-0.56.tar.gz
cd mha4mysql-manager-0.56
perl Makefile.PL
make && make install 7.3 编辑管理节点配置 在管理节点192.168.52.129上面 [root@data01 mha4mysql-manager-0.56]# vim/etc/masterha/app1.cnf [server default]
manager_workdir=/var/log/masterha/app1
manager_log=/var/log/masterha/app1/manager.log ssh_user=root #ssh免密钥登录的帐号名
repl_user=repl #mysql复制帐号，用来在主从机之间同步二进制日志等
repl_password=repl_1234
ping_interval= #ping间隔，用来检测master是否正常 [server1]
hostname=192.168.52.129
candidate_master= #master机宕掉后,优先启用这台作为新master
master_binlog_dir=/home/data/mysql/binlog/ [server2]
hostname=192.168.52.130
#candidate_master=
master_binlog_dir=/home/data/mysql/binlog/ [server3]
hostname=192.168.52.131
#candidate_master=
master_binlog_dir=/home/data/mysql/binlog/ #[server4]
#hostname=host4
#no_master= 7.4 利用mha工具检测ssh
然后check ssh成功，采用命令：masterha_check_ssh --conf=/etc/masterha/app1.cnf，如下所示：
[root@data01 ~]# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Tue Apr :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Apr :: - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Tue Apr :: - [info] Reading server configuration from /etc/masterha/app1.cnf..
Tue Apr :: - [info] Starting SSH connection tests..
Tue Apr :: - [debug]
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.129(192.168.52.129:) to root@192.168.52.130(192.168.52.130:)..
Warning: Permanently added '192.168.52.129' (RSA) to the list of known hosts.
Tue Apr :: - [debug] ok.
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.129(192.168.52.129:) to root@192.168.52.131(192.168.52.131:)..
Tue Apr :: - [debug] ok.
Tue Apr :: - [debug]
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.130(192.168.52.130:) to root@192.168.52.129(192.168.52.129:)..
Tue Apr :: - [debug] ok.
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.130(192.168.52.130:) to root@192.168.52.131(192.168.52.131:)..
Tue Apr :: - [debug] ok.
Tue Apr :: - [debug]
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.131(192.168.52.131:) to root@192.168.52.129(192.168.52.129:)..
Tue Apr :: - [debug] ok.
Tue Apr :: - [debug] Connecting via SSH from root@192.168.52.131(192.168.52.131:) to root@192.168.52.130(192.168.52.130:)..
Tue Apr :: - [debug] ok.
Tue Apr :: - [info] All SSH connection tests passed successfully.
[root@data01 ~]# 7.5 使用mha工具check检查repl环境
检测命令为：masterha_check_repl--conf=/etc/masterha/app1.cnf，检测结果如下：
[root@oraclem1 ~]# masterha_check_repl --conf=/etc/masterha/app1.cnf
Fri Apr :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr :: - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] MHA::MasterMonitor version 0.56.
Fri Apr :: - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Master configurations are as below:
Master 192.168.52.130(192.168.52.130:), replicating from 192.168.52.129(192.168.52.129:), read-only
Master 192.168.52.129(192.168.52.129:), replicating from 192.168.52.130(192.168.52.130:) Fri Apr :: - [info] GTID failover mode =
Fri Apr :: - [info] Dead Servers:
Fri Apr :: - [info] Alive Servers:
Fri Apr :: - [info] 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] 192.168.52.130(192.168.52.130:)
Fri Apr :: - [info] 192.168.52.131(192.168.52.131:)
Fri Apr :: - [info] Alive Slaves:
Fri Apr :: - [info] 192.168.52.130(192.168.52.130:) Version=5.6.-log (oldest major version between slaves) log-bin:enabled
Fri Apr :: - [info] Replicating from 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] 192.168.52.131(192.168.52.131:) Version=5.6.-log (oldest major version between slaves) log-bin:enabled
Fri Apr :: - [info] Replicating from 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Current Alive Master: 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Checking slave configurations..
Fri Apr :: - [info] Checking replication filtering settings..
Fri Apr :: - [info] binlog_do_db= user_db, binlog_ignore_db= information_schema,mysql,performance_schema,test
Fri Apr :: - [info] Replication filtering check ok.
Fri Apr :: - [info] GTID (with auto-pos) is not supported
Fri Apr :: - [info] Starting SSH connection tests..
Fri Apr :: - [info] All SSH connection tests passed successfully.
Fri Apr :: - [info] Checking MHA Node version..
Fri Apr :: - [info] Version check ok.
Fri Apr :: - [info] Checking SSH publickey authentication settings on the current master..
Fri Apr :: - [info] HealthCheck: SSH to 192.168.52.129 is reachable.
Fri Apr :: - [info] Master MHA Node version is 0.56.
Fri Apr :: - [info] Checking recovery script configurations on 192.168.52.129(192.168.52.129:)..
Fri Apr :: - [info] Executing command: save_binary_logs --command=test --start_pos= --binlog_dir=/home/data/mysql/binlog/ --output_file=/var/tmp/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.
Fri Apr :: - [info] Connecting to root@192.168.52.129(192.168.52.129:)..
Creating /var/tmp if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /home/data/mysql/binlog/, up to mysql-bin.
Fri Apr :: - [info] Binlog setting check done.
Fri Apr :: - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Fri Apr :: - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.52.130 --slave_ip=192.168.52.130 --slave_port= --workdir=/var/tmp --target_version=5.6.-log --manager_version=0.56 --relay_dir=/home/data/mysql/data --current_relay_log=mysqld-relay-bin. --slave_pass=xxx
Fri Apr :: - [info] Connecting to root@192.168.52.130(192.168.52.130:)..
Checking slave recovery environment settings..
Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.
Temporary relay log file is /home/data/mysql/data/mysqld-relay-bin.
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Apr :: - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='manager' --slave_host=192.168.52.131 --slave_ip=192.168.52.131 --slave_port= --workdir=/var/tmp --target_version=5.6.-log --manager_version=0.56 --relay_log_info=/home/data/mysql/data/relay-log.info --relay_dir=/home/data/mysql/data/ --slave_pass=xxx
Fri Apr :: - [info] Connecting to root@192.168.52.131(192.168.52.131:)..
Checking slave recovery environment settings..
Opening /home/data/mysql/data/relay-log.info ... ok.
Relay log found at /home/data/mysql/data, up to mysql-relay-bin.
Temporary relay log file is /home/data/mysql/data/mysql-relay-bin.
Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Fri Apr :: - [info] Slaves settings check done.
Fri Apr :: - [info]
192.168.52.129(192.168.52.129:) (current master)
+--192.168.52.130(192.168.52.130:)
+--192.168.52.131(192.168.52.131:) Fri Apr :: - [info] Checking replication health on 192.168.52.130..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] Checking replication health on 192.168.52.131..
Fri Apr :: - [info] ok.
Fri Apr :: - [warning] master_ip_failover_script is not defined.
Fri Apr :: - [warning] shutdown_script is not defined.
Fri Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is OK.
[root@oraclem1 ~]# ，管理mha操作
8.1 启动manager
nohup masterha_manager --conf=/etc/masterha/app1.cnf < /dev/null >/logs/mha/app1/manager.log >& &
执行后台情形，如下所示： [root@oraclem1 mha4mysql-manager-0.56]# nohup masterha_manager --conf=/etc/masterha/app1.cnf /logs/mha/app1/manager.log >& &
[]
[root@oraclem1 mha4mysql-manager-0.56]#
[root@oraclem1 mha4mysql-manager-0.56]# tail -f /logs/mha/app1/manager.log
Fri Apr :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr :: - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] Reading server configuration from /etc/masterha/app1.cnf.. 8.2 使用masterha_check_status检测下
[root@oraclem1 ~]# masterha_check_status--conf=/etc/masterha/app1.cnf
app1 (pid:) is running(:PING_OK),master:192.168.52.129
[root@oraclem1 ~]# 手动操作： 8.3 停止manager
命令：masterha_stop --conf=/etc/masterha/app1.cnf
[root@oraclem1 mha4mysql-manager-0.56]#masterha_stop --conf=/etc/masterha/app1.cnf
Stopped app1 successfully.
[]+ Exit nohupmasterha_manager --conf=/etc/masterha/app1.cnf < /dev/null >/logs/mha/app1/manager.log >&
[root@oraclem1 mha4mysql-manager-0.56]# 8.4 master死机自动切换测试
在mysql的master库52.129上，执行如下命令：echo c> /proc/sysrq-trigger
后果是：然后会看到master库变成了52.，而52.131从库也*去连接新的主库52.130了。 Manager自动完成了切换操作。 8.5 master手动切换
先停止manager：masterha_stop --conf=/etc/masterha/app1.cnf
在备选slave和master上添加crontab –e任务， 手动切换master，命令如下
masterha_master_switch--conf=/etc/masterha/app1.cnf --master_state=dead--dead_master_host=192.168.52.129
masterha_master_switch--conf=/etc/masterha/app1.cnf --master_state=alive--new_master_host=192.168.52.130 先设置原来的master为dead，如下所示：
[root@oraclem1 ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=dead --dead_master_host=192.168.52.129
--dead_master_ip= is not set. Using 192.168.52.129.
--dead_master_port= is not set. Using .
Fri Apr :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr :: - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] MHA::MasterFailover version 0.56.
Fri Apr :: - [info] Starting master failover.
Fri Apr :: - [info]
Fri Apr :: - [info] * Phase : Configuration Check Phase..
Fri Apr :: - [info]
Fri Apr :: - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Master configurations are as below:
Master 192.168.52.130(192.168.52.130:), replicating from 192.168.52.129(192.168.52.129:), read-only
Master 192.168.52.129(192.168.52.129:), replicating from 192.168.52.130(192.168.52.130:) Fri Apr :: - [info] GTID failover mode =
Fri Apr :: - [info] Dead Servers:
Fri Apr :: - [error][/usr/local/share/perl5/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover.
Fri Apr :: - [error][/usr/local/share/perl5/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/local/bin/masterha_master_switch line
[root@oraclem1 ~]# 然后设置新的master为alive，在切换过程中，界面几次自动输入YES，最后会有Switchingmaster to 192.168.52.130(192.168.52.130:) completed successfully.提示标志着手动切换成功，如下所示：
[root@oraclem1 ~]# masterha_master_switch --conf=/etc/masterha/app1.cnf --master_state=alive --new_master_host=192.168.52.130
Fri Apr :: - [info] MHA::MasterRotate version 0.56.
Fri Apr :: - [info] Starting online master switch..
Fri Apr :: - [info]
Fri Apr :: - [info] * Phase : Configuration Check Phase..
Fri Apr :: - [info]
Fri Apr :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Apr :: - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] Reading server configuration from /etc/masterha/app1.cnf..
Fri Apr :: - [info] Multi-master configuration is detected. Current primary(writable) master is 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Master configurations are as below:
Master 192.168.52.130(192.168.52.130:), replicating from 192.168.52.129(192.168.52.129:), read-only
Master 192.168.52.129(192.168.52.129:), replicating from 192.168.52.130(192.168.52.130:) Fri Apr :: - [info] GTID failover mode =
Fri Apr :: - [info] Current Alive Master: 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] Alive Slaves:
Fri Apr :: - [info] 192.168.52.130(192.168.52.130:) Version=5.6.-log (oldest major version between slaves) log-bin:enabled
Fri Apr :: - [info] Replicating from 192.168.52.129(192.168.52.129:)
Fri Apr :: - [info] 192.168.52.131(192.168.52.131:) Version=5.6.-log (oldest major version between slaves) log-bin:enabled
Fri Apr :: - [info] Replicating from 192.168.52.129(192.168.52.129:) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.52.129(192.168.52.129:)? (YES/no): YES
Fri Apr :: - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] Checking MHA is not monitoring or doing failover..
Fri Apr :: - [info] Checking replication health on 192.168.52.130..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] Checking replication health on 192.168.52.131..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] 192.168.52.130 can be new master.
Fri Apr :: - [info]
From:
192.168.52.129(192.168.52.129:) (current master)
+--192.168.52.130(192.168.52.130:)
+--192.168.52.131(192.168.52.131:) To:
192.168.52.130(192.168.52.130:) (new master)
+--192.168.52.131(192.168.52.131:) Starting master switch from 192.168.52.129(192.168.52.129:) to 192.168.52.130(192.168.52.130:)? (yes/NO): yes
Fri Apr :: - [info] Checking whether 192.168.52.130(192.168.52.130:) is ok for the new master..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] ** Phase : Configuration Check Phase completed.
Fri Apr :: - [info]
Fri Apr :: - [info] * Phase : Rejecting updates Phase..
Fri Apr :: - [info]
master_ip_online_change_script is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yes
Fri Apr :: - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Fri Apr :: - [info] Executing FLUSH TABLES WITH READ LOCK..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] Orig master binlog:pos is mysql-bin.:.
Fri Apr :: - [info] Waiting to execute all relay logs on 192.168.52.130(192.168.52.130:)..
Fri Apr :: - [info] master_pos_wait(mysql-bin.:) completed on 192.168.52.130(192.168.52.130:). Executed events.
Fri Apr :: - [info] done.
Fri Apr :: - [info] Getting new master's binlog name and position..
Fri Apr :: - [info] mysql-bin.:
Fri Apr :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.52.130', MASTER_PORT=, MASTER_LOG_FILE='mysql-bin.000058', MASTER_LOG_POS=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Apr :: - [info] Setting read_only= on 192.168.52.130(192.168.52.130:)..
Fri Apr :: - [info] ok.
Fri Apr :: - [info]
Fri Apr :: - [info] * Switching slaves in parallel..
Fri Apr :: - [info]
Fri Apr :: - [info] -- Slave switch on host 192.168.52.131(192.168.52.131:) started, pid:
Fri Apr :: - [info]
Fri Apr :: - [info] Log messages from 192.168.52.131 ...
Fri Apr :: - [info]
Fri Apr :: - [info] Waiting to execute all relay logs on 192.168.52.131(192.168.52.131:)..
Fri Apr :: - [info] master_pos_wait(mysql-bin.:) completed on 192.168.52.131(192.168.52.131:). Executed events.
Fri Apr :: - [info] done.
Fri Apr :: - [info] Resetting slave 192.168.52.131(192.168.52.131:) and starting replication from the new master 192.168.52.130(192.168.52.130:)..
Fri Apr :: - [info] Executed CHANGE MASTER.
Fri Apr :: - [info] Slave started.
Fri Apr :: - [info] End of log messages from 192.168.52.131 ...
Fri Apr :: - [info]
Fri Apr :: - [info] -- Slave switch on host 192.168.52.131(192.168.52.131:) succeeded.
Fri Apr :: - [info] Unlocking all tables on the orig master:
Fri Apr :: - [info] Executing UNLOCK TABLES..
Fri Apr :: - [info] ok.
Fri Apr :: - [info] All new slave servers switched successfully.
Fri Apr :: - [info]
Fri Apr :: - [info] * Phase : New master cleanup phase..
Fri Apr :: - [info]
Fri Apr :: - [info] 192.168.52.130: Resetting slave info succeeded.
Fri Apr :: - [info] Switching master to 192.168.52.130(192.168.52.130:) completed successfully.
[root@oraclem1 ~]# PS：手动切换后，使用masterha_check_repl不能使用原来的/etc/masterha/app1.cnf来做check，要用新的app2.cnf来做check，因为app1.cnf里面的master是原来旧的cnf，check会报错主从复制失败。如何生成新的app2.cnf，很简单，如下所示：
（）复制原理的app1.cnf为新的app2.cnf
cp /etc/masterha/app1.cnf/etc/masterha/app2.cnf
（）编辑app2.cnf，将里面的server1和server2的ip互换，也就是switch的两个主从的ip换掉，如下所示：
[server1]
hostname=192.168.52.130
candidate_master=
master_binlog_dir=/home/data/mysql/binlog/ [server2]
hostname=192.168.52.129
#candidate_master=
#master_binlog_dir=/home/data/mysql/binlog/
（）然后在使用masterha_check_repl --conf=/etc/masterha/app2.cnf进行check，就可以看到switch后的mha的主从是ok的了。 9报错记录总结
报错记录1：
[root@data01 ~]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Tue Apr :: - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Tue Apr :: - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Tue Apr :: - [info] Reading server configuration from/etc/masterha/app1.cnf..
Tue Apr :: - [info] MHA::MasterMonitor version 0.56.
Tue Apr :: - [error][/usr/local/share/perl5/MHA/Server.pm,ln303] Getting relay log directory orcurrent relay logfile from replication table failed on192.168.52.(192.168.52.130:)!
Tue Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/share/perl5/MHA/ServerManager.pmline
Tue Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Tue Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is NOT OK!
[root@data01 ~]# 解决办法：在192.168.52.130上面，vim /etc/my.cnf，在里面添加
relay-log=/home/data/mysql/binlog/mysql-relay-bin
然后重启mysql，再去重新设置slave连接。
STOP SLAVE;
RESET SLAVE;
CHANGE MASTER TOMASTER_HOST='192.168.52.129',MASTER_USER='repl',MASTER_PASSWORD='repl_1234',MASTER_LOG_FILE='mysql-bin.000178',MASTER_LOG_POS=;
START SLAVE;
Ok，搞定了。 报错记录2： [root@data01 perl]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Thu Apr :: - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Thu Apr :: - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Thu Apr :: - [info] Reading server configuration from/etc/masterha/app1.cnf..
Thu Apr :: - [info] MHA::MasterMonitor version 0.56.
Thu Apr :: - [error][/usr/local/share/perl5/MHA/Server.pm,ln306] Getting relay log directory orcurrent relay logfile from replication table failed on 192.168.52.130(192.168.52.130:)!
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at/usr/local/share/perl5/MHA/ServerManager.pm line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is NOT OK!
[root@data01 perl]# 解决方法：
/etc/masterha/app1.cnf文件里面的参数配置，user和repl_user都是mysql账号，需要创建好，这里是只创建了repl_user而没有创建好user账号：
user=manager
password=manager_1234
repl_user=repl
repl_password=repl_1234 在mysql节点上，建立允许manager 访问数据库的“ manager manager ”账户，主要用于SHOW SLAVESTATUS,RESET SLAVE; 所以需要执行如下命令：
GRANT SUPER,RELOAD,REPLICATIONCLIENT,SELECT ON *.* TO manager@'192.168.52.%' IDENTIFIED BY 'manager_1234'; 错误记录3： [root@oraclem1 ~]# masterha_check_repl--conf=/etc/masterha/app1.cnf
Thu Apr :: - [warning] Global configuration file/etc/masterha_default.cnf not found. Skipping.
Thu Apr :: - [info] Reading application default configuration from/etc/masterha/app1.cnf..
Thu Apr :: - [info] Reading server configuration from/etc/masterha/app1.cnf..
Thu Apr :: - [info] MHA::MasterMonitor version 0.56.
Thu Apr :: - [error][/usr/local/share/perl5/MHA/ServerManager.pm,ln781] Multi-master configuration is detected, but two or more masters areeither writable (read-only is not set) or dead! Check configurations fordetails. Master configurations are as below:
Master 192.168.52.130(192.168.52.130:),replicating from 192.168.52.129(192.168.52.129:)
Master 192.168.52.129(192.168.52.129:),replicating from 192.168.52.130(192.168.52.130:) Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at/usr/local/share/perl5/MHA/MasterMonitor.pm line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is NOT OK!
[root@oraclem1 ~]# 解决办法：
mysql> set global read_only=;
Query OK, rows affected (0.00 sec) mysql> 报错记录4： Thu Apr :: - [info] Checking SSH publickey authentication andchecking recovery script configurations on all alive slave servers..
Thu Apr :: - [info] Executing command : apply_diff_relay_logs --command=test--slave_user='manager' --slave_host=192.168.52.130 --slave_ip=192.168.52.130--slave_port= --workdir=/var/tmp --target_version=5.6.-log--manager_version=0.56 --relay_dir=/home/data/mysql/data--current_relay_log=mysqld-relay-bin. --slave_pass=xxx
Thu Apr :: - [info] Connecting to root@192.168.52.130(192.168.52.130:)..
Can't exec "mysqlbinlog": No suchfile or directory at /usr/local/share/perl5/MHA/BinlogManager.pm line 106.
mysqlbinlog version command failed with rc1:, please verify PATH, LD_LIBRARY_PATH, and client options
at/usr/local/bin/apply_diff_relay_logs line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln205] Slaves settings check failed!
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln413] Slave configuration failed.
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/bin/masterha_check_repl line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is NOT OK!
[root@oraclem1 ~]# 解决办法：
[root@data02 ~]# type mysqlbinlog
mysqlbinlog is/usr/local/mysql/bin/mysqlbinlog
[root@data02 ~]#
[root@data02 ~]# ln -s/usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog 报错记录5：
Thu Apr :: - [info] Connecting to root@192.168.52.130(192.168.52.130:)..
Checking slave recovery environment settings..
Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.
Temporary relay log file is /home/data/mysql/data/mysqld-relay-bin.
Testing mysql connection and privileges..sh: mysql: command not found
mysql command failed with rc :!
at/usr/local/bin/apply_diff_relay_logs line
main::check()called at /usr/local/bin/apply_diff_relay_logs line
eval{...} called at /usr/local/bin/apply_diff_relay_logs line
main::main()called at /usr/local/bin/apply_diff_relay_logs line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln205] Slaves settings check failed!
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln413] Slave configuration failed.
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln424] Error happened on checking configurations. at /usr/local/bin/masterha_check_repl line
Thu Apr :: - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm,ln523] Error happened on monitoring servers.
Thu Apr :: - [info] Got exit code (Not master dead). MySQL Replication Health is NOT OK! 解决办法：
ln -s /usr/local/mysql/bin/mysql/usr/bin/mysql 报错记录6：
Fri Apr :: - [info] Executing command : apply_diff_relay_logs--command=test --slave_user='manager' --slave_host=192.168.52.130--slave_ip=192.168.52.130 --slave_port= --workdir=/var/tmp--target_version=5.6.-log --manager_version=0.56--relay_dir=/home/data/mysql/data--current_relay_log=mysqld-relay-bin. --slave_pass=xxx
Fri Apr :: - [info] Connecting to root@192.168.52.130(192.168.52.130:)..
Checking slave recovery environment settings..
Relay log found at /home/data/mysql/data, up to mysqld-relay-bin.
Temporary relay log file is/home/data/mysql/data/mysqld-relay-bin.
Testing mysql connection and privileges..Warning: Using a password onthe command line interface can be insecure.
ERROR () at line : CREATEcommand denied to user 'manager'@'192.168.52.130' for table'apply_diff_relay_logs_test'
mysql command failed with rc :!
at/usr/local/bin/apply_diff_relay_logs line
main::check()called at /usr/local/bin/apply_diff_relay_logs line
eval{...} called at /usr/local/bin/apply_diff_relay_logs line
main::main()called at /usr/local/bin/apply_diff_relay_logs line
Fri Apr :: -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln205] Slaves settingscheck failed!
Fri Apr :: -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln413] Slave configurationfailed.
Fri Apr :: -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln424] Error happened onchecking configurations. at/usr/local/bin/masterha_check_repl line
Fri Apr :: -[error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln523] Error happened onmonitoring servers.
Fri Apr :: - [info] Got exitcode (Not master dead). MySQL Replication Health is NOT OK! 解决办法：
执行如下授权语句sql：
GRANT CREATE,INSERT,UPDATE,DELETE,DROP ON*.* TO manager@'192.168.52.%'; ，参考资料
参考：http://blog.csdn.net/largetalk/article/details/10006899
参考：http://www.tuicool.com/articles/NV3U3i
多实例参考：http://www.cnblogs.com/wingsless/p/4033093.html
参考：http://www.tuicool.com/articles/NV3U3i
参考人为模拟故障过程：
http://www.mysqlsystems.com/2012/03/figure-out-process-of-autofailover-on-mha.html#more-2172 ----------------------------------------------------------------------------------------------------------------
<版权所有，文章允许转载，但必须以链接方式注明源地址，否则追究法律责任!>
原博客地址： http://blog.itpub.net/26230597/viewspace-1570798/
原作者：黄杉 (mchdba)
----------------------------------------------------------------------------------------------------------------