【MySQL】MHA部署与MasterFailover代码分析

部署：

[root@dns packages]# tar -zxvf mha4mysql-manager-0.53.tar.gz

[root@dns mha4mysql-manager-0.53]# perl Makefile.PL

[root@dns mha4mysql-manager-0.53]# make && make install

配置有很多参数，可以看看lib\MHA下面的Config.pm代码

[root@dns mha4mysql-manager-0.53]# vim /etc/.cnf

[server default]

manager_log=/data1/masterha/3307_leju/manager.log

manager_workdir=/data1/masterha/3307_leju

multi_tier_slave=

password=passwd

ping_interval=

repl_password=x8uf7nbv5x64

repl_user=repl

shutdown_script="/samples/scripts/power_manager" //设置故障发生后关闭故障主机脚本（该脚本的主要作用是关闭主机放在发生脑裂
master_pid_file=/data1/mysql/3307_leju/mysql.pid //shutdown_script脚本中需要使用的pid文件位置，通过pid文件中的pid进程号强制杀掉主库实例

ssh_user=root

user=ha

[server1]

hostname=10.207.0.125

master_binlog_dir="/data1/mysql/3307_leju"

port=

[server2]

hostname=10.207.0.126

master_binlog_dir="/data1/mysql/3307_leju"

port=

[server3]

hostname=10.207.0.127

master_binlog_dir="/data1/mysql/3307_leju"

port=

[server4]

hostname=10.207.0.128

master_binlog_dir="/data1/mysql/3307_leju"

port=

启动前检查复制关系

[root@dns bin]# masterha_check_repl --conf=/etc/masterha/.cnf

启动manager

masterha_manager启动有很多参数，参数说明不列举说明
[root@dns bin]# masterha_manager --conf=/etc/masterha/.cnf --remove_dead_master_conf --ignore_last_failover

功能分析：

注意MHA需要super,select,replication slave,replication client权限，在测试MHA0.56时由于权限不够perl也没有相关报错，导致failover卡住，问题很难才定位到

MHA的故障切换是由lib/MHA/MasterFailover.pm代码完成。

编译后位置移动到/usr/local/share/perl5/MHA/MasterFailover.pm。

代码主体为do_master_failover函数完成，代码片段：

sub do_master_failover {

  my $error_code = ;

  my ( $dead_master, $new_master );

  eval {

    my @servers_config = init_config();

    $log->info("Starting master failover.");

    $log->info("* Phase 1: Configuration Check Phase..\n");
    $dead_master = check_settings( \@servers_config );

    $log->info("** Phase 1: Configuration Check Phase completed.\n");

    $log->info("* Phase 2: Dead Master Shutdown Phase..\n");
    force_shutdown($dead_master);

    $log->info("* Phase 2: Dead Master Shutdown Phase completed.\n");

    $log->info("* Phase 3: Master Recovery Phase..\n");

$log->info("* Phase 3.1: Getting Latest Slaves Phase..\n");
    check_set_latest_slaves();

    $log->info("* Phase 3.2: Saving Dead Master's Binlog Phase..\n");
    save_master_binlog($dead_master);

    $log->info("* Phase 3.3: Determining New Master Phase..\n");
    my $latest_base_slave = find_latest_base_slave($dead_master);

    $new_master = select_new_master( $dead_master, $latest_base_slave );

    my ( $master_log_file, $master_log_pos ) = recover_master( $dead_master, $new_master, $latest_base_slave );

    $new_master->{activated} = ;

    $log->info("* Phase 3: Master Recovery Phase completed.\n");

    $log->info("* Phase 4: Slaves Recovery Phase..\n");

    $error_code = recover_slaves( $dead_master, $new_master, $latest_base_slave, $master_log_file, $master_log_pos);

    if ( $g_remove_dead_master_conf && $error_code ==  ) { MHA::Config::delete_block_and_save( $g_config_file, $dead_master->{id}, $log );}

    cleanup();

  };

  ...

}

可以通过增加几行代码实现MHA切换通知DNS达到前端请求的切换。

# 建立连接

my $dbh = DBI->connect("DBI:mysql:database=dblayer;mysql_socket=/tmp/mysql_3307.sock","dblayer","dblayer");

# 下线主库域名，IsLive状态修改（Bind_DLZ的MySQL驱动查询增加IsLive字段判断）

my $rows = $dbh->do("UPDATE dns_records SET IsLive=0 WHERE data='$dead_master->{ip}' and port='$dead_master->{port}'");

# 获取主库host（Bind的dns_records表中使用IP和PORT对应一个唯一HOST，多实例的支持）

my $sth = $dbh->prepare("select host from dns_records where data='$dead_master->{ip}' and port='$dead_master->{port}'");

$sth->execute();

my $ref = $sth->fetchrow();

# 将新主库更新host

my $rows = $dbh->do("UPDATE dns_records SET HOST='$ref' WHERE data='$new_master->{ip}' and port='$dead_master->{port}'");

# 将切换记录到库

my $rows = $dbh->do("INSERT INTO masterfailover_log(old_master_ip,new_master_ip,port,type) values ('$dead_master->{ip}','$new_master->{ip}','$dead_master->{port}',1)");

# 断开连接
$sth->finish;
$dbh->disconnect();

在加入shutdown_script后，出现报错：

Thu Sep  ::  - [info]   /usr/local/mha_manager/samples/scripts/power_manager --command=status --ssh_user=root --host=10.207.0.125 --ip=10.207.0.125

Undefined subroutine &main::FIXME_xxx called at /usr/local/mha_manager/samples/scripts/power_manager line 387.

Thu Sep  ::  - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln235]  Failed to get power status with return code :.

Thu Sep  ::  - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln383] Error happend on checking configurations.  at /usr/local/bin/masterha_check_repl line

Thu Sep  ::  - [error][/usr/local/share/perl5/MHA/MasterMonitor.pm, ln478] Error happened on monitoring servers.

Thu Sep  ::  - [info] Got exit code  (Not master dead).

MySQL Replication Health is NOT OK!

打开power_manager可以看到，报错的部分是一个未定义函数FIXME_xxx，代码是想通过服务器管理地址进行操作，尝试将未定义函数部分去掉会出现更多问题。

在power_manager代码下面有一些注释说明，说明中有：

# killing mysqld with specified pid file. This is useful when you run multiple MySQL instances and want to stop only specified instance

power_manager --command=stopssh --host=master_server --ssh_user=root --pid_file=/var/lib/mysql/mysqld.pid

所以不在配置中添加shutdown_script，直接在MasterFailover.pm中force_shutdown_internal函数最后添加power_manager的调用。

system("/usr/local/mha_manager/samples/scripts/power_manager --command=stopssh --host='$dead_master->{ip}' --ssh_user=root --pid_file='$dead_master->{master_pid_file}'");

因为power_manager有未定义的函数，如果直接调用，在异常主库可以ssh通但是pid已经不存在或者根本ssh不通的情况下，因为stopssh函数返回值异常仍会报错未命名函数，所以改一下power_manager，注释掉一部分代码，防止killall -9 mysql mysqld。

sub stopssh {

  ...if ($pid_file) {

    $command =

"\"if [ ! -e $pid_file ]; then exit 1; fi; pid=\\\`cat $pid_file\\\`; rm -f $pid_file; kill -9 \\\$pid; a=\\\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \\\$1'}\\\`; if [ \"a\\\$a\" = \"a0\" ]; then exit 10; fi; sleep 1; a=\\\`ps ax | grep $pid_file | grep -v grep | wc | awk {'print \\\$1'}\\\`; if [ \"a\\\$a\" = \"a0\" ]; then exit 10; else exit 1; fi\"";

    ( $high_ret, $low_ret ) = MHA::ManagerUtil::exec_system(

      "ssh $ssh_user_host $MHA::ManagerConst::SSH_OPT_CHECK $command");

    if ( $high_ret == $SSH_STOP_OK && $low_ret ==  ) {

      print "ssh reachable. mysqld stopped. power off not needed.\n";

      return $high_ret;

    }

    elsif ( $high_ret ==  && $low_ret ==  ) {

      print "ssh reachable. not found $pid_file,mysqld stopped. power off not needed.\n";

      return ;

    }

    else {

      print "Killing mysqld instance based on $pid_file failed.\n";

return ;

    }

  }

  # 后面killall -9 mysql mysqld 注释

}

在配置里面加report_script=/usr/local/bin/send_report后

修改mha_manager/samples/scripts/send_report的代码可以通知ZABBIX和邮件通知。ZABBIX在mha_manager主机加上一个监控项，把Item设置为Zabbix trapper，Type of information设置为字符

my @array=split(/ /,$subject);

my $sender=join('',$array[],$array[]);

system("/data1/scripts/zabbix_sender --zabbix-server serverip --port 10051 --host 'mhaserver' --key mysql.MHA --value '$sender'");

system("/bin/echo '$body' | /usr/bin/mutt -s '$subject' 'jiangxu\@test.com'");

秒客网

【MySQL】MHA部署与MasterFailover代码分析

相关文章