lvs+ldirectord+pacemaker+corosync+mysql实现高可用负载均衡(一)

时间:2022-04-06 23:30:31

author:skate
tiome:2013/12/06


lvs+ldirectord+pacemaker+corosync+mysql实现高可用负载均衡

 

环境说明
vip:192.168.213.97
rip1:192.168.213.91
rip2:192.168.213.92
client:192.168.213.93

os:centos6.3
kernel:2.6.32-279.el6.x86_64
mysql version:5.5.28

 


1. ldirectord安装与配置


ldirectord下载
下载地址:http://horms.net/projects/ldirectord/download.shtml

# unzip resource-agents-master.zip
# cd resource-agents-master
# ./autogen.sh
# ./configure
# make && make install

此时ldirectord还是不能使用,比如查看状态,如下:


[root@localhost resource-agents-master]# service ldirectord status
Can't locate Socket6.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/sbin/ldirectord line 838.
BEGIN failed--compilation aborted at /usr/sbin/ldirectord line 838.
[root@localhost resource-agents-master]#

需要安装一些perl的包

[root@localhost resource-agents-master]# yum install perl-Socket6 perl-IO-Socket-INET6 perl-Email-Date-Format perl-TimeDate perl-Pod-Escapes perl-Pod-Simple perl-Test-Pod perl-MailTools perl-libwww-perl
....
....
Dependencies Resolved

====================================================================================================================================
 Package                                      Arch                   Version                             Repository            Size
====================================================================================================================================
Installing:
 perl-IO-Socket-INET6                         noarch                 2.56-4.el6                          base                  17 k
 perl-Socket6                                 x86_64                 0.23-4.el6                          base                  27 k
 perl-Test-Pod                                noarch                 1.40-1.el6                          base                  14 k
Updating:
 perl-Pod-Escapes                             x86_64                 1:1.04-136.el6                      base                  32 k
 perl-Pod-Simple                              x86_64                 1:3.13-136.el6                      base                 212 k
Installing for dependencies:
 db4-cxx                                      x86_64                 4.7.25-18.el6_4                     base                 588 k
 db4-devel                                    x86_64                 4.7.25-18.el6_4                     base                 6.6 M
 gdbm-devel                                   x86_64                 1.8.0-36.el6                        base                  25 k
 perl-CGI                                     x86_64                 3.51-136.el6                        base                 209 k
Updating for dependencies:
 db4                                          x86_64                 4.7.25-18.el6_4                     base                 563 k
 db4-utils                                    x86_64                 4.7.25-18.el6_4                     base                 130 k
 perl                                         x86_64                 4:5.10.1-136.el6                    base                  10 M
 perl-Archive-Extract                         x86_64                 1:0.38-136.el6                      base                  40 k
 perl-Archive-Tar                             x86_64                 1.58-136.el6                        base                  73 k
 perl-CPAN                                    x86_64                 1.9402-136.el6                      base                 246 k
 perl-CPANPLUS                                x86_64                 0.88-136.el6                        base                 307 k
 perl-Compress-Raw-Bzip2                      x86_64                 2.021-136.el6                       base                  45 k
 perl-Compress-Raw-Zlib                       x86_64                 1:2.021-136.el6                     base                  69 k
 perl-Compress-Zlib                           x86_64                 2.021-136.el6                       base                  45 k
 perl-Digest-SHA                              x86_64                 1:5.47-136.el6                      base                  64 k
 perl-ExtUtils-CBuilder                       x86_64                 1:0.27-136.el6                      base                  48 k
 perl-ExtUtils-Embed                          x86_64                 1.28-136.el6                        base                  31 k
 perl-ExtUtils-MakeMaker                      x86_64                 6.55-136.el6                        base                 293 k
 perl-ExtUtils-ParseXS                        x86_64                 1:2.2003.0-136.el6                  base                  45 k
 perl-File-Fetch                              x86_64                 0.26-136.el6                        base                  41 k
 perl-IO-Compress-Base                        x86_64                 2.021-136.el6                       base                  69 k
 perl-IO-Compress-Bzip2                       x86_64                 2.021-136.el6                       base                  48 k
 perl-IO-Compress-Zlib                        x86_64                 2.021-136.el6                       base                 135 k
 perl-IO-Zlib                                 x86_64                 1:1.09-136.el6                      base                  33 k
 perl-IPC-Cmd                                 x86_64                 1:0.56-136.el6                      base                  46 k
 perl-Locale-Maketext-Simple                  x86_64                 1:0.18-136.el6                      base                  31 k
 perl-Log-Message                             x86_64                 1:0.02-136.el6                      base                  46 k
 perl-Log-Message-Simple                      x86_64                 0.04-136.el6                        base                  28 k
 perl-Module-Build                            x86_64                 1:0.3500-136.el6                    base                 229 k
 perl-Module-CoreList                         x86_64                 2.18-136.el6                        base                  65 k
 perl-Module-Load                             x86_64                 1:0.16-136.el6                      base                  28 k
 perl-Module-Load-Conditional                 x86_64                 0.30-136.el6                        base                  34 k
 perl-Module-Loaded                           x86_64                 1:0.02-136.el6                      base                  27 k
 perl-Module-Pluggable                        x86_64                 1:3.90-136.el6                      base                  40 k
 perl-Object-Accessor                         x86_64                 1:0.34-136.el6                      base                  37 k
 perl-Package-Constants                       x86_64                 1:0.02-136.el6                      base                  26 k
 perl-Params-Check                            x86_64                 1:0.26-136.el6                      base                  35 k
 perl-Parse-CPAN-Meta                         x86_64                 1:1.40-136.el6                      base                  29 k
 perl-Term-UI                                 x86_64                 0.20-136.el6                        base                  39 k
 perl-Test-Harness                            x86_64                 3.17-136.el6                        base                 231 k
 perl-Test-Simple                             x86_64                 0.92-136.el6                        base                 112 k
 perl-Time-HiRes                              x86_64                 4:1.9721-136.el6                    base                  48 k
 perl-Time-Piece                              x86_64                 1.15-136.el6                        base                  46 k
 perl-core                                    x86_64                 5.10.1-136.el6                      base                  23 k
 perl-devel                                   x86_64                 4:5.10.1-136.el6                    base                 423 k
 perl-libs                                    x86_64                 4:5.10.1-136.el6                    base                 578 k
 perl-parent                                  x86_64                 1:0.221-136.el6                     base                  27 k
 perl-suidperl                                x86_64                 4:5.10.1-136.el6                    base                  50 k
 perl-version                                 x86_64                 3:0.77-136.el6                      base                  51 k
 zlib                                         x86_64                 1.2.3-29.el6                        base                  73 k
 zlib-devel                                   x86_64                 1.2.3-29.el6                        base                  44 k

Transaction Summary
====================================================================================================================================
Install       7 Package(s)
Upgrade      49 Package(s)

Total download size: 23 M

....
....

继续看看ldirectord是否可用

[root@localhost resource-agents-master]# service ldirectord status
Can not find ipvsadm at /usr/sbin/ldirectord line 887.
[root@localhost resource-agents-master]#

发现没有安装ipvsadm,安装如下


# yum install ipvsadm
....
Dependencies Resolved

====================================================================================================================================
 Package                        Arch                          Version                             Repository                   Size
====================================================================================================================================
Installing:
 ipvsadm                        x86_64                        1.26-2.el6                          base                         41 k

Transaction Summary
====================================================================================================================================
Install       1 Package(s)

Total download size: 41 k
....
....

[root@localhost resource-agents-master]# service ldirectord status
Config file ldirectord.cf not found

想查看配置文件放在何处,可以查看文件“/etc/init.d/ldirectord”和“/usr/sbin/ldirectord”;默认是放在“/etc/ha.d/ldirectord.cf”。把在安装目录的样本配置复制到“/etc/ha.d/”下,然后再修改。这时ldirectord服务才可以正常工作


这回可用了

[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord status
ldirectord is stopped for /etc/ha.d/ldirectord.cf
[root@localhost resource-agents-master]# service ldirectord start
Starting ldirectord... success
[root@localhost resource-agents-master]# service ldirectord status
ldirectord for /etc/ha.d/ldirectord.cf is running with pid: 28911
[root@localhost resource-agents-master]#

 

安装参考:
http://blog.chinaunix.net/uid-17268883-id-3145428.html

 


2. lvs安装

 

LVS软件包括二部分:


 A.IPVS模块,LVS已经是Linux标准内核的一部分,直接被编译在内核中!
 B.IPVS管理工具ipvsadm

 

检测IPVS模块是否真的编译到内核中去了
[root@master mysqlinstall]# modprobe -l |grep ipvs
kernel/net/netfilter/ipvs/ip_vs.ko
kernel/net/netfilter/ipvs/ip_vs_rr.ko
kernel/net/netfilter/ipvs/ip_vs_wrr.ko
kernel/net/netfilter/ipvs/ip_vs_lc.ko
kernel/net/netfilter/ipvs/ip_vs_wlc.ko
kernel/net/netfilter/ipvs/ip_vs_lblc.ko
kernel/net/netfilter/ipvs/ip_vs_lblcr.ko
kernel/net/netfilter/ipvs/ip_vs_dh.ko
kernel/net/netfilter/ipvs/ip_vs_sh.ko
kernel/net/netfilter/ipvs/ip_vs_sed.ko
kernel/net/netfilter/ipvs/ip_vs_nq.ko
kernel/net/netfilter/ipvs/ip_vs_ftp.ko


ipvsadm管理工具安装上面介绍了,通过yum安装比较方便,可以解决很多依赖关系,如下:
# yum install ipvsadm

 

安装参考:
http://beyondhdf.blog.51cto.com/229452/1331874
http://soulboy.blog.51cto.com/4007306/1299896
http://zh.linuxvirtualserver.org/node/98
http://itnihao.blog.51cto.com/1741976/744237

 


Directorer的lvsdr脚本

[root@localhost mysqlinstall]# more lvsdr.sh
###############################################################
#Director script:
################################################################
#!/bin/bash
#
# LVS script for VS/DR
#
# Startup script handle the initialisation of LVS
# chkconfig: - 28 72
# description: Initialise director of the Linux Virtual Server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator: 
#             zxg: repair bug
#
#
# Source function library
. /etc/rc.d/init.d/functions
#
VIP=192.168.213.97
RIP1=192.168.213.91
RIP2=192.168.213.92
ETHx=eth0:3
PORT=3306
#

start() {
      /sbin/ifconfig $ETHx $VIP broadcast $VIP netmask 255.255.255.255 up
      /sbin/route add -host $VIP dev $ETHx
# Since this is the Director we must be able to forward packets
     echo 1 > /proc/sys/net/ipv4/ip_forward
# Clear all iptables rules.
     /sbin/iptables -F
# Reset iptables counters.
     /sbin/iptables -Z
# Clear all ipvsadm rules/services.
     /sbin/ipvsadm -C
# Add an IP virtual service for VIP 192.168.0.219 port $PORT
# In this recipe, we will use the round-robin scheduling method.
# In production, however, you should use a weighted, dynamic scheduling method.
    /sbin/ipvsadm -A -t $VIP:$PORT -s wrr
# Now direct packets for this VIP to
# the real server IP (RIP) inside the cluster
    /sbin/ipvsadm -a -t $VIP:$PORT -r $RIP1 -g -w 1
    /sbin/ipvsadm -a -t $VIP:$PORT -r $RIP2 -g -w 3
    /bin/touch /var/lock/subsys/ipvsadm &> /dev/null
}

stop() {
# Stop forwarding packets
    echo 0 > /proc/sys/net/ipv4/ip_forward
# Reset ipvsadm
    /sbin/ipvsadm -C
# Bring down the VIP interface
    /sbin/ifconfig $ETHx down
    /sbin/route del $VIP
    /bin/rm -f /var/lock/subsys/ipvsadm
    echo "ipvs stopped..."

}

status() {
  if [ ! -e /var/lock/subsys/ipvsadm ]; then
    echo "ipvsadm stopped ..."
  else
    echo "ipvs is running ..."
    ipvsadm -L -n
  fi
}


restart() {
  stop
  start
}


case "$1" in
  start)
    [ -f "/var/lock/subsys/ipvsadm" ] && exit 0
    start        
  ;;
  stop)
    stop
  ;;

  restart)
    restart
  ;;
  status)
  
   status
  ;;
  *)
  echo "Usage: $0 {start|stop|status|restart}"
  ;;
esac

 

realserver的运行脚本

[root@master mysqlinstall]# more lvsrl.sh
#############################################################
#RealServer脚本:
 
##############################################################
#!/bin/bash
#
# Script to start LVS DR real server.
# description: LVS DR real server
#
# Author: zxg
# CreateTime: 2013/12/06
#
# Modificator: 
#             zxg: repair bug
#
#
.  /etc/rc.d/init.d/functions
VIP=192.168.213.97
host=`/bin/hostname`
case "$1" in
start)
       # Start LVS-DR real server on this machine.
        /sbin/ifconfig lo down
        /sbin/ifconfig lo up
        echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
        /sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up
        /sbin/route add -host $VIP dev lo:0
;;
stop)
        # Stop LVS-DR real server loopback device(s).
        /sbin/ifconfig lo:0 down
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
        echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
;;
status)
        # Status of LVS-DR real server.
        islothere=`/sbin/ifconfig lo:0 | grep $VIP`
        isrothere=`netstat -rn | grep "lo:0" | grep $VIP`
        if [ ! "$islothere" -o ! "isrothere" ];then
            # Either the route or the lo:0 device
            # not found.
            echo "LVS-DR real server Stopped."
        else
            echo "LVS-DR real server Running."
        fi
;;
*)
            # Invalid entry.
            echo "$0: Usage: $0 {start|status|stop}"
            exit 1
;;
esac


先把脚本lvsrl.sh部署到所有的realserver上,并启动
然后把脚本lvsdr.sh部署到director上,并启动


可以用如下两个命令查看lvs的运行状态
# ip addr show dev eth0     //查看ip的漂移情况
# ipvsadm -ln               //查看转发配置表

 

例如:

[root@localhost mysqlinstall]# ip addr show dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:50:56:b8:7a:78 brd ff:ff:ff:ff:ff:ff
    inet 192.168.213.90/24 brd 192.168.213.255 scope global eth0
    inet 192.168.213.97/32 brd 192.168.213.97 scope global eth0:3
    inet6 fe80::250:56ff:feb8:7a78/64 scope link
       valid_lft forever preferred_lft forever
[root@localhost mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.213.97:3306 rr persistent 2
  -> 192.168.213.91:3306          Route   1      0          0        
  -> 192.168.213.92:3306          Route   1      0          40       
[root@localhost mysqlinstall]#

 

在客户端写个循环连接的脚本

[root@slave93 mysqlinstall]#  more load_mysql.sh
while (true)
do
 mysql -uroot -proot -h192.168.213.97 -e "select count(*) from test.a;"     
 sleep 3 
done

 

在客户端执行脚本
[root@slave93 mysqlinstall]# sh load_mysql.sh

 

在director上查看连接情况
# ipvsadm -lcn

 

上面的lvs是采用的持久连接的方式,轮询的均衡算法;对于使用数据库大都会使用持久连接(300s),可以采用wrr的均衡算法

 

上面的director是独立的主机,为了节省成本,可以把director和某一台realserver部署同一台上,比如部署到91上

[root@master mysqlinstall]# sh lvsdr.sh status
ipvs is running ...
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.213.97:3306 wrr
  -> 192.168.213.91:3306          Local   1      0          0        
  -> 192.168.213.92:3306          Route   3      0          0 

 

到现在,lvs可以正常工作,可以实现负载均衡,但后端某个realserver有问题时,lvs把请求分配器上,就会有问题,这时就需要配置
前面我们安装的ldirectord服务,其可以自动监控后端realserver的健康状态,发现有问题时就更新director的虚拟服务器和RS链表。即自动从集群中摘除,待以后故障恢复,ldirectord服务会自动把其加入集群中


按如下修改ldirectord的配置文件
[root@master mysqlinstall]# more /etc/ha.d/ldirectord.cf
#
# Sample ldirectord configuration file to configure various virtual services.
#
# Ldirectord will connect to each real server once per second and request
# /index.html. If the data returned by the server does not contain the
# string "Test Message" then the test fails and the real server will be
# taken out of the available pool. The real server will be added back into
# the pool once the test succeeds. If all real servers are removed from the
# pool then localhost:80 is added to the pool as a fallback measure.

# Global Directives
checktimeout=3   //超时
checkinterval=1  //check的间隔时间
#fallback=127.0.0.1:80
#fallback6=[::1]:80
autoreload=yes   //实时读取配置文件
#logfile="/var/log/ldirectord.log"   //定义日志文件位置
#logfile="local0"
#emailalert="admin@x.y.z"
#emailalertfreq=3600
#emailalertstatus=all
quiescent=no   //真正的把故障realserver从lvs映射表中删除,以保证集群对外的不间断服务

#Sample configuration for a MySQL virtual service.
virtual = 192.168.213.97:3306
        #real=master->slave92:3306 gate 10
        real=192.168.213.91 gate 10
        real=192.168.213.92 gate 10
        fallback=127.0.0.1:3306
        service=mysql
        scheduler=wrr
        #persistent=600
        #netmask=255.255.255.255
        protocol=tcp
        checktype=negotiate
        login="root"
        passwd="root"
        database="test"
        request="SELECT * FROM a"
  

然后重启ldirectord服务
[root@master mysqlinstall]# service ldirectord restart 
[root@master mysqlinstall]# service ldirectord status

 

然后查看log看ldirectord的工作状况

[root@master mysqlinstall]# tail -f /var/log/ldirectord.log

 

测试ldirectord是否可以对realserver做健康检查,为了便于测试,我把均衡算法改为rr,并去掉持久连接

 

测试前:

[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.213.97:3306 rr
  -> 192.168.213.91:3306          Local   10     0          59       
  -> 192.168.213.92:3306          Route   10     0          59       
[root@master mysqlinstall]#

 

启动客户端测试脚本,每秒执行一次数据库查询操作
# sh load_mysql.sh

 

停止92的mysql

 

查看结果

[root@master mysqlinstall]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.213.97:3306 rr
  -> 192.168.213.91:3306          Local   10     0          62       
[root@master mysqlinstall]#


查看ldirectord的log
[root@master mysqlinstall]# tail -f /var/log/ldirectord.log
[Sat Dec  7 01:52:22 2013|ldirectord.cf|7445] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)
[Sat Dec  7 01:52:22 2013|ldirectord|26081] system(/sbin/ipvsadm -a -t 192.168.213.97:3306 -r 192.168.213.92:0 -g -w 10) failed:
[Sat Dec  7 01:52:22 2013|ldirectord|26081] Added real server: 192.168.213.92:0 (192.168.213.97:3306) (Weight set to 10)


[Sat Dec  7 02:02:53 2013|ldirectord.cf|7445] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)
[Sat Dec  7 02:02:53 2013|ldirectord|26081] system(/sbin/ipvsadm -d -t 192.168.213.97:3306 -r 192.168.213.92:3306) failed:
[Sat Dec  7 02:02:53 2013|ldirectord|26081] Deleted real server: 192.168.213.92:3306 mapped from 192.168.213.92:0 (192.168.213.97:3306)

前端的服务有短暂1s的影响

.....
+----------+
+----------+
| count(*) |
+----------+
|        2 |
+----------+
ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.213.97' (111)
+----------+
| count(*) |
+----------+
|        2 |
+----------+
......


到目前为止,lvs可以负载均衡前端的请求,并可以通过ldirectord服务实现对后端的realserver的健康检查,但是如过lvs和ldirectord所在的机器故障怎么办呢,就没办法高可用了,这时就需要corosync和pacemaker来做HA。


会继续下面会继续........

 

 

 

-------续---------