keepalived高可用

时间:2023-03-09 20:13:07
keepalived高可用

keepalived介绍

Keepalived是一个基于vrrp协议的高可用方案,vrrp协议的软件实现,原生设计的目的为了高可用ipvs服务。


1. vrrp协议

VRRP是一种容错协议,它通过把几台路由设备联合组成一台虚拟的路由设备,并通过一定的机制来保证当主机的下一跳设备出现故障时,可以及时将业务切换到其它设备,从而保持通讯的连续性和可靠性,

  • vrrp协议中常见术语:
  1. 虚拟路由器:Virtual Router
  2. 虚拟路由器标识:VRID(0-255)
  3. 物理路由器:master(主设备),backup(备用设备),priority(优先级)
  4. VIP:Virtual IP
  5. VMAC:Virtual MAC(00-00-5e-00-01-VIRD)
  • 虚拟路由器工作机制如下:
  1. 根据优先级的大小挑选Master设备,比较优先级的大小,优先级高者当选为Master
  2. 当两台优先级相同的路由器同时竞争Master时,比较接口IP地址大小。接口地址大者当选为Master

    备份路由器随时监听Master的状态。
  3. 当主路由器正常工作时,它会每隔一段时间(Advertisement_Interval)发送一个VRRP组播报文,以通知组内的备份路由器,主路由器处于正常工作状态。
  4. 当组内的备份路由器一段时间(Master_Down_Interval)内没有接收到来自主路由器的报文,则将自己转为主路由器。

2. HA Cluster配置

2.1 HA Cluster的配置前提

  1. 各节点时间必须同步(ntp,chrony)。
  2. 确保iptables及selinux不会成为阻碍。
  3. 各节点之间可通过主机名互相通信(对KeepAlived并非必须),简易使用/etc/hosts文件实现。
  4. 各节点之间的root用户可以基于密钥认证的ssh服务完成相互通信(非必须);
  5. 配置keepalived的网卡必须支持并开启多播(multicast)功能。(ip link set dev enoxxxxx multicast on|off)

2.2 HA Cluster的虚拟路由器配置

环境:两台linux主机,配置虚拟路由器组,使用 10.1.

  • 同步时间
  1. 安装ntp软件包

     	[root@ _8_ ~]# yum -y install ntp
  2. 编辑ntp配置文件,使本机作为ntp时间服务器,注释掉多行以server开头的行,添加server 127.127.0.1

    keepalived高可用

  3. 重启ntp服务

     	[root@ _9_ ~]# service ntpd restart
  4. 另一台主机2同步时间

     	[root@ _9_ ~]# ntpdate 10.1.6.11
    1 Nov 18:38:03 ntpdate[46881]: adjust time server 10.1.6.11 offset -0.000035 sec
  • keepalived 单主模型(10.1.7.19)
  1. 安装keepalived

     	[root@ _14_ ~]# yum -y install keepalived
  2. 主机1修改keepalived配置文件,注释掉Virtual server的内容(这里暂不配置)

     	[root@ _15_ ~]# vim /etc/keepalived/keepalived.conf
    
     	! Configuration File for keepalived
    global_defs {
    notification_email {
    root@localhost #管理员邮箱
    }
    notification_email_from keepalived@localhost #发送者keepalived
    smtp_server 127.0.0.1 #邮箱服务器
    smtp_connect_timeout 30 #邮件发送超时时间
    router_id node1 #当前路由器物理标识符
    vrrp_mcast_group4 224.0.200.158 #多播地址(默认开启),应该与其他组成虚拟路由器的主机一致
    } vrrp_instance VI_1 { #配置vrp示例,VI_1,随意定义,需唯一
    state MASTER #定义当前节点在此虚拟路由器上的初始状态;只能有一个是MASTER,其余都为BACKUP
    interface eth0 #绑定为当前虚拟路由器使用的物理接口
    virtual_router_id 16 #当前虚拟路由器的唯一标识(0-255)
    priority 100 #当前主机在此虚拟路由器中的优先级
    advert_int 1 #vrrp通告的时间间隔
    authentication {
    auth_type PASS #认证类型,PASS为简单认证,AH为复杂认证,推荐使用PASS
    auth_pass RrpIoZU7 #认证字符
    }
    virtual_ipaddress {
    10.1.7.19/16 dev eth0 #配置的接口虚拟ip
    }
    } 注:网卡多播功能开启与关闭:ip link set multicast on|off
  3. 使用scp拷贝给另一台主机2,修改对应参数 。

     	[root@ _15_ ~]# scp /etc/keepalived/keepalived.conf root@10.1.7.11:/etc/keepalived/keepalived.conf	
    
     修改
    state MASTER 为 state BACKUP
    priority 100 为 priority 98 备用节点优先级要比主节点低
  • 测试
  1. 启动主节点,能看到启动为MASTER,添加了10.1.7.19的地址

     	[root@ _1_ ~]# service keepalived start
    Starting keepalived: [root@ _1_ ~]# tail /var/log/message
    Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Opening file '/etc/keepalived/keepalived.conf'.
    Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Configuration is using : 7552 Bytes
    Nov 1 20:13:44 localhost kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
    Nov 1 20:13:44 localhost kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
    Nov 1 20:13:44 localhost kernel: IPVS: ipvs loaded.
    Nov 1 20:13:44 localhost Keepalived_healthcheckers[36312]: Using LinkWatch kernel netlink reflector...
    Nov 1 20:13:44 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 1 20:13:45 localhost Keepalived_healthcheckers[36312]: Netlink reflector reports IP 10.1.7.19 added
    Nov 1 20:13:45 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 20:13:47 localhost ntpd[2238]: Listen normally on 8 eth0 10.1.7.19 UDP 123
    Nov 1 20:13:50 localhost Keepalived_vrrp[36313]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 [root@ _2_ ~]# ip a
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.19/16 scope global eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever
  2. 启动备用节点,可看到由于主节点正常运行,备用节点并未抢占主节点ip

     	[root@ _3_ ~]# service keepalived start
    Starting keepalived: [ OK ] [root@ _1_ ~]# tail /var/log/message
    Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Opening file '/etc/keepalived/keepalived.conf'.
    Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Configuration is using : 7556 Bytes
    Nov 1 20:21:44 localhost Keepalived_healthcheckers[2229]: Using LinkWatch kernel netlink reflector... [root@ _4_ ~]# ip a
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever
  3. 关闭主节点keepalived服务后

     	[root@ _6_ ~]# service keepalived stop
    Stopping keepalived: [ OK ] 以下为主节点日志:keepalived服务关闭,vrrp ip被移除
    Nov 1 20:28:17 localhost Keepalived[36349]: Stopping Keepalived v1.2.13 (03/19,2015)
    Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) sending 0 priority
    Nov 1 20:28:17 localhost Keepalived_vrrp[36352]: VRRP_Instance(VI_1) removing protocol VIPs.
    Nov 1 20:28:17 localhost Keepalived_healthcheckers[36351]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever 以下为备用节点日志:转换为MASTER角色,配置10.1.7.19的vrrp ip
    Nov 1 20:28:18 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 1 20:28:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 added
    Nov 1 20:28:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 20:28:24 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.19/16 scope global secondary eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever
  4. 恢复主节点,启动主节点keepalived服务后

     	[root@ _8_ ~]# service keepalived start
    Starting keepalived: [ OK ] 以下为主节点日志,转换为MASTER角色,抢占10.1.7.19的ip
    Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 1 20:34:20 localhost Keepalived_healthcheckers[36430]: Netlink reflector reports IP 10.1.7.19 added
    Nov 1 20:34:20 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 20:34:22 localhost ntpd[2238]: Listen normally on 10 eth0 10.1.7.19 UDP 123
    Nov 1 20:34:25 localhost Keepalived_vrrp[36431]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.19/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever 以下为备用节点日志,转换为BACKUP角色,ip 10.1.7.19被移除
    Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Received higher prio advert
    Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Nov 1 20:34:19 localhost Keepalived_vrrp[2231]: VRRP_Instance(VI_1) removing protocol VIPs.
    Nov 1 20:34:19 localhost Keepalived_healthcheckers[2229]: Netlink reflector reports IP 10.1.7.19 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever
  • keepalived 双主模型(10.1.7.19,10.1.7.20)
  1. 在单主模型的基础上,主机1再增加一个vrrp_instance段,改动的地方为

     	vrrp_instance VI_2 { 		#vrrp示例编号要修改,不能与其他实例相同
    state BACKUP #初始状态,上一个实例为MASTER,此时这里为BACKUP
    interface eth0
    virtual_router_id 17
    priority 98 #优先级要比另一台主机MASTER的优先级低
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 2a6561b9 #认证字符串要修改
    }
    virtual_ipaddress {
    10.1.7.20/16 dev eth0 #配置的另一个ip要修改
    }
    }
  2. 与之对应的另一台主机2上增加的配置为

     	 vrrp_instance VI_2 {
    state MASTER
    interface eth0
    virtual_router_id 17
    priority 100
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 2a6561b9
    }
    virtual_ipaddress {
    10.1.7.20/16 dev eth0
    }
    }
  • 测试
  1. 两台主机先停止keepalived服务,然后先启动主机1

     	以下为主机1日志:VI_1启动为MASTER角色,配置10.1.7.19的ip,VI_2启动为MASTER角色,配置10.1.7.20的ip
    
     	Nov  1 20:57:42 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 1 20:57:43 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.19 added
    Nov 1 20:57:43 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 20:57:45 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Transition to MASTER STATE
    Nov 1 20:57:45 localhost ntpd[2238]: Listen normally on 11 eth0 10.1.7.19 UDP 123
    Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering MASTER STATE
    Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) setting protocol VIPs.
    Nov 1 20:57:46 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20
    Nov 1 20:57:46 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 added
    Nov 1 20:57:47 localhost ntpd[2238]: Listen normally on 12 eth0 10.1.7.20 UDP 123
    Nov 1 20:57:48 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 20:57:51 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.19/16 scope global secondary eth0
    inet 10.1.7.20/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever
  2. 启动主机2

     	以下为主机1日志:VI_2转换为BACKUP角色,10.1.7.20的ip被移除:
    Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Received higher prio advert
    Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) Entering BACKUP STATE
    Nov 1 21:03:36 localhost Keepalived_vrrp[36523]: VRRP_Instance(VI_2) removing protocol VIPs.
    Nov 1 21:03:36 localhost Keepalived_healthcheckers[36522]: Netlink reflector reports IP 10.1.7.20 removed 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.19/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever 以下为主机2的日志,VI_2转换为MASTER角色,配置了10.1.7.20的ip
    Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Transition to MASTER STATE
    Nov 1 21:03:36 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Received lower prio advert, forcing new election
    Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Entering MASTER STATE
    Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) setting protocol VIPs.
    Nov 1 21:03:37 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.20 added
    Nov 1 21:03:37 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20
    Nov 1 21:03:42 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_2) Sending gratuitous ARPs on eth0 for 10.1.7.20 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.20/16 scope global secondary eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever
  3. 停止主机1的keepalived服务。

     	以下为主机2日志:VI_1转换为MASTER角色,配置了10.1.7.19的ip
    Nov 1 21:07:47 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 1 21:07:48 localhost Keepalived_healthcheckers[2378]: Netlink reflector reports IP 10.1.7.19 added
    Nov 1 21:07:48 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19
    Nov 1 21:07:53 localhost Keepalived_vrrp[2380]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.7.19 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet 10.1.7.20/16 scope global secondary eth0
    inet 10.1.7.19/16 scope global secondary eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever

3. Keepalived集群+ipvs(DR)集群

拓扑环境

		10.1.6.11和10.1.6.12为两台real server,提供web服务。
左边两台服务器,主节点ip10.1.6.11,备节点ip10.1.6.12
主节点和备节点做成keepalived高可用集群。IP为10.1.8.88

keepalived高可用

  1. 两台real server 安装httpd,编辑测试主页,启动httpd服务,在主节点或备用节点上请求测试主页

     	[root@ _2_ ~]# yum -y install httpd
    [root@ _2_ ~]# cat /var/www/html/index.html
    <h1>Server 1</h1> [root@ _2_ ~]# yum -y install httpd
    [root@ _2_ ~]# cat /var/www/html/index.html
    <h1>Server 2</h1> [root@ _3_ ~]# curl http://10.1.7.11
    <h1>Server 1</h1>
    [root@ _4_ ~]# curl http://10.1.7.12
    <h1>Server 2</h1>
  2. Real Server上编写DR模型初始配置脚本,两台real server运行此脚本

     	#!/bin/bash
    
     	vip='10.1.8.88'
    vport='80'
    netmask='255.255.255.255'
    iface='lo:0' case $1 in
    start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up
    route add -host $vip dev $iface
    ;;
    stop)
    ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;;
    *)
    echo "Usage $(basename $0) start|stop"
    exit 1
    ;;
    esac 检查配置
    lo:0: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
    inet 10.1.8.88 netmask 255.255.255.255
    loop txqueuelen 0 (Local Loopback)
  3. 主节点和备用节点上配置keepalived

     以下为主节点配置,备用节点需修改state为BACKUP
    ! Configuration File for keepalived global_defs {
    notification_email {
    root@localhost
    }
    notification_email_from Keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id node1
    vrrp_mcast_group4 224.0.200.158
    } vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 16
    priority 98
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 2a6561b8
    }
    virtual_ipaddress {
    10.1.8.88/16 dev eth0
    }
    }
  4. 测试主备节点分别故障时,虚拟ip能来回切换

  5. 主节点和备节点安装ipvsadm,测试调度后端real server,确保调度正常

     主节点:
    [root@ _8_ ~]# yum -y install ipvsadm
    [root@ _8_ ~]# ipvsadm -A -t 10.1.8.88:80 -s rr
    [root@ _9_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.11 -g -w 1
    [root@ _10_ ~]# ipvsadm -a -t 10.1.8.88:80 -r 10.1.7.12 -g -w 1
    [root@ _11_ ~]# ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.8.88:80 rr
    -> 10.1.7.11:80 Route 1 0 0
    -> 10.1.7.12:80 Route 1 0 0 [root@ _13_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1> 调度正常,清空规则
    [root@ _25_ ~]# ipvsadm -C 备用节点同样方法测试一次
  6. 定义keepalived配置中Virtual Server

     主节点和备节点的keepalived配置文件中加入Virtual server配置段
    virtual_server 10.1.8.88 80 { #virtual_server ip地址
    delay_loop 3 #服务轮询时间间隔
    lb_algo rr #定义调度算法
    lb_kind DR #定义lvs的类型
    protocol TCP #服务协议,仅支持tcp real_server 10.1.7.11 80 { #real_server ip地址
    weight 1 #权重
    HTTP_GET { #请求方法
    url {
    path / #定义监控的url
    status_code 200 #判断上述检测机制为健康状态的响应码为200
    }
    connect_timeout 1 #连接超时时间
    nb_get_retry 3 #重试的次数
    delay_before_retry 1 #重试之前延迟时长
    }
    } real_server 10.1.7.12 80 {
    weight 1
    HTTP_GET {
    url {
    path /
    status_code 200
    }
    connect_timeout 1
    nb_get_retry 3
    delay_before_retry 1
    }
    }
    }
  7. 启动主节点和备节点keepalived服务,查看ip以及ipvs规则

     主节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever [root@ _33_ ~]# ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.8.88:80 rr
    -> 10.1.7.11:80 Route 1 0 0
    -> 10.1.7.12:80 Route 1 0 0 备节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link
    valid_lft forever preferred_lft forever [root@ _28_ ~]# ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.8.88:80 rr
    -> 10.1.7.11:80 Route 1 0 0
    -> 10.1.7.12:80 Route 1 0 0
  8. 使用客户端对其进行访问检测

     测试正常
    [root@ _16_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
  9. 使real server中有一个故障,检测访问

     停掉real server的httpd服务
    [root@ _5_ ~]# systemctl stop httpd 主节点上查看ipvs规则,real server 2已下线 [root@ _38_ ~]# ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.8.88:80 rr
    -> 10.1.7.11:80 Route 1 0 10 客户端请求访问
    [root@ _17_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    <h1>Server 1</h1>
    访问正常
  10. 使故障的real server恢复,检测访问

    	[root@ _6_ ~]# systemctl start httpd
    
    查看主节点上ipvs规则,real server已加入
    [root@ _39_ ~]# ipvsadm -Ln
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.8.88:80 rr
    -> 10.1.7.11:80 Route 1 0 0
    -> 10.1.7.12:80 Route 1 0 0 客户端访问测试
    [root@ _18_ ~]# for i in {1..10};do curl http://10.1.8.88 ;done
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    <h1>Server 2</h1>
    <h1>Server 1</h1>
    调度正常

4. keepalived主备节点上配置sorry server

  1. 主备节点分别安装httpd,编辑网页文件,最好都停掉keepalived服务

     	[root@ _41_ ~]# yum -y install httpd
    
     主节点编辑网页文件
    [root@ _38_ ~]# cat /var/www/html/index.html
    <h1>LB Cluster Fault,this is Sorry Server 1</h1> 主节点编辑网页文件
    [root@ _38_ ~]# cat /var/www/html/index.html
    <h1>LB Cluster Fault,this is Sorry Server 2</h1>
  2. 编辑keepalived配置文件,在Virtual server中添加sorry server配置,主节点和备节点都要配置

     	virtual_server 10.1.8.88 80 {
    delay_loop 3
    lb_algo rr
    lb_kind DR
    protocol TCP sorry_server 127.0.0.1 80 real_server 10.1.7.11 80 {
    weight 1
    ...
  3. 主备节点都启动httpd服务,启动keepalived服务,两台real server都停止httpd服务

     	[root@ _48_ ~]# service httpd start
    
     	[root@ _44_ ~]# service keepalived start
    Starting keepalived: [ OK ] [root@ _12_ ~]# systemctl stop httpd
  4. 客户端请求测试

     看到sorry server的响应
    [root@ _22_ ~]# curl http://10.1.8.88
    <h1>LB Cluster Fault,this is Sorry Server 1</h1>
  5. 启动一台real server的httpd服务,客户端测试

     响应正常
    [root@ _23_ ~]# curl http://10.1.8.88
    <h1>Server 2</h1>

5.keeplioved调用外部脚本,由结果实时调整优先级

  1. 脚本的定义与调用介绍

  2.  (1)脚本先定义
    vrrp_script<SCRIPT_NAME> {
    script ""
    interval INT
    weight -INT
    }
    (2)再调用
    track_script {
    SCRIPT_NAME_1
    SCRIPT_NAME_2
    ...
    }
  3. 主备节点keepalived配置加入脚本段,脚本检测到/etc/keepalived/down则返回失败

     主节点
    ! Configuration File for keepalived global_defs {
    notification_email {
    root@localhost
    }
    notification_email_from Keepalived@localhost
    smtp_server 127.0.0.1
    smtp_connect_timeout 30
    router_id node1
    vrrp_mcast_group4 224.0.200.158
    } vrrp_script chk_down { #定义脚本名
    script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0" #判断down文件,存在则返回失败
    interval 1 #每隔多长时间脚本执行一次
    weight -5 #脚本失败动作,权重-5,确保-5后低于备用优先级
    } vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 16
    priority 100
    advert_int 1
    authentication {
    auth_type PASS
    auth_pass 2a6561b8
    }
    virtual_ipaddress {
    10.1.8.88/16 dev eth0
    }
    track_script { #监控的脚本
    chk_down #调用的脚本名称
    }
    } 备节点需要修改state为BACKUP,priority为98
  4. 主备节点启动keepalived服务,查看ip

     	[root@ _72_ /etc/keepalived]# service keepalived stop
    Starting keepalived: [ OK ] 主节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0 备节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
  5. 创建/etc/keepalived/down文件,查看ip转移

     	[root@ _161_ /etc/keepalived]# touch down
    
     主节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link 备节点
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link 主节点检测脚本,返回失败,权重-5,转换为BACKUP角色,ip 10.1.8.88被移除
    Nov 3 08:24:02 localhost Keepalived_vrrp[4853]: VRRP_Script(chk_down) failed
    Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Received higher prio advert
    Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Entering BACKUP STATE
    Nov 3 08:24:03 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) removing protocol VIPs.
    Nov 3 08:24:03 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 removed
  6. 删除主节点/etc/keepalived/down文件,查看ip转移

     	[root@ _163_ /etc/keepalived]# rm -rf down
    
     主节点,ip已夺回
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link 日志
    Nov 3 08:32:01 localhost Keepalived_healthcheckers[4852]: Netlink reflector reports IP 10.1.8.88 added
    Nov 3 08:32:01 localhost Keepalived_vrrp[4853]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 10.1.8.88
    Nov 3 08:32:03 localhost ntpd[4558]: Listen normally on 11 eth0 10.1.8.88 UDP 123

6. keepalived结合nginx调度(并使用辅助脚本检测nginx服务)

  1. 清空上例在real server上所做的lo:0接口规则以及ARP限制规则,并停止keepalived服务

     	[root@ _16_ ~]# bash set_dr stop
    
     	#!/bin/bash
    
     	vip='10.1.8.88'
    vport='80'
    netmask='255.255.255.255'
    iface='lo:0' case $1 in
    start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce ifconfig $iface $vip netmask $netmask broadcast $vip up
    route add -host $vip dev $iface
    ;;
    stop)
    ifconfig $iface down echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce ;;
    *)
    echo "Usage $(basename $0) start|stop"
    exit 1
    ;;
    esac
  2. 主备节点停止为sorry server启动的httpd服务

     	[root@ _50_ ~]# service httpd stop
    Stopping httpd: [ OK ]
  3. 主备节点安装nginx

     	[root@ _173_ /etc/keepalived]# yum -y install nginx
  4. 编辑nginx配置文件,实现反代

     在/etc/nginx/nginx.conf的http上下文中添加
    upstream websrvs {
    server 10.1.7.11;
    server 10.1.7.12;
    }
    在/etc/nginx/conf.d/default.conf的location上下文中添加 proxy_pass http://websrvs;

    location / {
    root /usr/share/nginx/html;
    proxy_pass http://websrvs;
    index index.html index.htm;
    }
  5. 主备节点启动nginx服务,keepalived服务,访问测试

     	[root@ _18_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>
    [root@ _19_ /etc]# curl http://10.1.8.88
    <h1>Server 2</h1>
    [root@ _20_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>
    [root@ _21_ /etc]# curl http://10.1.8.88
    <h1>Server 2</h1>
    访问正常
  6. 主备节点添加vrrp_script脚本,在nginx没有启动时触发

     	vrrp_script chk_nginx {
    script "killall -0 nginx && exit 0 || exit 1"
    interval 1
    weigth
    } 追踪脚本中也需要加入chk_nginx
    track_script {
    chk_down
    chk_nginx
    }
  7. 主节点重启keepalived服务,而后备节点重启keepalived服务

     此时虚拟ip 10.1.8.88在主节点上
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever 客户端访问正常
    [root@ _22_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>
    [root@ _23_ /etc]# curl http://10.1.8.88
    <h1>Server 2</h1>
    [root@ _24_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>
  8. 停止主节点nginx服务

     	[root@ _12_ ~]# service nginx stop
    Stopping nginx: [ OK ] 主节点ip已移除
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:9c:14:7c brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.11/16 brd 10.1.255.255 scope global eth0
    inet6 fe80::20c:29ff:fe9c:147c/64 scope link
    valid_lft forever preferred_lft forever 日志信息,检测到vrrp_script失败,转换为BACKUP模式,ip已移除
    Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Script(chk_nginx) failed
    Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Entering FAULT STATE
    Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) removing protocol VIPs.
    Nov 3 18:00:25 localhost Keepalived_vrrp[75164]: VRRP_Instance(VI_1) Now in FAULT state
    Nov 3 18:00:25 localhost Keepalived_healthcheckers[75163]: Netlink reflector reports IP 10.1.8.88 removed 备节点ip与日志,10.1.8.88地址已获取,转换为MASTER角色
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
    inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
    inet 10.1.8.88/16 scope global secondary eth0
    inet6 fe80::20c:29ff:feaf:fdec/64 scope link Nov 3 18:00:26 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Transition to MASTER STATE
    Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) Entering MASTER STATE
    Nov 3 18:00:27 localhost Keepalived_vrrp[75084]: VRRP_Instance(VI_1) setting protocol VIPs.
    Nov 3 18:00:27 localhost Keepalived_healthcheckers[75083]: Netlink reflector reports IP 10.1.8.88 added
  9. 客户端访问测试

     调度正常
    [root@ _25_ /etc]# curl http://10.1.8.88
    <h1>Server 2</h1>
    [root@ _26_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>
    [root@ _27_ /etc]# curl http://10.1.8.88
    <h1>Server 2</h1>
    [root@ _28_ /etc]# curl http://10.1.8.88
    <h1>Server 1</h1>

补充:keepalived发生角色转移时运行指定脚本

  1. 备节点上编写脚本/etc/keepalived/motify.sh,当角色切换时,给root用户发送邮件

     		#!/bin/bash
    #
    contact='root@localhost' notify() {
    mailsubject="$(hostname) to be $1, vip floating."
    mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
    echo "$mailbody" | mail -s "$mailsubject" $contact
    } case $1 in
    master)
    notify master
    ;;
    backup)
    notify backup
    ;;
    fault)
    notify fault
    ;;
    *)
    echo "Usage: $(basename $0) {master|backup|fault}"
    exit 1
    ;;
    esac
  2. 在vrrp_instance端中调用脚本,并重启keepalived服务

     	    notify_master "/etc/keepalived/notify.sh master"
    notify_backup "/etc/keepalived/notify.sh backup"
    notify_fault "/etc/keepalived/notify.sh fault"

3.主节点停止keepalived服务,查看root用户邮件

	备节点:收到转换为master角色的邮件
>N 1 root Thu Nov 3 18:41 18/731 "localhost.localdomain to be master, vip floating."
& 1
Message 1:
From root@localhost.localdomain Thu Nov 3 18:41:46 2016
Return-Path: <root@localhost.localdomain>
X-Original-To: root@localhost
Delivered-To: root@localhost.localdomain
Date: Thu, 03 Nov 2016 18:41:46 +0800
To: root@localhost.localdomain
Subject: localhost.localdomain to be master, vip floating.
User-Agent: Heirloom mailx 12.4 7/29/08
Content-Type: text/plain; charset=us-ascii
From: root@localhost.localdomain (root)
Status: R 2016-11-03 18:41:46: vrrp transition, localhost.localdomain changed to be master ip 10.1.8.88已添加
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:af:fd:ec brd ff:ff:ff:ff:ff:ff
inet 10.1.6.12/16 brd 10.1.255.255 scope global eth0
inet 10.1.8.88/16 scope global secondary eth0