整合apache+tomcat+keepalived实现高可用tomcat集群

时间:2022-05-05 08:51:12

Apache是一个强大的Web服务器在处理静态页面、处理大量网络客户请求、支持服务的种类以及可配置方面都有优势,高速并且强壮。但是没有JSP/Servlet的解析能力。整合Apache和Tomcat可以看作是用Tomcat做Apache的jsp/servlet解析插件,将两者优势结合起来。不过Tomcat作为一个Web服务器,本身具备了基本的Web服务功能,在SUN的力推下,将来或许越来越强壮到不需要借助Apache优势的地步。

Tomcat是一个免费的开放源代码的Web 应用服务器,属于轻量级应用服务器,在中小型系统和并发访问用户不是很多的场合下被普遍使用,是开发和调试JSP 程序的首选。Tomcat 很受广大程序员的喜欢,因为它运行时占用的系统资源小,扩展性好,支持负载平衡与邮件服务等开发应用系统常用的功能;而且它还在不断的改进和完善中,任何一个感兴趣的程序员都可以更改它或在其中加入新的功能。

Apache与tomcat的异同:

apache支持静态页,tomcat支持动态的,比如servlet等。

一般使用apache+tomcat的话,apache只是作为一个转发,对jsp的处理是由tomcat来处理的。

apache可以支持php\cgi\perl,但是要使用java的话,你需要tomcat在apache后台支撑,将java请求由apache转发给tomcat处理。

apache是web服务器,Tomcat是应用(java)服务器,它只是一个servlet(jsp也翻译成servlet)容器,可以认为是apache的扩展,但是可以独立于apache运行。

Apache是专门用了提供HTTP服务的,以及相关配置的(例如虚拟主机、URL转发等等)。

Tomcat是Apache组织在符合J2EE的JSP、Servlet标准下开发的一个JSP服务器;

Apache与tomcat整合的好处

如果客户端请求的是静态页面,则只需要Apache服务器响应请求。

如果客户端请求动态页面,则是Tomcat服务器响应请求。

因为jsp是服务器端解释代码的,这样整合就可以减少Tomcat的服务开销。

一.环境与功能说明

整合apache+tomcat+keepalived实现高可用tomcat集群

前端:apache配置负载均衡、反向代理(mod_jk模块)

后端:tomcat服务器

前端高可用:通过keepalived实现

系统版本:CentOS 6.4_x86_64(最小化)

apache版本:httpd-2.4.9

arp版本:apr-1.4.6

apr-util版本:apr-util-1.4.1

tomcat版本:apache-tomcat-7.0.53

Jdk版本:jdk-7u40-linux-x64

tomcat-commectors版本:tomcat-connectors-1.2.40

keepalived版本:keepalived-1.2.7-3.el6.x86_64(使用yum安装)

二.安装与配置apache服务器(172.16.10.72、172.16.10.75执行同样的安装与配置,这里只演示一次)

1.安装编译工具和依赖包

yum groupinstall "Development Tools" "Development Libraries" –y

2.下载软件包

yum -y install wget
wget http://mirror.esocc.com/apache//httpd/httpd-2.4.9.tar.gz
wget http://mirrors.axint.net/apache//apr/apr-1.4.6.tar.gz
wget http://mirrors.axint.net/apache//apr/apr-util-1.4.1.tar.gz

3.安装apache

3.1.安装apr和apr-tuil

httpd-2.4.9需要较新版本的apr和apr-util,因此需要事先对其进行升级。升级方式有两种,一种是通过源代码编译安装,一种是直接升级rpm包。以下是采用前者进行安装:

# tar xf apr-1.4..tar.gz
# tar xf apr-util-1.4..tar.gz
# cd apr-1.4.
# ./configure --prefix=/usr/local/apr
# make
# make install
# cd .. # cd apr-util-1.4.
# ./configure --prefix=/usr/local/apr-util --with-apr=/usr/local/apr
# make
# make install
# cd ..

3.2.安装http-2.4.9

# ./configure --prefix=/usr/local/apache --sysconfdir=/etc/httpd24 --enable-so --enable-ssl --enable-cgi --enable-rewrite --with-zlib --with-pcre --with-apr=/usr/local/apr --with-apr-util=/usr/local/apr-util/ --enable-modules=most --enable-mpms-shared=all
# make && make install

4.配置SysV服务脚本/etc/init.d/httpd

# cp build/rpm/httpd.init /etc/init.d/httpd
#修改如下内容:

# vi /etc/init.d/httpd
httpd=${HTTPD-/usr/local/apache/bin/httpd}
pidfile=${PIDFILE-/usr/local/apache/logs/${prog}.pid}
lockfile=${LOCKFILE-/var/lock/subsys/${prog}}
RETVAL=
# check for 1.3 configuration
check13 () {
CONFFILE=/etc/httpd24/httpd.conf #修改环境变量并把Httpd加入到系统服务

# echo "PATH=/usr/local/apache/bin:$PATH" >> /etc/profile.d/http.sh
  # . /etc/profile.d/http.sh
  # ln -s /usr/local/apache/include/ /usr/include/httpd
  # chkconfig --add httpd

5.安装tomcat-connectors

# wget http://apache.fayea.com/apache-mirror/tomcat/tomcat-connectors/jk/tomcat-connectors-1.2.40-src.tar.gz
# tar xf tomcat-connectors-1.2.-src.tar.gz
# cd tomcat-connectors-1.2.-src/native/
# ./configure --with-apxs=/usr/local/apache/bin/apxs
# make
# make install

安装完成后会生成如下文件

# ls /usr/local/apache/modules/|grep 'mod_jk'
mod_jk.so

启动httpd服务

# service httpd start

执行./configure --with-apxs=/usr/local/apache/bin/apxs时可能会遇到以下报错:

need to check for Perl first, apxs depends on it...
checking for perl... /usr/bin/perl
could not find /usr/local/apache/bin/apxs
configure: error: You must specify a valid --with-apxs path

解法办法:

vi /usr/local/apache/bin/apxs
把第一行改成:#!/usr/bin/perl -w

另外检查系统中是否已经安装过perl,如果没有安装,用yum安装即可。

三.安装与配置tomcat服务器(172.16.10.61、172.16.10.62执行同样的安装,这里只演示一次)

1.安装JDK

下载地址:http://download.oracle.com/

# rpm -ivh jdk-7u40-linux-x64.rpm
# vim /etc/profile.d/java.sh
export JAVA_HOME=/usr/java/latest
export PATH=$JAVA_HOME/bin:$PATH
# . /etc/profile.d/java.sh
# java -version
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) -Bit Server VM (build 24.0-b56, mixed mode)

2.安装Tomcat

下载地址:http://apache.fayea.com/apache-mirror/tomcat/tomcat-7/v7.0.53/bin/apache-tomcat-7.0.53.tar.gz

# tar xf apache-tomcat-7.0..tar.gz -C /usr/local/
# cd /usr/local/
# ln -s apache-tomcat-7.0. tomcat
# vim /etc/profile.d/tomcat.sh
export CATALINA_HOME=/usr/local/tomcat
export PATH=$CATALINA_HOME/bin:$PATH
# . /etc/profile.d/tomcat.sh
# /usr/local/tomcat/bin/catalina.sh version
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/7.0.
Server built: Mar ::
Server number: 7.0.53.0
OS Name: Linux
OS Version: 2.6.-.el6.x86_64
Architecture: amd64
JVM Version: 1.7.0_40-b43
JVM Vendor: Oracle Corporation

3.启动Tomcat服务并访问测试,默认访问端口为8080:

# /usr/local/tomcat/bin/catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
# ss -antpl | grep
::: :::* users:(("java",,))

整合apache+tomcat+keepalived实现高可用tomcat集群

4.开启Tomcat管理及状态页面

# vim /usr/local/tomcat/conf/tomcat-users.xml(在</tomcat-users>标签上一行添加如下两行)
<role rolename="manager-gui"/>
<user username="tomcat" password="tomcat" roles="manager-gui"/> #后台管理页面登录名及密码均为tomcat
# /usr/local/tomcat/bin/catalina.sh stop
# /usr/local/tomcat/bin/catalina.sh start

5.访问Tomcat后台管理

整合apache+tomcat+keepalived实现高可用tomcat集群

整合apache+tomcat+keepalived实现高可用tomcat集群

四.配置Apache使用mod_jk模块实现代理及负载均衡

1.编辑Apache主配置文件,实现代理功能(172.16.10.72)

echo "Include /etc/httpd24/extra/mod_jk.conf" >> /etc/httpd24/httpd.conf  #将此指令添加到文件末尾即可

2.创建mod_jk.conf及workers.properties文件

# cat >>/etc/httpd24/extra/mod_jk.conf <<EOF
LoadModule jk_module modules/mod_jk.so
JkWorkersFile /etc/httpd24/extra/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel debug
JkMount /* TomcatA
JkMount /status/ stat1
EOF # cat >>/etc/httpd24/extra/workers.properties <<EOF
worker.list=TomcatA,stat1
worker.TomcatA.port=8009
worker.TomcatA.host=172.16.10.61
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1
worker.stat1.type = status
EOF

3.编辑httpd.conf修改ServerName并重启httpd服务

#vi /etc/httpd24/httpd.conf
ServerName 172.16.10.72:80
# service httpd restart

4.修改tomcat服务器(172.16.10.61),添加测试页面

# cd /usr/local/tomcat/conf/
# cp server.xml server.xml.bak
# vi server.xml
<Engine name="Catalina" defaultHost="172.16.10.72">
<Host name="172.16.10.72" appBase="/web/eivll0m"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="eivll0m_access_log." suffix=".txt"
pattern="%h %l %u %t &quot;%r&quot; %s %b" />
<Context path="" docBase="/web/eivll0m" />
</Host>

创建网站存放目录并创建测试页

# mkdir -p /web/eivll0m
# vi /web/eivll0m/index.jsp
<%@ page language="java" %>
<html>
<head><title>TomcatA</title></head>
<body>
<h1><font color="red">TomcatA </font></h1>
<table align="centre" border="">
<tr>
<td>Session ID</td>
<% session.setAttribute("abc","abc"); %>
<td><%= session.getId() %></td>
</tr>
<tr>
<td>Created on</td>
<td><%= session.getCreationTime() %></td>
</tr>
</table>
</body>
</html>

重启tomcat服务并查看是否启动成功

# catalina.sh stop
# catalina.sh configtest
# catalina.sh start
# ss -antpl | grep java
::ffff:127.0.0.1: :::* users:(("java",,))
::: :::* users:(("java",,))
::: :::* users:(("java",,))

5.访问Apache主机(172.16.10.72),验证是否代理成功

整合apache+tomcat+keepalived实现高可用tomcat集群

6.修改Apache配置文件(mod_jk.conf、workers.properties),实现负载均衡

# cat >>/etc/httpd24/extra/mod_jk.conf <<EOF
LoadModule jk_module modules/mod_jk.so
JkWorkersFile /etc/httpd24/extra/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel debug
JkMount /* lbcluster
JkMount /status/ stat1
EOF

# cat >>/etc/httpd/extra/workers.properties <<EOF
worker.list=lbcluster,stat1
worker.TomcatA.port=8009
worker.TomcatA.host=172.16.10.61
worker.TomcatA.type=ajp13
worker.TomcatA.lbfactor=1
worker.TomcatB.port = 8009
worker.TomcatB.host=172.16.10.62
worker.TomcatB.type = ajp13
worker.TomcatB.lbfactor = 1
worker.lbcluster.type = lb
worker.lbcluster.sticky_session = 0 #取值为{1|0}1将用户session与后端服务器绑定,0为不绑定,如果支持session复制或session共享可以设置为0
worker.lbcluster.balance_workers = TomcatA, TomcatB
worker.stat1.type = status
EOF

7.修改tomcat服务器(172.16.10.62),添加测试页面

# cd /usr/local/tomcat/conf/
# cp server.xml server.xml.bak
# vi server.xml
<Engine name="Catalina" defaultHost="172.16.10.72">
<Host name="172.16.10.72" appBase="/web/eivll0m"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="eivll0m_access_log." suffix=".txt"
pattern="%h %l %u %t &quot;%r&quot; %s %b" />
<Context path="" docBase="/web/eivll0m" />
</Host>

创建网站存放目录并创建测试页

# mkdir -p /web/eivll0m
# vi /web/eivll0m/index.jsp
<%@ page language="java" %>
<html>
<head><title>TomcatB</title></head>
<body>
<h1><font color="blue">TomcatB </font></h1>
<table align="centre" border="">
<tr>
<td>Session ID</td>
<% session.setAttribute("abc","abc"); %>
<td><%= session.getId() %></td>
</tr>
<tr>
<td>Created on</td>
<td><%= session.getCreationTime() %></td>
</tr>
</table>
</body>
</html>

重启tomcat服务并查看是否启动成功

# catalina.sh stop
# catalina.sh configtest
# catalina.sh start
# ss -antpl | grep java
::ffff:127.0.0.1: :::* users:(("java",,))
::: :::* users:(("java",,))
::: :::* users:(("java",,))

8.访问Apache服务器(172.16.10.72),测试是否实现负载均衡(如下表示测试成功)

整合apache+tomcat+keepalived实现高可用tomcat集群

整合apache+tomcat+keepalived实现高可用tomcat集群

五.安装并配置Keepalived实现高可用

1.在apache服务器(172.16.10.72、172.16.10.75)安装keepalived

yum -y install keepalived

2.配置Keepalived(172.16.10.72)

# vi /etc/keepalived/keepalived.conf
! Configuration File for keepalived global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from admin@eivll0m.com
smtp_server 172.16.10.0
smtp_connect_timeout
router_id LVS_DEVEL
} vrrp_script chk_httpd {
script "killall -0 httpd"
interval
weight -
} vrrp_instance httpd_1 {
state MASTER
interface eth0
virtual_router_id
priority 100 #定义优先级
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
172.16.10.77 #vip
}
track_script {
chk_httpd
}
} # service keepalived start #启动keepalived服务
# chkconfig keepalived on

3.配置Keepalived(172.16.10.75)

# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from admin@eivll0m.com
smtp_server 172.17.10.0
smtp_connect_timeout
router_id LVS_DEVEL
} vrrp_script chk_httpd {
script "killall -0 httpd"
interval
weight -
} vrrp_instance httpd_1 {
state BACKUP
interface eth0
virtual_router_id
priority 99 #定义优先级
advert_int
authentication {
auth_type PASS
auth_pass
}
virtual_ipaddress {
172.16.10.77 #vip
}
track_script {
chk_httpd
}
}
# service keepalived start  
# chkconfig keepalived on

4.编辑httpd.conf(172.16.10.72),并拷贝至172.16.10.75

# vi /etc/httpd24/httpd.conf
ServerName www.eivll0m.com:
# service httpd restart
# cd /etc/httpd24/
# scp httpd.conf 172.16.10.75:/etc/httpd24/
# cd extra/
# scp mod_jk.conf workers.properties 172.16.10.75:/etc/httpd24/extra/
# service httpd restart #两台apache服务器分别执行

5.编辑tomcat服务器(172.17.10.61和172.16.10.62)的server.xml

# vi server.xml
<Engine name="Catalina" defaultHost="www.eivll0m.com">
<Host name="www.eivll0m.com" appBase="/web/eivll0m"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="eivll0m_access_log." suffix=".txt"
pattern="%h %l %u %t &quot;%r&quot; %s %b" />
<Context path="" docBase="/web/eivll0m" />
</Host>
# catalina.sh stop
# catalina.sh start

5.通过www.eivll0m.com(172.16.10.77)进行访问测试

注意:测试时关闭所有服务器iptables,如果没有做DNS解析,可在测试机上修改hosts文件来实现。

整合apache+tomcat+keepalived实现高可用tomcat集群

整合apache+tomcat+keepalived实现高可用tomcat集群

6.模拟一台apache故障,验证keepalived是否可以成功转移服务

# ip addr show eth0   #172.16..72上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c::e9:b8: brd ff:ff:ff:ff:ff:ff
inet 172.16.10.72/ brd 172.16.10.255 scope global eth0
inet 172.16.10.77/ scope global eth0
inet6 fe80::20c:29ff:fee9:b852/ scope link tentative dadfailed
valid_lft forever preferred_lft forever # ip addr show eth0 #172.16..75上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c::c8:1a: brd ff:ff:ff:ff:ff:ff
inet 172.16.10.75/ brd 172.16.10.255 scope global eth0
inet 192.168.200.16/ scope global eth0
inet 192.168.200.17/ scope global eth0
inet 192.168.200.18/ scope global eth0
inet6 fe80::20c:29ff:fec8:1a08/ scope link tentative dadfailed
valid_lft forever preferred_lft forever # service httpd stop #停止172.16.10.72上的httpd服务
# ip addr show eth0 #172.16..72上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c::e9:b8: brd ff:ff:ff:ff:ff:ff
inet 172.16.10.72/ brd 172.16.10.255 scope global eth0
inet6 fe80::20c:29ff:fee9:b852/ scope link tentative dadfailed
valid_lft forever preferred_lft forever # ip addr show eth0 #172.16.10.75上执行
: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu qdisc pfifo_fast state UP qlen
link/ether :0c::c8:1a: brd ff:ff:ff:ff:ff:ff
inet 172.16.10.75/ brd 172.16.10.255 scope global eth0
inet 192.168.200.16/ scope global eth0
inet 192.168.200.17/ scope global eth0
inet 192.168.200.18/ scope global eth0
inet 172.16.10.77/ scope global eth0 #vip已经成功转移过来
inet6 fe80::20c:29ff:fec8:1a08/ scope link tentative dadfailed
valid_lft forever preferred_lft forever

通过上面的步骤我们可以看到,在172.16.10.72上停止apache服务,keepalived会检测到,另一台keepalived服务器会接管vip,web服务不会中断,从而实现了高可用性。

7.配置tomcat实现session共享

7.1.配置172.16.10.61这台tomcat服务器

# vi /usr/local/tomcat/conf/server.xml  #在<Engine>标签内(对所有主机生效)添加或修改如下内容,另外也可以添加在<Host>标签内,对当前主机生效。并去掉此段前后注释
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="">
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="224.0.0.1" #组播地址
port=""
frequency=""
dropTime=""/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="172.16.10.61"
port=""
autoBind=""
selectorTimeout=""
maxThreads=""/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=""/>
<Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
# mkdir /web/eivll0m/WEB-INF
# cp /usr/local/tomcat/conf/web.xml /web/eivll0m/WEB-INF/
# vim /web/eivll0m/WEB-INF/web.xml
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
version="3.0">
<distributable/> #添加此行内容
# catalina.sh stop
# catalina.sh start
# ss -tanlp | grep java
::ffff:127.0.0.1: :::* users:(("java",,))
::: :::* users:(("java",,))
::: :::* users:(("java",,))
::ffff:172.16.10.61: :::* users:(("java",,))

7.2.配置172.16.10.62这台tomcat服务器

# scp /usr/local/tomcat/conf/server.xml 172.16.10.62:/usr/local/tomcat/conf/
# scp -r /web/eivll0m/WEB-INF 172.16.10.62:/web/eivll0m/
# vi /usr/local/tomcat/conf/server.xml
address="172.16.10.62" #更改IP
port=""
autoBind=""
selectorTimeout=""
maxThreads=""/>
# catalina.sh stop
# catalina.sh start
# ss -antpl|grep java
::ffff:127.0.0.1: :::* users:(("java",,))
::: :::* users:(("java",,))
::: :::* users:(("java",,))
::ffff:172.16.10.62: :::* users:(("java",,))

注:因为tomcat的session同步功能需要用到组播,windows默认情况下是开通组播服务的,但是linux默认情况下并没有开通,可以通过指令打开route add -net 224.0.0.0 netmask 240.0.0.0 dev eth0,如果需要服务器启动时即开通组播需在/etc/sysconfig/static-routes文件内加入eht0 net 224.0.0.0 netmask 240.0.0.0。

7.3.验证session共享是否实现

整合apache+tomcat+keepalived实现高可用tomcat集群

整合apache+tomcat+keepalived实现高可用tomcat集群

通过以上截图可以看出session共享已经实现。