Tomcat9.0.13 Bug引发的java.io.IOException:(打开的文件过多 Too many open files)导致服务假死

时间：2021-07-16 10:51:35

问题背景：

笔者所在的项目组最近把生产环境Tomcat迁移到Linux，算是顺利运行了一段时间，最近一个低概率密度的（too many open files)问题导致服务假死并停止响应客户端客户端请求。

进入服务器查看日志，发现tomcat凌晨6-7点的日志丢失，查看进程端口仍旧开放。

root@# lsof -i:

COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

java     root   67u  IPv4       0t0  TCP *: (LISTEN)

从存档的日志找到一些端倪，发现凌晨1点的日志就开始出现异常

-Dec- ::12.514 严重 [http-nio--Acceptor-] org.apache.tomcat.util.net.Acceptor.run Socket accept failed

 java.io.IOException: 打开的文件过多

    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:)

    at java.lang.Thread.run(Thread.java:)

错误日志一直追到凌晨6-7点，这个错误仍旧比较高密度的出现

-Dec- ::06.932 严重 [http-nio--Acceptor-] org.apache.tomcat.util.net.Acceptor.run Socket accept failed

 java.io.IOException: 打开的文件过多

    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:)

    at java.lang.Thread.run(Thread.java:)

-Dec- ::07.692 严重 [http-nio--Acceptor-] org.apache.tomcat.util.net.Acceptor.run Socket accept failed

 java.io.IOException: 打开的文件过多

    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:)

    at java.lang.Thread.run(Thread.java:)

-Dec- ::08.532 严重 [http-nio--Acceptor-] org.apache.tomcat.util.net.Acceptor.run Socket accept failed

 java.io.IOException: 打开的文件过多

    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:)

    at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:)

    at java.lang.Thread.run(Thread.java:)

故障排除步骤：

笔者最初怀疑自己在预安装生产环境的过程中忘记调优内核参数，于是按步骤查询了最大文件打开数：

root@# ulimit -a

core file size          (blocks, -c)

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e)

file size               (blocks, -f) unlimited

pending signals                 (-i)

max locked memory       (kbytes, -l)

max memory size         (kbytes, -m) unlimited

open files                      (-n) 4096

pipe size            ( bytes, -p)

POSIX message queues     (bytes, -q)

real-time priority              (-r)

stack size              (kbytes, -s)

cpu time               (seconds, -t) unlimited

max user processes              (-u)

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

open files那一行就代表系统目前允许单个进程打开的最大句柄数，这里是4096。

按照这个参数想了一下，即使class文件没有打包，项目中也不会消耗这么多文件句柄的。

有打开内存参数确认了一下

vim /etc/security/limits.conf

#<domain>      <type>  <item>         <value>

#

#*               soft    core

#root            hard    core

#*               hard    rss

#@student        hard    nproc

#@faculty        soft    nproc

#@faculty        hard    nproc

#ftp             hard    nproc

#ftp             -       chroot          /ftp

#@student        -       maxlogins       

# End of file

*soft   nofile

*hard   nofile

这一切看上去都没有文件，文件句柄看上去足够tomcat使用。

查看tomcat进程

jps

或者

ps -ef | grep -i 'bootstrap.jar' | grep -v grep | awk '{print $2}'

查得tomcat进程

root@# ps -ef | grep -i 'bootstrap.jar' | grep -v grep | awk '{print $2}'

笔者调大文件句柄最多打开数

然后用命令查看进程打开句柄数

lsof -p | wc -l

root@# lsof -p 29511| wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs
Output information may be incomplete.
265

root@# lsof -p 29511| wc -l
lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs
Output information may be incomplete.
268

通过以下命令查看进程打开的文件句柄详情：

root@# lsof -p

lsof: WARNING: can't stat() fuse.gvfsd-fuse file system /run/user/108/gvfs

      Output information may be incomplete.

COMMAND   PID USER   FD      TYPE             DEVICE  SIZE/OFF      NODE NAME

java     root  cwd       DIR               ,       /opt/tomcat-9.0./logs

java     root  rtd       DIR                ,                /

java     root  txt       REG                ,         /opt/jdk1..0_181/bin/java

java     root  mem       REG                ,        /var/cache/fontconfig/945677eb7aeaf62f1d50efc3fb3ec7d8-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/2cd17615ca594fa2959ae173292e504c-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/04aabc0a78ac019cf9454389977116d2-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/385c0604a188198f04d133e54aba7fe7-le64.cache-

java     root  mem       REG                ,       /opt/jdk1..0_181/jre/lib/amd64/libjpeg.so

java     root  mem       REG                ,       /opt/jdk1..0_181/jre/lib/amd64/libt2k.so

java     root  mem       REG                ,       /opt/jdk1..0_181/jre/lib/amd64/libfontmanager.so

java     root  mem       REG                ,         /var/cache/fontconfig/0d8c3b2ac0904cb8a57a757ad11a4a08-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/1ac9eb803944fde146138c791f5cc56a-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/dc05db6664285cc2f12bf69c139ae4c3-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/767a8244fc0220cfb567a839d0392e0b-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/4794a0821666d79190d59a36cb4f44b5-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/8801497958630a81b71ace7c5f9b32a8-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/bab58bb527bb656aaa9f116d68a48d89-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/3047814df9a2f067bd2d96a2b9c36e5a-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/56cf4f4769d0f4abc89a4895d7bd3ae1-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/b9d506c9ac06c20b433354fa67a72993-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/b47c4e1ecd0709278f4910c18777a504-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/d52a8644073d54c13679302ca1180695-le64.cache-

java     root  mem       REG                ,         /var/cache/fontconfig/551ecf3b0e8b0bca0f25c0944f561853-le64.cache-

java     root  mem       REG                ,        /var/cache/fontconfig/d589a48862398ed80a3d6066f4f56f4c-le64.cache-

.....

java     root  133r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  134r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  135r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  136r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  137r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  138r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  139r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  140r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  141r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  142r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  143r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  144r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  145r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  146r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  147r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  148w      REG               ,        /opt/tomcat-9.0./logs/logs_tomcat.log

发现一直还在长，于是查看进程打开文件，多次对比查看打开的文件句柄，发现tomcat-users.xml只增不减

java     root  171r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  172r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  173r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  174r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  175r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  176r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  177r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  178r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  179r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  180r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  181r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  182r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  183r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  184r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  185r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  186r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  187r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  188r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  189r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  190r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  191r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  192r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  193r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  194r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  195r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  196r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  197r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  198r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  199r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  200r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  201r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  202r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  203r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

java     root  204r      REG               ,        /opt/tomcat-9.0./conf/tomcat-users.xml

至此，问题单症结基本确定了，就是tomcat这里

通过Google查询：too many tomcat-users.xml open by tomcat

最终确定这是tomcat9.0.13的一个bug：Tomcat too many files open (tomcat-users.xml)

This was caused by a bug in Tomcat 9.0.13 which has been fixed in Tomcat 9.0.14.

升级tomcat到9.0.14解决问题

笔者按：以笔者目前粗浅的win32知识触类旁通来分析这个问题，进程打开的文件句柄数达到设定的最大值，在向操作系统申请资源的时候一直处于等待状态，操作系统不再分拨资源

给他，于是进程还在，没死掉，只是资源被他耗尽，待客户端来访问的时候，他还会去申请文件句柄，一直就处于假死状态了。

参考：

Tomcat报java.io.IOException: 打开的文件过多

too many open files(打开的文件过多)解决方法

Error in tomcat “too many open files”

How to really fix the too many open files problem for Tomcat in Ubuntu

相关文章