Jboss EAP 6.3线程使用log4j ConsoleAppender获得死锁

After upgrading from Jboss 5 to Jboss EAP 6.3.0 we found an application to fail after some time running. Taking a look it started to grow the number of threads accepting HTTP requests but being unable to server them. After that a ThreadDump showed us a deadlock with log4j:

从Jboss 5升级到Jboss EAP 6.3.0后,我们发现一个应用程序在运行一段时间后失败。看一下,它开始增加接受HTTP请求但无法为它们提供服务的线程数。之后,ThreadDump向我们展示了log4j的死锁:

Found one Java-level deadlock:
=============================
"http-/0.0.0.0:39002-109":
  waiting to lock monitor 0x0000000013d2a218 (object 0x00000000c1cc7aa8, a org.apache.log4j.ConsoleAppender),
  which is held by "http-/0.0.0.0:39002-12"
"http-/0.0.0.0:39002-12":
  waiting to lock monitor 0x000000001456c128 (object 0x00000000c1cca400, a java.io.PrintStream),
  which is held by "http-/0.0.0.0:39002-74"
"http-/0.0.0.0:39002-74":
  waiting to lock monitor 0x0000000013d2a218 (object 0x00000000c1cc7aa8, a org.apache.log4j.ConsoleAppender),
  which is held by "http-/0.0.0.0:39002-12"

Speaking with development team all references to log4j were removed inside the application leaving only the configuration on standalone.xml ir order to log some classes to some specific files, this is the relevant configuration regarding logging:

与开发团队一起在应用程序内部删除了对log4j的所有引用,只留下standalone.xml上的配置,以便将某些类记录到某些特定文件,这是有关日志记录的相关配置:

    <periodic-rotating-file-handler name="FILE" autoflush="true">
        <formatter>
            <named-formatter name="PATTERN"/>
        </formatter>
        <file relative-to="jboss.server.log.dir" path="server.log"/>
        <suffix value=".yyyy-MM-dd"/>
        <append value="true"/>
    </periodic-rotating-file-handler>
    <size-rotating-file-handler name="DocumentalServices">
        <file relative-to="jboss.server.log.dir" path="DocumentalServices.log"/>
        <rotate-size value="10m"/>
        <max-backup-index value="5"/>
    </size-rotating-file-handler>
    <size-rotating-file-handler name="DataServices">
        <file relative-to="jboss.server.log.dir" path="DataServices.log"/>
        <rotate-size value="10m"/>
        <max-backup-index value="5"/>
    </size-rotating-file-handler>
    <size-rotating-file-handler name="Audit">
        <file relative-to="jboss.server.log.dir" path="audit.log"/>
        <rotate-size value="10m"/>
        <max-backup-index value="5"/>
    </size-rotating-file-handler>
    <logger category="com.documentation.framework.ws" use-parent-handlers="false">
        <level name="DEBUG"/>
        <handlers>
            <handler name="DocumentalServices"/>
        </handlers>
    </logger>
    <logger category="com.documentation.framework.data" use-parent-handlers="false">
        <level name="DEBUG"/>
        <handlers>
            <handler name="DataServices"/>
        </handlers>
    </logger>
    <logger category="com.documentation.framework.core.audit" use-parent-handlers="false">
        <level name="DEBUG"/>
        <handlers>
            <handler name="Audit"/>
        </handlers>
    </logger>
    <root-logger>
        <level name="INFO"/>
        <handlers>
            <handler name="FILE"/>
        </handlers>
    </root-logger>

Seems a bit strange the deadlock on ConsoleAppender, taking in mind we are not logging to console inside root-logger, just to FILE (server.log).

看来ConsoleAppender上的死锁有点奇怪,记住我们没有登录到root-logger中的控制台,只是登录到FILE(server.log)。

1 个解决方案

#1

Something has to be adding the log4j ConsoleAppender. It could be Spring as I know they do some logging trickery. During start-up you could try passing -Dorg.jboss.as.logging.per-deployment=false which should disable JBoss EAP's attempts to configure log4j if a log4j.xml or log4j.properties file is found.

有些东西必须添加log4j ConsoleAppender。它可能是Spring,因为我知道他们做了一些记录技巧。在启动期间,您可以尝试传递-Dorg.jboss.as.logging.per-deployment = false,如果找到log4j.xml或log4j.properties文件,则应禁用JBoss EAP尝试配置log4j。

Another option might be to set the add-logging-api-dependencies to false in the logging subsystem. You'd need to include a log4j library in your deployment as it appears something is depending on it. Here's what the CLI would look like to change the default.

另一个选项可能是在日志记录子系统中将add-logging-api-dependencies设置为false。您需要在部署中包含一个log4j库,因为它似乎取决于它。这是CLI更改默认值的样子。

/subsystem=logging:write-attribute(name=add-logging-api-dependencies,value=false)

Note thought disabling this affects all deployments.

注意禁用此功能会影响所有部署。

That said the issue is the way the log4j ConsoleAppender works. It always logs to System.xxx directly. JBoss EAP wraps stdout and stderr in a logger which is what leads to the deadlock.

这就是问题是log4j ConsoleAppender的工作方式。它始终直接登录到System.xxx。 JBoss EAP将stdout和stderr包装在一个导致死锁的记录器中。

There's a fix that should end up in a 6.4 release at some point.

有一个修复程序应该在某个时候以6.4版本结束。