Websphere服务器宕机问题解决

时间:2024-04-16 10:37:16

   有一项目客户使用的是websphere6.0(补丁打到21),当时出现一个莫名其妙的问题,就是跑一会儿就宕机,分析器后台Systemout.txt日志发现有:

 R java.lang.OutOfMemoryError: JVMCI046: allocateMemory failed
 at sun.misc.Unsafe.allocateMemory(Native Method)
 at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java(Inlined Compiled Code))
 at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java(Inlined Compiled Code))
 at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateCommon(WsByteBufferPoolManagerImpl.java(Compiled Code))
 at com.ibm.ws.buffermgmt.impl.WsByteBufferPoolManagerImpl.allocateDirect(WsByteBufferPoolManagerImpl.java(Compiled Code))
 at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.getNewByteBuffer(WCCByteBufferOutputStream.java(Inlined Compiled Code))
 at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.checkWriteArray(WCCByteBufferOutputStream.java(Compiled Code))
 at com.ibm.ws.webcontainer.channel.WCCByteBufferOutputStream.write(WCCByteBufferOutputStream.java(Compiled Code))
 at com.ibm.ws.webcontainer.srt.SRTOutputStream.write(SRTOutputStream.java(Compiled Code))
 at com.ibm.ws.webcontainer.srt.BufferedServletOutputStream.writeOut(BufferedServletOutputStream.java(Inlined Compiled Code))
 at com.ibm.ws.webcontainer.srt.BufferedServletOutputStream.write(BufferedServletOutputStream.java(Compiled Code))
 at com.ibm.ws.webcontainer.servlet.FileServletWrapper.writeResponseToClient(FileServletWrapper.java(Compiled Code))
 at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:588)
 at com.ibm.ws.webcontainer.servlet.FileServletWrapper.service(FileServletWrapper.java:764)
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled Code))
 at com.opensymphony.webwork.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled Code))
 at org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java(Compiled Code))
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled Code))
 at com.opensymphony.webwork.dispatcher.ActionContextCleanUp.doFilter(ActionContextCleanUp.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled Code))
 at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java(Compiled Code))
 at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java(Compiled Code))
 at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java(Compiled Code))
 at com.ibm.ws.webcontainer.servlet.FileServletWrapper.handleRequest(FileServletWrapper.java:363)
 at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:89)
 at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java(Compiled Code))
 at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java(Compiled Code))
 at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java(Compiled Code))
 at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java(Compiled Code))
 at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:288)
 at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminaters(NewConnectionInitialReadCallback.java:207)
 at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:109)
 at com.ibm.ws.tcp.channel.impl.WorkQueueManager.requestComplete(WorkQueueManager.java(Compiled Code))
 at com.ibm.ws.tcp.channel.impl.WorkQueueManager.attemptIO(WorkQueueManager.java(Compiled Code))
 at com.ibm.ws.tcp.channel.impl.WorkQueueManager.workerRun(WorkQueueManager.java(Compiled Code))
 at com.ibm.ws.tcp.channel.impl.WorkQueueManager$Worker.run(WorkQueueManager.java(Compiled Code))
 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
内存分配异常,问题很严重啊,此项目在tomcat、weblogic(8.16)websphere(7.0)下均运行没有问题。在网上查阅资料,发现IBM官方给出了如下一篇文章(Excessive native memory use in IBM WebSphere Application Server 6.0 and 6.1):

Excessive native memory use in IBM WebSphere Application Server 6.0 and 6.1
 Technote (troubleshooting)
 
Problem(Abstract)
The use of asynchronous data transfer may require an excessive number of buffers to send data over a TCP/IP connection.


This predominately occurs when files are being transferred (e.g., a PDF being returned in a response, the dmgr updating the nodeagents, etc.) but, can also occur with normal responses from IBM® WebSphere® Application Server. 
 
Symptom
The process size for the application server or deployment manager can grow quite large.


You may also see OutOfMemoryError’s thrown and malloc errors in SystemOut.log. 
JVMDBG001: malloc failed to allocate … 
 



Cause
A new feature was added to allow the sending of data asynchronously as a performance enhancement. Unfortunately, this can require the allocation of enough buffers to contain the entire file being transferred (java.nio.DirectByteBuffer\'s). 


Depending on the size of the file, this can be a bad decision, as it may require a large amount of memory. There are also occasions when these buffers will not be freed up, causing even more problems with native memory utilization. 
 

Environment
WebSphere Application Server 6.0.2 and 6.1 
 

Diagnosing the problem
Check the JVM’s™ process size using operating system specific tools (e.g., ps, top, perfmon). Check the Application Server logs for malloc errors. 
 

Resolving the problem
A Java™ property can be set which will force the data transfer to be synchronous rather than asynchronous. Since the asynchronous transfer is generally faster, at least for small files, the use of synchronous transfer may result in a slight performance impact. This performance impact should be quite small, and would be far outweighed by the performance impact of the extra memory used by all of the buffers allocated by the asynchronous transfer algorithm.

This property was added in later Application Server FixPacks, so you may need to upgrade your Application Server installation before setting the property. 


1. Check if your Application Server installation is at least at 6.0.2.21 or 6.1.0.11, depending on whether you are on Application Server 6.0.2 or 6.1. If not, please install one of the FixPacks to get your installation up to that level or more recent.

2. Set the channelwritetype property to use synchronous data transfer (sync):

For an application server
In the Admin Console: 
Servers -> Application Servers -> serverName -> Web Container Settings -> Web Container -> Custom Properties:
Press New:
Add the following pair: 
Name: com.ibm.ws.webcontainer.channelwritetype
Value: sync
Press OK, and then save the configuration.

The application server must be recycled to pick up the property.


For a deployment manager
Start an interactive wsadmin session: 
<dmgr-profile-root>\bin>wsadmin -lang jacl

Copy the following block of lines, then paste them all at once at the wsadmin> prompt: 
set dmgr [$AdminConfig getid /Server:dmgr/]
set webcontainer [$AdminConfig list WebContainer $dmgr]
$AdminConfig create Property $webcontainer {{name com.ibm.ws.webcontainer.channelwritetype} {value sync}} properties
$AdminConfig show $webcontainer
$AdminConfig save

The deployment manager must be recycled to pick up the property.
,照此设置即可。