ASMB的BUG(ORA-04030 kfmditer)导致数据库宕机

时间:2024-01-06 18:23:44

ASMB的BUG(ORA-04030 kfmditer)导致数据库宕机
现象:

客户的一个重要生产系统RAC的一个实例宕机,查看alert日志:

Fri Jun 21 17:05:52 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):

ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc

Fri Jun 21 17:05:55 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):

ORA-04030: out of process memory when trying to allocate bytes (,)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc

Fri Jun 21 17:06:14 2013

Instance terminated by ASMB, pid = 11391

查看asmb trace文件:

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc (incident=31397):

ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc

Fri Jun 21 17:05:52 2013

Trace dumping is performing id=[cdmp_20130621170552]

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc:

ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

ASMB (ospid: 11391): terminating the instance due to error 4030

System state dump is made for local instance

System State dumped to trace file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_diag_11345.trc

Fri Jun 21 17:05:53 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms0_11363.trc (incident=31301):

ORA-04030: out of process memory when trying to allocate bytes (,)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31301/jyj1_lms0_11363_i31301.trc

Fri Jun 21 17:05:53 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lmon_11359.trc (incident=31277):

ORA-04030: out of process memory when trying to allocate bytes (,)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31277/jyj1_lmon_11359_i31277.trc

Fri Jun 21 17:05:53 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_lms1_11367.trc (incident=31309):

ORA-04030: out of process memory when trying to allocate bytes (,)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31309/jyj1_lms1_11367_i31309.trc

Fri Jun 21 17:05:54 2013

ORA-1092 : opitsk aborting process

Fri Jun 21 17:05:54 2013

License high water mark = 327

Fri Jun 21 17:05:55 2013

Errors in file /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_rbal_11389.trc (incident=31389):

ORA-04030: out of process memory when trying to allocate bytes (,)

Incident details in: /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31389/jyj1_rbal_11389_i31389.trc

Fri Jun 21 17:06:14 2013

Instance terminated by ASMB, pid

jyj1_asmb_11391_i31397.trc:

Dump file /opt/app/diag/rdbms/jyj/jyj1/incident/incdir_31397/jyj1_asmb_11391_i31397.trc

Oracle Database 11g Enterprise Edition Release 11.1.0.6.0 - 64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

and Real Application Testing options

ORACLE_HOME = /opt/app/ora11gR1db

System name: Linux

Node name: KSJYJ_DB01

Release: 2.6.18-164.el5

Version: #1 SMP Thu Sep 3 04:15:13 EDT 2009

Machine: x86_64

Instance name: jyj1

Redo thread mounted by this instance: 1

Oracle process number: 24

Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)

*** 2013-06-21 17:05:52.045

*** SESSION ID:(532.1) 2013-06-21 17:05:52.046

*** CLIENT ID:() 2013-06-21 17:05:52.046

*** SERVICE NAME:(SYS$BACKGROUND) 2013-06-21 17:05:52.046

*** MODULE NAME:() 2013-06-21 17:05:52.046

*** ACTION NAME:() 2013-06-21 17:05:52.046

 

Dump continued from file: /opt/app/diag/rdbms/jyj/jyj1/trace/jyj1_asmb_11391.trc

ORA-04030: out of process memory when trying to allocate 592 bytes (callheap,kfmditer)

========= Dump for incident 31397 (ORA 4030) ========

*** 2013-06-21 17:05:52.046

----- SQL Statement (None) -----

Current SQL information unavailable - no cursor.

skdstdst <- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp

 <- ksfdmp <- dbgexPhaseII <- dbgexProcessError <- dbgeExecuteForError <- dbgePostErrorKGE

 <- 1774 <- dbkePostKGE_kgsf <- kgesev <- kgesec3 <- kghnospc

 <- kghalf <- kfmdIterInit <- kfkIterInit <- kfnbIostatiterOp <- 110

 <- kfnbRun <- ksbrdp <- opirip <- opidrv <- sou2o

Process state

-----------------------

SO: 0x940dd1b98, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3

 proc=0x940dd1b98, name=process, file=ksu.h LINE:10286, pg=0

 (process) Oracle pid:24, ser:1, calls cur/top: 0x920f28eb8/0x920f28eb8

 flags: (0x6) SYSTEM

 int error: 0, call error: 0, sess error: 0, txn error 0

 (post info) last post received: 0 0 34

 last post received-location: ksr2.h LINE:594 ID:ksrpublish

 last process to post me: 950dfd540 47 2

 last post sent: 0 0 64

 last post sent-location: kso2.h LINE:316 ID:ksoreq_reply

 last process posted by me: 930e5c948 1 0

 (latch info) wait_event=0 bits=0

 Process Group: DEFAULT, pseudo proc: 0x950e4c060

 O/S info: user: oracle, term: UNKNOWN, ospid: 11391

 OSD pid info: Unix process pid: 11391, image: oracle@KSJYJ_DB01 (ASMB)

Dump of memory from 0x00000009D0DC0A10 to 0x00000009D0DC0C18

分析:

从报错信息(ORA-04030)看来,怀疑是Oracle的BUG导致的,因为以前碰到过类似的ASMB进程内存泄露的BUG,

于是搜索metalink关键词:asmb 04030

发现第一篇就跟客户的问题吻合。

ASMB process grows raising ora-4030 intermittently (Doc ID 735180.1)

ASMB process grows on memory, eventually leading to ora-4030 errors

which causes DB crash.

The reported error:

ORA-04030: out of process memory when trying to allocate 552 Bytes (callheap,kfmditer)

 

In the ASMB process heapdump we can see most of memory chunks are for 'kfmditer',

example:

BreakDown

 ~~~~~~~~~

 Type     Count   Sum        Average

 ~~~~     ~~~~~   ~~~        ~~~~~~~

 Free     285684  142841492  500.00

 kfmditer 285685  157698132  552.00   <-- 在ASMB的HEAPDUMP中也看到了绝大多数都为kfmditer的内存片

Total = 300539624 bytes 293495.73k 286.62MB

 

 这个BUG在11.1以后的大版本中都有出现,但是在以下的patchset中被修复:

 

 This issue is fixed in

11.2.0.1 (Base Release)

11.1.0.7.1 (Patch Set Update)

10.2.0.5 (Server Patch Set)

11.1.0.7 Patch 11 on Windows Platforms

11.1.0.7 RAC Recommended Patch Bundle #1

11.1.0.6 Patch 11 on Windows Platforms

如果不想做patchset升级的话,也可以直接打个小Patch 6851110可以解决这个问题。

You can check if Patch 6851110 is available for your patchset release and

O/S environment.:  Patch 6851110

解决方法:

在客户的数据库上打patch  6851110,经过持续观察一段时间,该问题未再现。