debug实战:COM组件GetToSTA导致高内存+GC被阻塞

时间:2023-03-09 04:27:54
debug实战:COM组件GetToSTA导致高内存+GC被阻塞

最近花了好几周解决一个WPF高内存的问题,问题的表象是内存不断增加、未被回收,根源是GC的FinalizeThread被阻塞,导致整个GC挂掉。从以下几步来分析这个问题:

1.用ANTS Memory Profiler去掉强引用

既然是高内存,肯定要先从内存着手。这里必须要赞一下ANTS的这个工具,图形化做的非常好,一目了然,个人觉得比SciTech的.net memory profiler好用。找个基准点take一个SnapShot,打开关闭窗口后再take一个snapshot,比较2个快照里多出了哪些对象,或者窗口对象被什么强引用了导致未被释放,都很清楚。一般来说是自己代码的问题,但也有第三方组件的坑,比如:

  1. DevExpress.Data.DelayedExecutionExtension里有static的Dictionary,会持有很多控件的强引用,需要在窗口Close时调用RemoveDelayedExecute()
  2. TypeDescriptor相关字样的一堆类(包括DPCustomTypeDescriptor、DependencyPropertyDescriptor、DependencyObjectProvider等),都通过DependencyObject._effectiveValues持有对窗口等控件的强引用,需要在窗口Close时调用TypeDescriptor.Refresh(object)

    注:不查不知道,这个TypeDescriptor还大有来头,不仅管理着所有.net object的metadata,还可以动态修改这些metadata,这对于封装一些代理类、提供Transparent功能的场景应该很有用。详情见这篇博客TypeDescriptor

2.当去掉所有强引用后,大量对象堆积在FinalizeQueue上

在ANTS里看到,所有希望回收的对象,都堆积在FinalizeQueue上,即使内存飙到1G也无法回收。手动调GC强制回收,阻塞在WaitForPendingFinalizers()上一直无法返回。

GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect();

这时就只能抓dump用windbg分析到底是哪里卡住了。

3.抓dump分析线程堆栈

用ProcDump -ma [ProcessName]抓dump,分析如下:

/*0:000> .loadby sos clr
0:000> !threads
ThreadCount: 63
UnstartedThread: 0
BackgroundThread: 34
PendingThread: 0
DeadThread: 28
Hosted Runtime: no
Lock
ID OSID ThreadOBJ State GC Mode GC Alloc Context Domain Count Apt Exception
0 1 1b80 00000000008c9cf0 26020 Preemptive 0000000000000000:0000000000000000 000000000087ce90 0 STA
2 2 ba8 00000000008d4790 2b220 Preemptive 0000000000000000:0000000000000000 000000000087ce90 0 MTA (Finalizer)
......
40 63 1748 0000000021ecc3d0 1029220 Preemptive 0000000000000000:0000000000000000 000000000087ce90 0 MTA (Threadpool Worker)*/ 0:000> !syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
-----------------------------
Total 173
CCW 5
RCW 5
ComClassFactory 0
Free 25 //并没有线程被锁住,看看2号Finalizer线程在干嘛
/*0:000> ~2kb
RetAddr : Args to Child : Call Site
000007fe`fcf210dc : 00000000`00000000 00000000`1b66f308 00000000`1b66ee60 00000000`1b66edd0 : ntdll!NtWaitForSingleObject+0xa
000007fe`fd2de68e : 00000000`1bf51bf0 00000000`00911fb0 00000000`00000000 00000000`0000044c : KERNELBASE!WaitForSingleObjectEx+0x79
000007fe`fd413700 : 00000000`008fd0b0 00000000`1bf51bf0 00000000`00000246 00000000`008fd0b0 : ole32!GetToSTA+0x8a
000007fe`fd41265b : 00000000`00000000 00000000`ffffffff 00000000`61a6d4cc 00000000`ffffffff : ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x13b
......
000007fe`e53b383c : 00000000`1bfd2380 00000000`1b66f9a8 00000000`00000000 000007fe`e5417ad3 : clr!CtxEntry::EnterContext+0x232
000007fe`e53b37e6 : 00000000`1b66f9a8 000007fe`e524307c 00000000`008d4790 00000000`1b66f9f0 : clr!RCW::EnterContext+0x3d
000007fe`e544319f : 000007fe`e5b055b0 00000000`008d4790 00000000`008d4790 00000000`00000000 : clr!SyncBlockCache::CleanupSyncBlocks+0xc2
000007fe`e536ab47 : 00000000`00000001 00000000`00000001 00000000`008d4790 00000000`00000000 : clr!Thread::DoExtraWorkForFinalizer+0xdc
000007fe`e52b458c : 0030002e`00340076 00000000`1b66fcc0 00000000`00390031 00000000`00001000 : clr!WKS::GCHeap::FinalizerThreadWorker+0x109
000007fe`e52b451a : 00000000`1b66fcc0 00000000`00000000 0000cd2d`f0a20ef6 000007fe`e53ff57a : clr!Frame::Pop+0x50
...
000007fe`e5391d90 : 00000000`00000000 00000000`00000000 00000000`00000001 00000000`0000001e : clr!ManagedThreadBase_NoADTransition+0x3f
000007fe`e53133de : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!WKS::GCHeap::FinalizerThreadStart+0xb4
00000000`76fa59ed : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : clr!Thread::intermediateThreadProc+0x7d
00000000`770dc541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d*/ //强烈建议关注一下GetToSTA,这个方法是COM组件引起GC阻塞的典型特征。原因是STA的COM组件必须在创建它的线程上被回收,所以FinalizerThread想GetToSTA线程去执行回收的代码。但它想GetTo的是哪个线程?那个线程又因为什么阻塞了呢?
000007fe`fd413700 : 00000000`008fd0b0 00000000`1bf51bf0 00000000`00000246 00000000`008fd0b0 : ole32!GetToSTA+0x8a //首先用|查看进程ID
0:000> |
. 0 id: 3b8c examine name: E:\...\Process.exe
//然后在这个方法的参数列表上,依次执行dd并查找进程ID:3b8c,进程ID旁边就是它要去的线程ID,我是在第二个参数里找到的。
0:000> dd 1bf51bf0
00000000`1bf51bf0 fd4283e0 000007fe fd450628 000007fe
00000000`1bf51c00 00000000 00000000 00000001 0000102a
00000000`1bf51c10 00000000 00000000 0000044c 00000000
00000000`1bf51c20 00000000 00000000 00003400 1b803b8c
//最后一行的1b80 3b8c,后半段是进程ID、前半段是它要去STA线程。回上面一看,原来就是0号主线程。
0 1 1b80 00000000008c9cf0 26020 Preemptive 0000000000000000:0000000000000000 000000000087ce90 0 STA //再看看0号主线程在干嘛?原来停在ConnectNamedPipe里,对应的.net代码是NamedPipeStreamServer.WaitForConnection()。这是个非托管的阻塞方法,只要等不到Connection,就会一直阻塞。
/*0:000> kb
RetAddr : Args to Child : Call Site
000007fe`fcf33c2f : 000007fe`e5311ed4 00000000`0027e428 00000000`0027e480 00000000`05d5cac8 : ntdll!NtFsControlFile+0xa
*** WARNING: Unable to verify checksum for System.Core.ni.dll
000007fe`e1ab8017 : 000007fe`e169adb8 00000000`00000000 00000000`00000000 00000000`05d5c7d8 : KERNELBASE!ConnectNamedPipe+0x6f
......
000007fe`e53a2a7e : 00000000`00000000 00000000`00000004 00000000`00000000 00000000`00000004 : clr!MethodDescCallSite::CallTargetWorker+0x2e2
000007fe`e53a31d6 : 00000000`00000004 00000000`00000000 00000000`00000000 00000000`03162eb8 : clr!RunMain+0x1e7
000007fe`e53a30d0 : 00000000`008f13b0 00000000`00000200 00000000`008f13b0 00000000`00000200 : clr!Assembly::ExecuteMainMethod+0xb6
000007fe`e53a2c46 : 00000000`0027f8c8 00000000`00bf0000 00000000`00000000 00000000`00000000 : clr!SystemDomain::ExecuteMainMethod+0x45e
000007fe`e53a2b9e : 00000000`00bf0000 00000000`0027fa20 00000000`00000000 000007fe`f6f441c0 : clr!ExecuteEXE+0x3f
000007fe`e53a3574 : ffffffff`ffffffff 00000000`00000000 00000000`00000000 00000000`00000000 : clr!CorExeMainInternal+0xae
000007fe`f6ee77ad : 00000000`00000000 000007fe`00000091 00000000`00000000 00000000`0027f988 : clr!CorExeMain+0x14
000007fe`f6fa5b21 : 00000000`00000000 000007fe`e53a3560 00000000`00000000 00000000`00000000 : mscoreei!CorExeMain+0xe0
00000000`76fa59ed : 000007fe`f6ee0000 00000000`00000000 00000000`00000000 00000000`00000000 : mscoree!CorExeMain_Exported+0x57
00000000`770dc541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!BaseThreadInitThunk+0xd
00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!RtlUserThreadStart+0x1d*/

怪不得FinalizerThread会挂掉,因为它要释放COM组件,所以要进到创建COM组件的STA线程,而这个STA线程又在无限期的等待Connection,所以就挂掉了。GC都挂了,永远不执行垃圾回收,当然会高内存。另:GetToSTA的手法太妖了,这么hack的方法我当然想不出来,请参见gcHang0gcHang1gcHang2gcHang3

4.解决方案

  1. 把创建COM的线程改为MTA,因为主线程必须是STA的,所以只能新建一个MTA的线程来干这个事儿了。
  2. 如果非得在STA的线程里干这事儿,那就不能使用非托管的阻塞方法,比如WaitForConnection,而要使用托管的阻塞方法,比如WaitHandle.WaitOne, WaitAny, WaitAll, Monitor.Enter, Monitor.Block, Thread.Join, GC.WaitForPendingFinalizers这些都是,这些方法在阻塞线程的同时,还能正确的pump messages。这些托管的阻塞方法,配合对应的Begin/End异步方法,就能响应各种消息了。

5.演示的Demo

最后用一个小Demo把问题重现了一遍,也把解决方案附在里面了,有兴趣的同学可以试一下。