.NET垃圾收集器和x64虚拟内存

Running a .NET application on Windows Server 2008 x64 with 16GB of RAM. This application needs to fetch and analyze a very large amount of data (about 64GB), and keep it all in memory at one time.

在具有16GB RAM的Windows Server 2008 x64上运行.NET应用程序。此应用程序需要获取和分析大量数据(大约64GB),并将其全部保存在内存中。

What I expect to see: Process size expands past 16GB to 64GB. Windows uses virtual memory to page the extra data to/from disk as needed. This is the classic virtual memory use case.

我期待看到的:流程大小扩展到16GB到64GB。 Windows根据需要使用虚拟内存将额外数据分页到磁盘或从磁盘分页。这是经典的虚拟内存使用案例。

What I actually see: Process size is limited to the amount of physical memory (16GB). Application spends 99.8% of its time in the garbage collector.

我实际看到的:进程大小限制为物理内存量(16GB)。应用程序将99.8%的时间花在垃圾收集器上。

Why is our application failing to use virtual memory? Is this a problem in the configuration of the .NET garbage collector, or in the Windows x64 virtual memory manager itself? What can I do to get our application to use virtual memory rather than be limited to physical memory?

为什么我们的应用程序无法使用虚拟内存?这是.NET垃圾收集器配置中的问题,还是Windows x64虚拟内存管理器本身的问题?我该怎么做才能让我们的应用程序使用虚拟内存而不仅限于物理内存?

Thanks.

-- Brian

Update: I have written a very small program that exhibits the same behavior:

更新:我写了一个非常小的程序,表现出相同的行为:

using System;

namespace GCTest
{
    class Program
    {
        static void Main()
        {
            byte[][] arrays = new byte[100000000][];
            for (int i = 0; i < arrays.Length; ++i)
            {
                arrays[i] = new byte[320];
                if (i % 100000 == 0)
                {
                    Console.WriteLine("{0} arrays allocated", i);
                    System.Threading.Thread.Sleep(100);
                }
            }
        }
    }
}

If you want to try it, make sure to build for x64. You may have to modify the constants a bit to stress your system. The behavior I see is that the process bogs down as it approaches a size of 16GB. There is no error message or exception thrown. Performance monitor reports that the % of CPU time in GC approaches 100%.

如果您想尝试它,请确保为x64构建。您可能需要稍微修改常量以强调您的系统。我看到的行为是,当进程接近16GB的大小时,进程陷入困境。没有抛出错误消息或异常。性能监视器报告GC中的CPU时间百分比接近100%。

Isn't this unacceptable? Where's the virtual memory system?

这不是不可接受的吗?虚拟内存系统在哪里?

2 个解决方案

#1

Have you checked to make sure that your paging file is configured so that it can expand to that size?

您是否已检查以确保您的页面文件已配置为可以扩展到该大小?

Update

I've been playing around with this quite a bit with your given example, and here's what I see.

我用你的例子一直在玩这个,这就是我所看到的。

System: Windows 7 64bit, 6GB of triple-channel RAM, 8 cores.

系统:Windows 7 64位,6GB三通道RAM,8核。

You need an additional paging file on another spindle from your OS or this sort of investigation will hose your machine. If everything is fighting over the same paging file, it makes things worse.

您需要在操作系统的另一个主轴上添加一个页面文件,否则这种调查会给您的机器带来麻烦。如果所有内容都在同一页面文件中进行争夺,那就会让事情变得更糟
I am seeing a large amount of data being promoted from generation to generation in the GC, plus a large number of GC sweeps\collections, and a massive amount of page faults as a result as physical memory limits are reached. I can only assume that when physical memory is exhausted\very high, that this triggers generation sweeps and promotions thus causing a large amount of paged-out memory to be touched which is leading to a death spriral as touched memory is paged in and other memory is forced out. The whole thing ends in a soggy mess. This seems to be inevitable when allocating a large number of long-lived objects which end up in the Small Object Heap.

我看到GC中代代相传的大量数据,以及大量的GC扫描\集合,以及因达到物理内存限制而导致的大量页面错误。我只能假设当物理内存耗尽\非常高时,这会触发生成扫描和促销,从而导致大量的页面调出内存被触摸,这会导致触发内存被分页和其他内存中的死亡事件*出局。整个事情在一个潮湿的混乱中结束。当分配大量长寿命对象时,这似乎是不可避免的,这些对象最终会出现在小对象堆中。

Now compare this to allocating objects in a fashion will allocate them directly into the Large Object Heap (which does not suffer the same sweeping and promotion issues):

现在比较一下,以一种方式分配对象会将它们直接分配到大对象堆(它没有遭受同样的清扫和促销问题):

private static void Main()
{
    const int MaxNodeCount = 100000000;
    const int LargeObjectSize = (85 * 1000);

    LinkedList<byte[]> list = new LinkedList<byte[]>();

    for (long i = 0; i < MaxNodeCount; ++i)
    {
        list.AddLast(new byte[LargeObjectSize]);

        if (i % 100000 == 0)
        {
            Console.WriteLine("{0:N0} 'approx' extra bytes allocated.",
               ((i + 1) * LargeObjectSize));
        }
    }
}

This works as expected i.e. virtual memory is used and then eventually exhausted - 54GB in my environment\configuration.

这按预期工作,即使用虚拟内存,然后最终耗尽 - 在我的环境\配置中54GB。

So it appears that allocating a mass of long-lived small objects will eventually lead to a vicious cycle in the GC as generation sweeps and promotions are made when physical memory has been exhausted - it's a page-file death spiral.

所以看来,分配大量长寿命小物体最终会导致GC中的恶性循环,因为当物理内存耗尽时会产生一代扫描和促销 - 这是一个页面文件死亡螺旋。

Update 2

Whilst investigating the issue I played with a number of options\configurations which made no appreciable difference:

在调查这个问题时,我使用了许多选项\配置,但没有产生明显的差异:

Forcing Server GC mode.

强制服务器GC模式。

Configuring low latency GC.

配置低延迟GC。

Various combinations of forcing GC to try to amortize GC.

迫使GC尝试分摊GC的各种组合。

Min\Max process working sets.

Min \ Max流程工作集。

#2

It sounds like you're not keeping a reference to the large data. The garbage collector will not collect referenced objects.

听起来你没有保留对大数据的引用。垃圾收集器不会收集引用的对象。

#1