我应该压缩内存中的C#对象以获得更好的性能吗?

时间:2022-12-27 21:22:03

I have an application (C#, WPF) that displays many financial charts with live data streaming from server. The data that is collected in-memory may grow to be a bit large, and I don't want to keep any data on disk.

我有一个应用程序(C#,WPF)显示许多金融图表,其中包含来自服务器的实时数据流。在内存中收集的数据可能会变得有点大,我不想在磁盘上保留任何数据。

Since the historical data itself doesn't change, but only added to, will it make sense to keep that data (which is stored in a collection object) in some compressed format?

由于历史数据本身不会改变,但只是添加到,将数据(存储在集合对象中)保存为某种压缩格式是否有意义?

Is it possible and can anyone recommend a good practice for it if so?

是否有可能,任何人都可以推荐一个好的做法,如果是这样的话?

UPDATE

UPDATE

Some notes about performance and tradeoff: I am aware that compression will add a delay accessing the data, but, the user only needs fast updates on new data arriving. When accessing the data that was already rendered (for example, to study or re-render it) he doesn't require a quick response.

关于性能和权衡的一些注意事项:我知道压缩会增加访问数据的延迟,但是,用户只需要对到达的新数据进行快速更新。当访问已经渲染的数据(例如,研究或重新渲染它)时,他不需要快速响应。

5 个解决方案

#1


13  

Compressing and decompressing will make your application slower so for performance (speed) it is not a good option. Compression is only useful when you are worried about available memory. It might be easier to store/swap the data to a temp folder.

压缩和解压缩会使您的应用程序变慢,因此对于性能(速度)而言,这不是一个好的选择。压缩仅在您担心可用内存时才有用。将数据存储/交换到临时文件夹可能更容易。

The key to performance is measuring. Only take action when you have crunched the numbers.

绩效的关键是衡量。只有在你碾压数字时才采取行动。

#2


6  

Compressing the data has advantages in terms of memory usage, but disadvantages in terms of making the data unusable (you'll have to decompress it to use it again), as well as taking up extra CPU.

压缩数据在内存使用方面具有优势,但在使数据无法使用方面存在缺点(您必须将其解压缩才能再次使用),以及占用额外的CPU。

The tradeoff point where this would become beneficial is difficult to know without a lot more information - it's up to you. However, if you're not using this old, stale data, it may be better to just throw it away (ie: let it go out of scope/stop storing it) instead of compressing it.

如果没有更多的信息,很难知道这会变得有利的权衡点 - 这取决于你。但是,如果您没有使用这个旧的陈旧数据,最好将它扔掉(即:让它超出范围/停止存储它)而不是压缩它。

Compression can be done via the classes in System.IO.Compression, and is fairly easy. These classes, in general, don't perform very well, however, so you may want to also consider a third party alternative, such as DotNetZip.

压缩可以通过System.IO.Compression中的类完成,并且非常简单。但是,这些类通常表现不佳,因此您可能还想考虑第三方替代方案,例如DotNetZip。

#3


1  

It's a trade off between performance and memory footprint and also depends on the data structures you are using. "Generic" compression (ie. gzip, run-length encoding etc.) doesn't make sense for many types of data.

它是性能和内存占用之间的折衷,也取决于您使用的数据结构。 “通用”压缩(即gzip,游程编码等)对于许多类型的数据没有意义。

One approach that might be applicable to you is picking a more appropriate data structure that optimizes memory footprint - i.e. for your chart do you really have to store independent stock prices or can you live by just storing delta values? If the later is true you probably could reduce the bits needed for each data point. Another thing is re-occuring patterns that are needed in all charts - could you factor those out in a separate object used by all charts hence only instantiated once?

一种可能适用于您的方法是选择一种更合适的数据结构来优化内存占用 - 例如,对于您的图表,您是否真的需要存储独立的股票价格,还是只能存储delta值?如果后者为真,则可能会减少每个数据点所需的位数。另一件事是重新发生所有图表中需要的模式 - 你可以将它们分解为所有图表使用的单独对象,因此只实例化一次吗?

#4


0  

If you are looking for better performance, compression is not the way to go. As long as your client host has enough memory to handle the data then keeping the data uncompressed will result in maximum performance. Compressing the data will require uncompression and compression algorithms to run whenever the data is accessed.

如果您正在寻求更好的性能,压缩是不可取的。只要您的客户端主机有足够的内存来处理数据,那么保持数据不压缩将导致最大的性能。压缩数据将需要解压缩和压缩算法,以便在访问数据时运行。

If you are running out of memory on the client host, then you will be in a situation where you are forced to compress stored data. Note, however, that this will only save memory when the data is compressed and Garbage Collection has collected the in memory objects that are not compressed. Because the data will need to be uncompressed to be utilized, this will never provide a solution for maxing out client RAM.

如果客户端主机上的内存不足,那么您将面临*压缩存储数据的情况。但请注意,这只会在压缩数据时节省内存,并且垃圾收集已收集未压缩的内存对象。由于数据需要解压缩才能使用,因此永远不会提供最大化客户端RAM的解决方案。

With all this kept in mind, .NET provides the System.IO.Compression namespace for performing gzip compression. If you need compression I would start by looking there.

考虑到所有这些,.NET提供了用于执行gzip压缩的System.IO.Compression命名空间。如果你需要压缩,我会从那里开始。

#5


0  

If you're willing to code it yourself, space-efficient data structures exist that don't require decoding/decompression to be used. Steve Hanov describes Succinct Data Structures in his latest blog post. His example is a succinct trie but there's nothing stopping you from representing other objects and structures. He cites several alternative implementations.

如果您愿意自己编写代码,则存在节省空间的数据结构,不需要使用解码/解压缩。 Steve Hanov在他最新的博客文章中描述了简洁数据结构。他的例子是一个简洁的特里,但没有什么能阻止你代表其他物体和结构。他列举了几种替代实施方案。

Obviously, this isn't an out-of-the-box solution. You'll have to decide if it's worth the effort to build and test a succinct representation.

显然,这不是一个开箱即用的解决方案。你必须决定是否值得努力建立和测试简洁的表示。

#1


13  

Compressing and decompressing will make your application slower so for performance (speed) it is not a good option. Compression is only useful when you are worried about available memory. It might be easier to store/swap the data to a temp folder.

压缩和解压缩会使您的应用程序变慢,因此对于性能(速度)而言,这不是一个好的选择。压缩仅在您担心可用内存时才有用。将数据存储/交换到临时文件夹可能更容易。

The key to performance is measuring. Only take action when you have crunched the numbers.

绩效的关键是衡量。只有在你碾压数字时才采取行动。

#2


6  

Compressing the data has advantages in terms of memory usage, but disadvantages in terms of making the data unusable (you'll have to decompress it to use it again), as well as taking up extra CPU.

压缩数据在内存使用方面具有优势,但在使数据无法使用方面存在缺点(您必须将其解压缩才能再次使用),以及占用额外的CPU。

The tradeoff point where this would become beneficial is difficult to know without a lot more information - it's up to you. However, if you're not using this old, stale data, it may be better to just throw it away (ie: let it go out of scope/stop storing it) instead of compressing it.

如果没有更多的信息,很难知道这会变得有利的权衡点 - 这取决于你。但是,如果您没有使用这个旧的陈旧数据,最好将它扔掉(即:让它超出范围/停止存储它)而不是压缩它。

Compression can be done via the classes in System.IO.Compression, and is fairly easy. These classes, in general, don't perform very well, however, so you may want to also consider a third party alternative, such as DotNetZip.

压缩可以通过System.IO.Compression中的类完成,并且非常简单。但是,这些类通常表现不佳,因此您可能还想考虑第三方替代方案,例如DotNetZip。

#3


1  

It's a trade off between performance and memory footprint and also depends on the data structures you are using. "Generic" compression (ie. gzip, run-length encoding etc.) doesn't make sense for many types of data.

它是性能和内存占用之间的折衷,也取决于您使用的数据结构。 “通用”压缩(即gzip,游程编码等)对于许多类型的数据没有意义。

One approach that might be applicable to you is picking a more appropriate data structure that optimizes memory footprint - i.e. for your chart do you really have to store independent stock prices or can you live by just storing delta values? If the later is true you probably could reduce the bits needed for each data point. Another thing is re-occuring patterns that are needed in all charts - could you factor those out in a separate object used by all charts hence only instantiated once?

一种可能适用于您的方法是选择一种更合适的数据结构来优化内存占用 - 例如,对于您的图表,您是否真的需要存储独立的股票价格,还是只能存储delta值?如果后者为真,则可能会减少每个数据点所需的位数。另一件事是重新发生所有图表中需要的模式 - 你可以将它们分解为所有图表使用的单独对象,因此只实例化一次吗?

#4


0  

If you are looking for better performance, compression is not the way to go. As long as your client host has enough memory to handle the data then keeping the data uncompressed will result in maximum performance. Compressing the data will require uncompression and compression algorithms to run whenever the data is accessed.

如果您正在寻求更好的性能,压缩是不可取的。只要您的客户端主机有足够的内存来处理数据,那么保持数据不压缩将导致最大的性能。压缩数据将需要解压缩和压缩算法,以便在访问数据时运行。

If you are running out of memory on the client host, then you will be in a situation where you are forced to compress stored data. Note, however, that this will only save memory when the data is compressed and Garbage Collection has collected the in memory objects that are not compressed. Because the data will need to be uncompressed to be utilized, this will never provide a solution for maxing out client RAM.

如果客户端主机上的内存不足,那么您将面临*压缩存储数据的情况。但请注意,这只会在压缩数据时节省内存,并且垃圾收集已收集未压缩的内存对象。由于数据需要解压缩才能使用,因此永远不会提供最大化客户端RAM的解决方案。

With all this kept in mind, .NET provides the System.IO.Compression namespace for performing gzip compression. If you need compression I would start by looking there.

考虑到所有这些,.NET提供了用于执行gzip压缩的System.IO.Compression命名空间。如果你需要压缩,我会从那里开始。

#5


0  

If you're willing to code it yourself, space-efficient data structures exist that don't require decoding/decompression to be used. Steve Hanov describes Succinct Data Structures in his latest blog post. His example is a succinct trie but there's nothing stopping you from representing other objects and structures. He cites several alternative implementations.

如果您愿意自己编写代码,则存在节省空间的数据结构,不需要使用解码/解压缩。 Steve Hanov在他最新的博客文章中描述了简洁数据结构。他的例子是一个简洁的特里,但没有什么能阻止你代表其他物体和结构。他列举了几种替代实施方案。

Obviously, this isn't an out-of-the-box solution. You'll have to decide if it's worth the effort to build and test a succinct representation.

显然,这不是一个开箱即用的解决方案。你必须决定是否值得努力建立和测试简洁的表示。