I need to write a large amount of data in my application. Data arrive periodically and pushed into the queue.
我需要在我的应用程序中编写大量数据。数据定期到达并推入队列。
//producer
queue_.push( new_data );// new_data is a 64kb memory range
PostThreadMessage(worker_thread_id_, MyMsg, 0, 0);//notify the worker thread
after that, i need to write this data to the file. I have another thread, that get data from the queue and writes it.
之后,我需要将此数据写入文件。我有另一个线程,从队列中获取数据并写入它。
//consumer
while(GetMessage(msg, 0, 0))
{
data = queue_.pop();
WriteFile(file_handle_, data.begin(), data.size(), &io_bytes, NULL);
}
This is simple, but not efficient. Another oportunity is to use IO completion ports.
这很简单,但效率不高。另一个机会是使用IO完成端口。
//producer
//file must be associated with io completion port
WriteFile( file_handle_
, new_data.begin()
, new_data.size()
, &io_bytes
, new my_overlaped(new_data) );
consumer is just delete unused buffers
consumer只是删除未使用的缓冲区
my_completion_key* ck = 0;
my_overlapped* op = 0;
DWORD res = GetQueuedIOCompletionStatus(completion_port_, &io_bytes, &ck, &op);
//process errors
delete op->data;
What is the preferred way, to perform large amount of file i/o operation on windows?
在Windows上执行大量文件i / o操作的首选方法是什么?
1 个解决方案
#1
Hard to tell because it is not clear what kind of problem you are experiencing with your current approach. Going async in a dedicated "writer" thread shouldn't result in considerable performance improvement.
很难说,因为目前尚不清楚您使用当前的方法遇到了什么样的问题。在专用的“编写器”线程中执行异步不应导致相当大的性能提升。
If you have small but frequent writes (so that frequent WriteFile's cost becomes a bottleneck) I would suggest implementing a kind of lazy writes - collect data and flush it to file when some volume threshold is reached.
如果你有小但频繁的写入(因此频繁的WriteFile成本成为瓶颈)我建议实现一种惰性写入 - 收集数据并在达到某个音量阈值时将其刷新到文件。
#1
Hard to tell because it is not clear what kind of problem you are experiencing with your current approach. Going async in a dedicated "writer" thread shouldn't result in considerable performance improvement.
很难说,因为目前尚不清楚您使用当前的方法遇到了什么样的问题。在专用的“编写器”线程中执行异步不应导致相当大的性能提升。
If you have small but frequent writes (so that frequent WriteFile's cost becomes a bottleneck) I would suggest implementing a kind of lazy writes - collect data and flush it to file when some volume threshold is reached.
如果你有小但频繁的写入(因此频繁的WriteFile成本成为瓶颈)我建议实现一种惰性写入 - 收集数据并在达到某个音量阈值时将其刷新到文件。