none
Fast File Write with overlapped IO RRS feed

  • Question

  • Hi i was looking the last few days a way to write binary files in a fast way (10 MBps and higher).when i tried using FileStream it slowed me down and my application failed.Even when i tried to use async file writing it didn't seem to improve.
    I am looking for a mechanism in c# which is aquivalent to c++ overlapped IO (usin ght ewritefile method).
    Does anyone knows where can i find it?

    c#/c++ developer
    Wednesday, January 7, 2009 10:48 PM

Answers

  • More than you ever wanted to know about file io in  .NET.

    http://research.microsoft.com/apps/pubs/default.aspx?id=64538

    and the code

    Sequential File Programming in .Net

    Quoting from this paper:

    To summarize our findings:

    (1) For single disks, use the defaults of the .NET framework – they deliver excellent performance for sequential file
    access.
    (2) Pre-allocate large sequential files (using the
    SetLength() method) when the file is created. This typically
    improves speed by about 13% when compared to a fragmented file.
    (3) At least for now, disk arrays require un-buffered IO to achieve the highest performance. Buffered IO can be 8x
    slower than un-buffered IO. We expect this problem will be addressed in later releases of the .NET framework.
    (4) If you do your own buffering, use large request sizes (64KB is a good place to start).
    (5) Using the .NET framework, a single processor can read and write a disk array at over 800 MBps using unbuffered
    IO.

    This is a December, 2004 paper.  I would imagine that performance has only improved since then.  Improved processors and controllers should have narrowed the difference in speed between buffered and unbuffered io for large disk arrays.  

    • Marked as answer by alomay Saturday, January 10, 2009 4:06 PM
    Saturday, January 10, 2009 5:58 AM
  • There may be a miscommunication regarding terminology: overlapped I/O allows you to write any chunk size. Are you possibly thinking of unbuffered I/O?

    Unbuffered I/O is using the FILE_FLAG_NO_BUFFERING flag, and requires offsets and chunks as multiples of the sector size (usually 512 but may be different, especially as hard drives get very large); it also requires the memory buffer to be sector aligned, so you have to use less-common APIs for allocating the buffer.

    Unbuffered I/O is usually used along with overlapped I/O (FILE_FLAG_OVERLAPPED), but it can be used without overlapped I/O too.

    Overlapped I/O is supported by the .NET libraries using the IAsyncResult interface (which allows saving of state, BTW). Unbuffered I/O is not supported.

    Note that getting a truly asynchronous file stream is not directly straightforward: you have to either use a FileStream constructor that takes a FileOptions parameter, passing FileOptions.Asynchronous; or use the FileStream constructor that takes the "bool useAsync" parameter. This will actually open the file for asynchronous access. If you don't do this, then the FileStream will just fake the asynchronous access using ThreadPool, but the underlying file I/O will be synchronous. File.OpenWrite will never use overlapped I/O.

    Also, I believe you can pend multiple asynchronous writes by setting Position before each one. There is a "hidden interaction" between Seek (or Position) and BeginWrite, with BeginWrite allocating an OVERLAPPED structure (when opened in true asynchronous mode) and then saving the Position in the offset of the OVERLAPPED structure. Just make sure when you get a large network packet that you're not actually waiting for any of the writes to complete.

    There is also a note in the documentation for FileStream.BeginWrite stating that write buffers smaller than 64KB are done synchronously instead. It's not clear whether this would affect true asynchronous streams. Even so, I'd recommend writing as much as possible from each network read, instead of splitting them up into 512 (or any other size) chunks.

    If all else fails, post your socket reading + stream writing code, and everyone will argue discuss the best way to do it. :)

           -Steve
    • Marked as answer by alomay Saturday, January 10, 2009 4:06 PM
    Saturday, January 10, 2009 3:26 AM

All replies

  • You should be able to achieve 50 MBs writes to the non-system drive easily with a current premium motherboard using Windows.  The syntax of the language calling the Windows functions shouldn't matter.  Write file x to drive y.  The OS does the work.  You must be getting  in the OS's way.
    Wednesday, January 7, 2009 11:24 PM
  • I used File stream and it worked extremely slow , even when i used async writing. Then i wrapped the writeFile method and used overlapped IO and it worked extremly fast . Filestream made my app. block ,while the wrapper works perfectly.
    I was wondering if there is a standard API to do this .

    c#/c++ developer
    Thursday, January 8, 2009 5:35 PM
  • There has to be something wrong with your code.
    Thursday, January 8, 2009 6:00 PM
  •  Use FastCopy under unsafe. The gain in speed will be up to x10.

    One of the examples.
    AlexB
    Thursday, January 8, 2009 7:58 PM
  • FastCopy can copy faster than the OS?  It'll  turn my 50 MBs HDD into a 500 MBs HDD?  Any method I have tried to completely copy a file, not in chunks, resulted in approximately the same transfer time.  Copying from the system drive to the system drive with apps running with high page faults will bring everything to a crawl.
    Thursday, January 8, 2009 8:11 PM
  •  Except for the WriteFile or WriteFileEx with associated functions FastCopy will be the fastest.
    AlexB
    Thursday, January 8, 2009 8:37 PM
  • Are you sure you don't mean FastCopy for copying arrays?  Unsafe code used to be much faster for array operations.  Once a complete file transfer operation is started and left to the OS, the transfer time should be independant of the means of starting the transfer.  It might take a millisecond longer to start the operation from managed code compared to native code, but that's insignificant.
    Thursday, January 8, 2009 8:51 PM
  •  John, Arrays or not, as long as you've got a pointer you can either use FastCopy or WriteFile or WriteFileEx to copy blocks of HDD content no matter what the underlying code is. Don't make this distinction. It is NOT C# at all. It is not .NET.

    When I recommended FastCopy I did not see that the OP mentioned WriteFile. I believe they may be comparable but who knows. Only a person with vested interest who has tried both can tell for sure. Perhaps FastCopy will be faster.


    AlexB
    Thursday, January 8, 2009 9:04 PM
  •  When I recommended FastCopy I did not see that the OP mentioned WriteFile. I believe they may be comparable but who knows. Only a person with vested interest who tried both can tell for sure. erhaps FasCopy will prevail. It is a Fast Copy!
    AlexB
    Thursday, January 8, 2009 9:11 PM
  • My whole point on this thread is copying files, not portions of files.  The OP needs to post code to demonstrate his problem.  He's looking for overlapped IO in C#, but he says he used it (async IO) with unsatisfactory results.  He's used writefile, why use unsafe code.
    Thursday, January 8, 2009 9:21 PM
  •  You use WriteFile under unsafe only. This is my understanding. It is a WIn32 API. You need a pointer to the file to use WriteFile: IntPtr.
    AlexB
    Thursday, January 8, 2009 9:27 PM
  • You P/Invoke WriteFile.  No need to compile with the unsafe switch.
    Thursday, January 8, 2009 9:39 PM
  •  You are correct but I did it under unsafe this is why it stuck in my mind. I am sorry.
    AlexB
    Thursday, January 8, 2009 11:06 PM
  • FileStream.BeginWrite is the C# equivalent to overlapped I/O.

    I think the op's problem lies elsewhere.

            -Steve
    Friday, January 9, 2009 2:36 AM
  • Hi All
    To make things clear: i am not copying files from one place to another . i am creating files. i write binary data which i receive over tcp/ip to the HD. when i use FileStream, it makes me block on the network ( it slows me down) and i work in a 100mbps network which means approx 12 mBytes per second. It's not so fast in terms of file write.
    regarding the use of writefile: i wrote a wrapper in c++/CLI to wrap the use of writefile.i did it because :
    1) i need to save state : when you call writefile in overlapped io mode you need to tell it the offset in the file to write to . so between two file write i need to keep track of the offset.
    2) overlapped IO works if you write in chunks that are multiplies of 512 bytes.i write data that come in various sizes. so i need to split it and keep the residue to the next write.

    this wrapper works extremely fast and i can write in a 30-40 Mbytes per second easily.

    when i tried FileStream with async flag set to true, and using beginwrite it didn't seem to work any faster.

    There is not problem in the app. because as i said , when i use the wrapper the app. works perfectly with no problems.when i use FileStream i doesn't keep up.

    Thanks a lot,
    Alon

    c#/c++ developer
    Friday, January 9, 2009 2:48 PM
  •  There is not problem in the app

    Then why did you post? Just kidding.

    So, it is your network that is the bottleneck. Stephen Cleary was correct in his intuition.

    AlexB
    Friday, January 9, 2009 5:13 PM
  • Most File IO is cached by the OS.  To slow down the IO you must be opening the file, writing a small chunk, flushing the buffers and closing the file.  Most coders find that IO is hard to time (much too fast) because of the caching.  If you want IO as rapid as can be done, don't close the stream.  It won't matter whether you use managed or unmanaged code.  Processing the code is much faster than the IO.

    Friday, January 9, 2009 6:56 PM
  • Alex , please note that i have clearly noted that the file writing made my notwork comm slow down and not the oposite (the network is not the problem !!!)
    To john, i use an opened stream with pretty big chunks and with no flushing.it's pretty simple i open a stream and keep it open. i close it only when the communication ends.

    c#/c++ developer
    Friday, January 9, 2009 7:55 PM
  • I guess the rest of us will never know how you achieved your result.  For most of us, it's hard to get percieved IO as slow as File.WriteAllBytes, which gives a fairly accurate measure of the disc speed.  For example, I wrote a 100 MB array of random bytes with varying buffer sizes

                                Time          Speed
    File.WriteAllBytes     1637 ms    61 MBs

    Write(Buffer,Offset,Bytes)

                Bytes        Time         Speed  
                256            983 ms    101 MBs
                512            530 ms    188 MBs
              1024            329 ms     303 MBs
              2048            231 ms     432 MBs
              4096            173 ms     578 MBs
              8192            116 ms     862 MBs
            16384              84 ms    1190 MBs
            32768              62 ms    1612 MBs
            65536              54 ms    1851 MBs
          131072              53 ms    1886 MBs
          262144              49 ms    2040 MBs
          524288           1597 ms       62 MBs
         1048576          2434 ms       41 MBs

    Friday, January 9, 2009 8:20 PM
  •  file writing made my notwork comm slow down and not the oposite

    This is kind of interesting. Could you support it? What is your explanation? Is it some sort of synchronization issue? How can you prove that it is the case? What is you network speed? The rates you showed appear to be pretty low. Is it 10/100 network?

    AlexB
    Friday, January 9, 2009 8:52 PM
  • There may be a miscommunication regarding terminology: overlapped I/O allows you to write any chunk size. Are you possibly thinking of unbuffered I/O?

    Unbuffered I/O is using the FILE_FLAG_NO_BUFFERING flag, and requires offsets and chunks as multiples of the sector size (usually 512 but may be different, especially as hard drives get very large); it also requires the memory buffer to be sector aligned, so you have to use less-common APIs for allocating the buffer.

    Unbuffered I/O is usually used along with overlapped I/O (FILE_FLAG_OVERLAPPED), but it can be used without overlapped I/O too.

    Overlapped I/O is supported by the .NET libraries using the IAsyncResult interface (which allows saving of state, BTW). Unbuffered I/O is not supported.

    Note that getting a truly asynchronous file stream is not directly straightforward: you have to either use a FileStream constructor that takes a FileOptions parameter, passing FileOptions.Asynchronous; or use the FileStream constructor that takes the "bool useAsync" parameter. This will actually open the file for asynchronous access. If you don't do this, then the FileStream will just fake the asynchronous access using ThreadPool, but the underlying file I/O will be synchronous. File.OpenWrite will never use overlapped I/O.

    Also, I believe you can pend multiple asynchronous writes by setting Position before each one. There is a "hidden interaction" between Seek (or Position) and BeginWrite, with BeginWrite allocating an OVERLAPPED structure (when opened in true asynchronous mode) and then saving the Position in the offset of the OVERLAPPED structure. Just make sure when you get a large network packet that you're not actually waiting for any of the writes to complete.

    There is also a note in the documentation for FileStream.BeginWrite stating that write buffers smaller than 64KB are done synchronously instead. It's not clear whether this would affect true asynchronous streams. Even so, I'd recommend writing as much as possible from each network read, instead of splitting them up into 512 (or any other size) chunks.

    If all else fails, post your socket reading + stream writing code, and everyone will argue discuss the best way to do it. :)

           -Steve
    • Marked as answer by alomay Saturday, January 10, 2009 4:06 PM
    Saturday, January 10, 2009 3:26 AM
  • More than you ever wanted to know about file io in  .NET.

    http://research.microsoft.com/apps/pubs/default.aspx?id=64538

    and the code

    Sequential File Programming in .Net

    Quoting from this paper:

    To summarize our findings:

    (1) For single disks, use the defaults of the .NET framework – they deliver excellent performance for sequential file
    access.
    (2) Pre-allocate large sequential files (using the
    SetLength() method) when the file is created. This typically
    improves speed by about 13% when compared to a fragmented file.
    (3) At least for now, disk arrays require un-buffered IO to achieve the highest performance. Buffered IO can be 8x
    slower than un-buffered IO. We expect this problem will be addressed in later releases of the .NET framework.
    (4) If you do your own buffering, use large request sizes (64KB is a good place to start).
    (5) Using the .NET framework, a single processor can read and write a disk array at over 800 MBps using unbuffered
    IO.

    This is a December, 2004 paper.  I would imagine that performance has only improved since then.  Improved processors and controllers should have narrowed the difference in speed between buffered and unbuffered io for large disk arrays.  

    • Marked as answer by alomay Saturday, January 10, 2009 4:06 PM
    Saturday, January 10, 2009 5:58 AM
  •  John, I looked at the numbers you posted on the previous page of this thread as well as some diagrams in the pdf document on Sequential file programming pattern andm it gave me some thoughts. I just formatted a 10,000 Rpm 300GB VelociRaptor for my server. I chose 4096 block size. Looking at your numbers it seems I would be better off with a block of larger size, perhaps at least double of that.

    My reasoning is this. Vista as well as WinSer2008 perform constant defragmentation if you leave them overnight. Also file compression of amazing efficiency is constanly taking place. So, there is a good chance all my files will be contiguous and the big unknown will be the average file size. If it is smaller than the block size then some space will be wasted but this is least of my worries. If the average size is small will the whole blocks have to  be fetched anyway or not? I don't think it is true. Once the EOF is reached the read process or write process should terminate. Thus it sounds that I should go for a larger block size. Shall I quadruple the size of my HDD block perhaps?

    AlexB
    Sunday, January 11, 2009 12:10 AM
  • AlexBB:

    I haven't seen anything about how the disk block size affects the speed.  The numbers I posted show the effect of caching on the percieved speed of synchronous io. 
    Sunday, January 11, 2009 6:32 AM
  • This paper is excellent. I used it a few years ago ('07 or '08) when I tried to improve I/O performance in some .NET code. The thing I found most useful was the IOSpeed.exe sample, which proved to me that it was possible to reach the disk's speed limit using pure .NET code.
    Sunday, January 8, 2012 2:28 PM