none
WriteFile() slow to return when writing at random large file offset RRS feed

  • Question

  • Hello everyone!
    I've recently encountered an issue with the Win32 API WriteFile() that I have yet to understand or resolve. Here is a sample of the code I have written that simulates my problem (I have remove the error handling for clarity):

    int _tmain(int argc, _TCHAR* argv[]) {
    HANDLE hTestFile1 = NULL;
    LARGE_INTEGER Offset;
    char buffer[65536];
    DWORD bufferLen = 65536, bytesWritten = 0;
    DWORD start = 0, end = 0;

    wstring testFile1 = L"c:\\temp\\test\\file.ext";

    hTestFile1 = CreateFile(testFile1.c_str(), GENERIC_WRITE, 0, 0, OPEN_ALWAYS, FILE_FLAG_WRITE_THROUGH, 0);

    Offset.QuadPart =
    12000000000;
    SetEndOfFile(hTestFile1);     // Allocating the file size beforehand didn't impact the writting speed

    Offset.QuadPart = 9352843264;

    SetFilePointerEx(hTestFile1, Offset, NULL, FILE_BEGIN);

    WriteFile(hTestFile1, buffer, bufferLen, &bytesWritten, NULL);

    if (hTestFile1 != NULL)
        CloseHandle(hTestFile1);
    }

    In this scenario, the file was recently created, empty or was just starting to get filled when a write operation at a large offset (like offset 9 GB) is applied. When this happens, WriteFile() hangs for a few minutes before successfully completing the operation.
    Opening the file for random access had no impact on the performances of the operation.

    I would like to understand the phenomenon, why it is taking so much time to return, but more importantly what can be done to have the API return in a more rapidly (i.e. in the following milliseconds the call was made) so that other write operations be immediately applied.

    Thanks in advance for your help.
    JF

    Thursday, October 15, 2009 3:37 PM

Answers

  • You can speed this up marking the file as a sparse file:

      hTestFile1 = CreateFile(testFile1.c_str(), GENERIC_WRITE, 0, 0, OPEN_ALWAYS, FILE_FLAG_WRITE_THROUGH, 0);
      DeviceIoControl(hTestFile1, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &bytesWritten, NULL);

    //etc...

    Hans Passant.
    Thursday, October 15, 2009 5:15 PM
    Moderator

All replies

  • Hi,

    First: FILE_FLAG_WRITE_THROUGH is the answer to the delayed write.
    Secondary: (hTestFile1 != NULL) test, at the end (very odd) is equal to zero, in opposition to the value it is testing. Somebody thought that it might help and put the test, but it is useless: a filehandle is never 0.

    Third: A test should be performed at every step of the IO operation.

    Spend 15 minits of your time and read the MSDN detailed documentation related to CreateFile.

    If you wish to flush the file from the HDD buffers, use FlushFileBuffers.

    Regards,

    Dev s r'us
    Thursday, October 15, 2009 4:01 PM
  • Hi Codu,

    I just ran a test, modifying the CreateFile options from FILE_FLAG_WRITE_THROUGH to FILE_FLAG_RANDOM_ACCESS | FILE_ATTRIBUTE_NORMAL. It had no impact on the WriteFile() performance. It still took well over 2 minutes to return.

    Thursday, October 15, 2009 4:21 PM
  • You can speed this up marking the file as a sparse file:

      hTestFile1 = CreateFile(testFile1.c_str(), GENERIC_WRITE, 0, 0, OPEN_ALWAYS, FILE_FLAG_WRITE_THROUGH, 0);
      DeviceIoControl(hTestFile1, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &bytesWritten, NULL);

    //etc...

    Hans Passant.
    Thursday, October 15, 2009 5:15 PM
    Moderator
  • Hi nobugz

    I just included this change into my test app in order to repoduce it and it definetely improved the performaces. Now the same operation only take a few milliseconds.


    Was the reason it took so much time to write at large offset on the new file due to the system zeroing the blocks between the last written block and the
    targetted one?

    Is this the only way a Sparse file can be created?

    It looks as though creating the file as a Sparse file will generate disk fragmentation. Is there a way to counter that programatically?

    thanks for your help.
    Thursday, October 15, 2009 6:47 PM
  • Yes, writing 9 gigabytes takes time.  Typical disks transfer at 60MB/sec: 9000/60 = 2.5 minutes

    Afaik, why do you need another?

    Probably.  Yes, don't create sparse files.

    Hans Passant.
    Thursday, October 15, 2009 7:10 PM
    Moderator