ultra fast binary writes

Question

Hello, can anyone please help...

This is a bit of a general question. My C knowledge is limited to the K&R book.

I have used the fsutil tool on windows xp and noticed that the command:
fsutil file createnew c:\$delthis.bin  1234567890 which creates a file over a gigabyte in size is extremely fast - it takes about a second on a PIII 800MHz with 256MB memory. This seems to be amazing given that it zeros the whole file, rather than simply creating a directory entry.

If I wanted to code a 10 line program in C to do exactly the same (for simplicity with hard coded values -  no parameter options) and run just as quickly, how would it be coded? Or is this a 10 page exercise?

Would bog standard C be able to accomplish this or would I need to access low level system calls?

For info on fsutil, see:-http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/fsutil.mspx

Thanks in advance for any help given.

Hal.

Answer 1

If you want to create code that will write a file as fast as possible you want to use the underlying Win32 APIs.

Take a look at CreateFile:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/createfile.asp

and WriteFile:

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/writefile.asp

Note: the speed of writing a large file will depend a lot on the configuration and fragmentation of your disk.

Answer 2

From spelunking around in fsutil.exe, my guess is that the program is calling DeviceIOControl() with the dwIoControlCode parameter set to FSCTL_SET_ZERO_DATA.  This will be very fast since you are talking directly to a device driver but, form looking at the API, it looks like it can only write zeros to the file, not arbitrary data.

-Ron Pihlgren
 VC++ Testing

Answer 3

When you use "CreateFile" function try to use flags "FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN".

I knew from the past it was a little faster, but depends on OS you're using.

Bye
Martin

Answer 4

Hi Hal!
> What I would really like is a 10 line program that will work as fast as 
> fsutil as described in the original post.

A little bit more than 10 lines, but it creates you sprase-files (up to 
many TB).


#define _WIN32_WINNT 0x0500
#include 
#include 
#include 
#include 

int _tmain()
{
   HANDLE h = CreateFile(_T("C:\\Sparse-file.txt"),
     GENERIC_WRITE, 0, NULL,
     CREATE_ALWAYS, 0, NULL);
   if (h == INVALID_HANDLE_VALUE) {
     printf("\nFailed to create file!");
     return -1;
   }
   DWORD dw;
   BOOL bRet = DeviceIoControl(h, FSCTL_SET_SPARSE, NULL,
     0, NULL, 0, &dw, NULL);
   if (bRet == FALSE) {
     printf("\nFailed to set file as sparse-file (%8.8x)", GetLastError());
     return -1;
   }
   LONG lDist = 0x0800;   // 8 TB!!!
   SetFilePointer(h, 0, &lDist, FILE_BEGIN);
   SetEndOfFile(h);
   CloseHandle(h);
   return 0;
}


-- 
Greetings
   Jochen

    My blog about Win32 and .NET
    http://blog.kalmbachnet.de/

Answer 5

If you just want to create a big empty file quickly SetEndOfFile and SetFileValidData might be what you need - no need to talk to the driver directly.

Answer 6

What I would really like is a 10 line program that will work as fast as fsutil as described in the original post.

I have already said my C skills are limited when it comes to Windows.

I'm a bit surprised that people tell me this is simple to code and yet I have seen no working code!

Writing the fastest code to do this is something to be proud of isn't it?  Is this a programming forum or what?!

Come on guys help me out here! A 10-liner please!

Hal

Answer 7

Just typed it in here. Never compiled. No error handling. Usage would be something like "makebigfile empty.file 1000000". Assuming you compile it as makebigfile.exe that is. This might behave better if you set things like the sparse file and no buffering flags.

What are you trying to do with this?

#include <windows.h>

void main(int argc, char** argv)
{
   HANDLE hFile = CreateFile(argv[0], GENERIC_WRITE, 0, 0, CREATE_NEW, 0, 0);
   SetFilePointer(hFile, atoi(argv[1]), 0, FILE_BEGIN);
   SetEndOfFile(hFile);
   CloseHandle(hFile);
}

Answer 8

I'm trying to create a large zeroed file just like fsutil does!

Have you tried doing what I did with fsutil? Are you not impressed with the speed?

I dont think fsutil creates a sparse file.

Some code - thanks! Will it work! Does it compile! Is it fast!

Answer 9

It's certainly interesting code!

However, when I said I didn't think fsutil created sparse files, I meant that if it doesnt, this is a good thing since the whole file is actually cleared. Sorry if I wasn't clear.

The reason I believe fsutil does not create sparse files is that if you right click on a created file's properties you can see the size on disk is about the same as the file size. Whereas using a utility like mksparse ( http://www.insidewindows.info/mksparse.zip ) you see the size on disk when you right click is about 4k for a 1GB sparse file. Incidently the source code for this utility is in the zip file.
 
FSutil could still be using some neo-sparse format I guess with the file being zeroed only when accessed. This would explain the speed.

A sparse file creator should be fast since it is doing so much less work - it isn't zeroing the file up front, the zeroing work is done later on as the file is accessed and expanded.

Any chance of a non-sparse version and some times? Are we talking minutes or seconds for non-sparse on an NTFS file system?

A DeviceIOControl version (with timings!) would be interesting too.

I should have mentioned that the fsutil timing given originally - around a second to create and zero a 1.15GB file - was for an NTFS partition. Exactly the same command on a FAT32 partition of the smae disk took around 3 minutes - which is still pretty fast but not quite so impressive!

Glad to see some more code... hope there's more coming!

Hal

Answer 10

Hi Hal!

> However, when I said I didn't think fsutil created sparse files, I meant 
> that if it doesnt, this is a good thing since the whole file is actually 
> cleared. Sorry if I wasn't clear.

The big plus on sparse-files is that the file is also cleared!
You can read the whole 4 TB file and you will always retrive 0 as data.

Only that part of the file which was explicite writte contains the 
values which was written. All other parts will return 0!

> A sparse file creator should be fast since it is doing so much less work 
> - it isn't zeroing the file up front, 

The file is "virtually" zeroed. As I said: If you read the file you will 
return zero for all none-written areas.

 > Glad to see some more code... hope there's more coming!

theumpteenthbrian posted the correct code which was used by the fsutil
(after debugging the fsutil-program... ;-) =

void main()
{
   HANDLE hFile = CreateFile(_T("c:\test.bin"),
     GENERIC_WRITE, 0, 0, CREATE_NEW, 0, 0);
   LARGE_INTEGER li;
   li.QuadPart = 10000; // very big number...
   SetFilePointerEx(hFile, atoi(argv[1]), li, NULL, FILE_BEGIN);
   SetEndOfFile(hFile);
   CloseHandle(hFile);
}

-- 
Greetings
   Jochen

    My blog about Win32 and .NET
    http://blog.kalmbachnet.de/

Answer 11

Hi Hal!

> Yes, it was good of Brian but as he indicated, it wasn't tested, timed 
> or even compiled. Also when I read the description on msdn about those 
> functions, I did not see it stated that using either SetFilePointerEx or 
> SetEndOfFile or both would set the contents to zero so at that time 
> there was no reason for me to believe it would work!

Yes. It seems that this is an undocumented feature... (at least for NTFS).

So in summary I can say: This is the wrong solution.... if you want to 
have a zeroed-file you either should use "sparse-files" or you must zero 
it by hand...

The documentation of "SetFilePointer(Ex)" and "SetEndOfFile" explicite 
states:
"If the file is extended, the contents of the file between the old end 
of the file and the new end of the file are not defined."
(http://msdn.microsoft.com/library/en-us/fileio/fs/setendoffile.asp)

and

"Note that it is not an error to set the file pointer to a position 
beyond the end of the file. The size of the file does not increase until 
you call the SetEndOfFile, WriteFile, or WriteFileEx function. A write 
operation increases the size of the file to the file pointer position 
plus the size of the buffer written, leaving the intervening bytes 
uninitialized."
(http://msdn.microsoft.com/library/en-us/fileio/fs/setfilepointerex.asp)

sorry for this bad news...


But as I said: "fsutil" only uses CreateFile, SetFilePointerEx, 
SetEndOfFile, CloseFile (at least for NTFS). SO I think it must be a 
undocumented feature that this operations will set the content to zero.
(http://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/fsutil_file.mspx)

-- 
Greetings
   Jochen

    My blog about Win32 and .NET
    http://blog.kalmbachnet.de/

Answer 12

Hi Jochen,

Thanks for your encouragement!

> The file is "virtually" zeroed. As I said: If you read the file you will
> return zero for all none-written areas.

Yes I know. As I indicated in my previous post, I am familiar with sparse files!

However, I wanted the file (the disk) physically zereod. I was thinking of using it to wipe empty parts of a disk but I didnt want 20 replies telling me better ways to do it because I was interested in how it was coded.

> theumpteenthbrian posted the correct code which was used by the fsutil
> (after debugging the fsutil-program... ;-) =

Yes, it was good of Brian but as he indicated, it wasn't tested, timed or even compiled. Also when I read the description on msdn about those functions, I did not see it stated that using either SetFilePointerEx or SetEndOfFile or both would set the contents to zero so at that time there was no reason for me to believe it would work!

Perhaps its an undocumented feature or I just missed it?
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/fs/setfilepointerex.asp

Anyway as you have endorsed his code I have given it a whirl and came up with this slight variant which is 40 lines!....

#include <windows.h>
#include <stdio.h>

void exitm(char *s)
{
   fprintf(stderr,"%s problem (error %d)\n", s, GetLastError());
   exit (9);
}

int main(int argc, char** argv)
{
  HANDLE hFile;
  LARGE_INTEGER li;

  if (argc!=3)
   {
     fprintf(stderr,"format is: '%s filename size'\n",  argv[0]);
     exit (9);
   }

   hFile = CreateFile(TEXT(argv[1]), GENERIC_WRITE, 0, 0, CREATE_NEW, 0, 0);

   if(hFile == INVALID_HANDLE_VALUE)
     exitm("CreateFile");
    
   li.LowPart = atol(argv[2]);  /* don't need much more than 1234567890 for now */
   li.HighPart = 0;

   if ( ! SetFilePointerEx(hFile, li , NULL, FILE_BEGIN) )
   exitm("SetFilePointerEx");

   if ( ! SetEndOfFile(hFile) )
   exitm("SetEndOfFile");

   if ( ! CloseHandle(hFile) )
   exitm("CloseHandle");

 return 0;
}

This has been tested and it works! Cool, thank you both for your help.!

The timing on my system for an NTFS drive is subsecond and for a FAT32 drive is under 2 minutes.

This has been fun!

Any pointers to missing or undocumented features of those 2 functions would still be welcome!

Hal.

Answer 13

Microsoft and undocumented code... Deja vu?

I thought some more timings might be of interest.

(zeroxxx04a is the program I posted above.
 binwrite02 is a simple program that writes non-zero data to a file.)
------------------------------------------------------------------------------------------

f: is NTFS
-----------
TimeThis :  Command Line :  fsutil file createnew f:\$$a 1234567890
TimeThis :  Elapsed Time :  00:00:00.260  FAST!!!!!!!!!!

TimeThis :  Command Line :  zeroxxx04a f:\$$b 1234567890
TimeThis :  Elapsed Time :  00:00:00.080  FAST!!!!!!!!!!

This was run on a non-zero file....

TimeThis :  Command Line :  fsutil file setzerodata offset=0 length=1234567890 f:\$$x
TimeThis :  Elapsed Time :  00:02:20.512  not so fast.

Running the same command again is faster since it checks to see if the data is zero
and so does not have to write the zeros again....

TimeThis :  Command Line :  fsutil file setzerodata offset=0 length=1234567890 f:\$$x
Zero data is changed
TimeThis :  Elapsed Time :  00:00:00.130  FAST!!!!!!!!!!

This was used to create the non-zero file above....

TimeThis :  Command Line :  binwritet02.exe f:\$$x 1234567890
TimeThis :  Elapsed Time :  00:02:21.803  not so fast.

------------------------------------------------------------------------------------------

e: is FAT32
-----------
TimeThis :  Command Line :  fsutil file createnew e:\$$p 1234567890
TimeThis :  Elapsed Time :  00:02:06.101  not so fast.

TimeThis :  Command Line :  zeroxxx04a e:\$$q 1234567890
TimeThis :  Elapsed Time :  00:01:49.617  not so fast.

This won't work as it isn't NTFS...

TimeThis :  Command Line :  fsutil file setzerodata offset=0 length=1234567890 e:\$$r
The FSUTIL utility requires a local NTFS volume.
TimeThis :  Elapsed Time :  00:00:00.120

Just for completeness...
TimeThis :  Command Line :  binwritet02.exe e:\$$s 1234567890
TimeThis :  Elapsed Time :  00:02:19.390  not so fast.

------------------------------------------------------------------------------------------
Hal

Answer 14

Can I use this feature on my ASP page?

Thanks,