fopen and large files, failing on "r+", why?

Question

Hi All,

I'm very new to Windows programming, and am developing an application under XP (Win32) that needs to read, modifywrite, and append to a file. For portability I'm using standard C functionality (fopen) to open the file. When I attempt to open a file that is larger than 4 GB for reading and modifying ("r+") I get an error--invalid parameter. I can open the file for reading ("r") without any errors.

Any idea as to what's causing this, and how to fix it?

My own thoughts are that it's probably not related to the OS being 32-bit--the fact that I can open the file for reading without any errors seems to shoot that idea down. Plus, I'm assuming XP *can* read and write to large files (multimedia requirements come to mind). Is there something I forgot to do/set/call before opening the file, should I be using different I/O functions (open? CreateFile?), or....?

Please forgive what may be a mundane question, but after several hours of searching I wasn't able to come up with anything helpful.

Thanks very much!

-Josh

Answer 1

The problem is caused by an internal check in fopen, which attempts to seek one byte prior of the end of the file, and read the last byte. To do this seek, fopen uses _lseek_nolock, which in turn calls SetFilePointer with a lDistanceToMove = -1, lpDistanceToMoveHigh = NULL and dwMoveMethod = FILE_END. That appears not to work with 64bit file pointers, so I'm inclined to call it a bug in the standard library. Line 112 of lseek.c should perhaps read

 

Code Snippet

if ((newpos = SetFilePointer(osHandle, pos, &pos, mthd)) == -1)

 

rather than

 

Code Snippet

if ((newpos = SetFilePointer(osHandle, pos, NULL, mthd)) == -1)

 

 

Answer 2

It's also worth mentioning that quite a few of the other standard C file stream functions also cramp out when you go beyond 4GB, so you'd probably be a whole lot better off using the storage and file APIs from the Platform / Windows SDK. Try CreateFile / WriteFile / ReadFile / SetFilePointer.

Answer 3

einaros wrote:

It's also worth mentioning that quite a few of the other standard C file stream functions also cramp out when you go beyond 4GB, so you'd probably be a whole lot better off using the storage and file APIs from the Platform / Windows SDK. Try CreateFile / WriteFile / ReadFile / SetFilePointer.

I rewrote everything using CreateFile etc, and it appears to be working--thanks very much! I/O performance is a little strange, though, so I'll have to do some tweaking.

-Josh