none
fopen and large files, failing on "r+", why?

    Question

  • Hi All,

    I'm very new to Windows programming, and am developing an application under XP (Win32) that needs to read, modifywrite, and append to a file. For portability I'm using standard C functionality (fopen) to open the file. When I attempt to open a file that is larger than 4 GB for reading and modifying ("r+") I get an error--invalid parameter. I can open the file for reading ("r") without any errors.

    Any idea as to what's causing this, and how to fix it?

    My own thoughts are that it's probably not related to the OS being 32-bit--the fact that I can open the file for reading without any errors seems to shoot that idea down. Plus, I'm assuming XP *can* read and write to large files (multimedia requirements come to mind). Is there something I forgot to do/set/call before opening the file, should I be using different I/O functions (open? CreateFile?), or....?

    Please forgive what may be a mundane question, but after several hours of searching I wasn't able to come up with anything helpful.

    Thanks very much!

    -Josh
    Thursday, July 12, 2007 1:34 AM

Answers

All replies

  • The problem is caused by an internal check in fopen, which attempts to seek one byte prior of the end of the file, and read the last byte. To do this seek, fopen uses _lseek_nolock, which in turn calls SetFilePointer with a lDistanceToMove = -1, lpDistanceToMoveHigh = NULL and dwMoveMethod = FILE_END. That appears not to work with 64bit file pointers, so I'm inclined to call it a bug in the standard library. Line 112 of lseek.c should perhaps read

     

    Code Snippet
    if
    ((newpos = SetFilePointer(osHandle, pos, &pos, mthd)) == -1)

     

    rather than

     

    Code Snippet

    if ((newpos = SetFilePointer(osHandle, pos, NULL, mthd)) == -1)

     

     

    Thursday, July 12, 2007 8:24 AM
    Moderator
  • It's also worth mentioning that quite a few of the other standard C file stream functions also cramp out when you go beyond 4GB, so you'd probably be a whole lot better off using the storage and file APIs from the Platform / Windows SDK. Try CreateFile / WriteFile / ReadFile / SetFilePointer.
    Thursday, July 12, 2007 9:50 AM
    Moderator
  •  einaros wrote:
    It's also worth mentioning that quite a few of the other standard C file stream functions also cramp out when you go beyond 4GB, so you'd probably be a whole lot better off using the storage and file APIs from the Platform / Windows SDK. Try CreateFile / WriteFile / ReadFile / SetFilePointer.


    I rewrote everything using CreateFile etc, and it appears to be working--thanks very much! I/O performance is a little strange, though, so I'll have to do some tweaking.

    -Josh
    Thursday, July 12, 2007 11:04 PM