none
ifstream with large files (7 GBytes) fails to open file on x64 architecture

    Question

  • My simple file streamer code fails to open files larger than 32bit (4Gb), using VS2012 Express Desktop.

    Basically opening the large file, the following line won't open the file:

    ifstream myfile ( argv[1], ios::in|ios::binary|ios::ate);

    ...while it works perfectly for smaller files.

    I'm thinking about wrong project settings for my x64 configuratio, possibly ?!

    Any suggestion !?

    Many thanks!

    Tuesday, April 02, 2013 12:14 AM

Answers

  • On 4/2/2013 3:40 AM, PixPush wrote:

    please, someone confirm this isn't a VS bug !?

    Looks very much like it. I stepped through the implementation. It uses fseek (which takes long as offset parameter - which is not an immediate problem here since it's called with 0 and SEEK_END, but it's a bad sign). This eventually calls SetFilePointer passing NULL for the third parameter (so it has no way to pass a 64-bit offset, nor capture the resulting 64-bit position). For a file larger than 4G (and possibly 2G, I haven't tried), this fails with ERROR_INVALID_PARAMETER, just as the documentation says it would ("If lpDistanceToMoveHigh is NULL and the new file position does not fit in a 32-bit value, the function fails").

    On the other hand, if I open without ios::ate and then use myfile.seekg(0, ios::end), this calls _fseeki64 and correctly handles 64-bit offsets.

    So there's a bug in the implementation of ios::ate; using an explicit seekg is a workaround.


    Igor Tandetnik

    • Marked as answer by PixPush Tuesday, April 02, 2013 3:16 PM
    Tuesday, April 02, 2013 12:34 PM

All replies

  • On 4/1/2013 8:14 PM, PixPush wrote:

    My simple file streamer code fails to open files larger than 32bit (4Gb), using VS2012 Express Desktop.

    Basically opening the large file, the following line won't open the file:

    ifstream myfile ( argv[1], ios::in|ios::binary|ios::ate);

    ...while it works perfectly for smaller files.

    Does it work if you drop ios::ate? What's the purpose of ate here, anyway? It's clear why you might want that for an output file, but why for input one?

    I suspect it's the seek-to-the-end implied by ios::ate that fails for some reason. I wonder what would happen if you open without ios::ate, then explicitly call seekg(0, ios::end)


    Igor Tandetnik

    Tuesday, April 02, 2013 12:35 AM
  • Hi Igor,

    thanks for the suggestion.
    In reality the ios::ate is there for that very purpose... it's actually checking if everything's right with the file size.
    Instead of having bad surprises after 4Gbytes, I can verify if there are issues right at the beginning of operations.

    Anyway, I'll give it a try and see if it fails during read operations (which is likely since it cannot seek till end of file).

    Tuesday, April 02, 2013 12:55 AM
  • Oh, and as additional info, the very same source works under Mac - Snow Leopard, without issues.
    That's why I suspect either bad x64 config or a VS bug ( here's an old one on the same topic: http://connect.microsoft.com/VisualStudio/feedback/details/627639 )
    Tuesday, April 02, 2013 12:58 AM
  • Still not working...

    please, someone confirm this isn't a VS bug !?

    Tuesday, April 02, 2013 7:40 AM
  • Still not working...

    please, someone confirm this isn't a VS bug !?

    Well, large file support in fstream should be working
    in VS2012 (VC++11). See:

    http://connect.microsoft.com/VisualStudio/feedback/details/627639/std-fstream-use-32-bit-int-as-pos-type-even-on-x64-platform

    "... large file support should work correctly now (regardless
    of x86/x64 platform)."

    If you can't find the problem, you may want to post a
    report at the Connect site:

    http://connect.microsoft.com/VisualStudio

    - Wayne

    Tuesday, April 02, 2013 8:39 AM
  • On 4/2/2013 3:40 AM, PixPush wrote:

    please, someone confirm this isn't a VS bug !?

    Looks very much like it. I stepped through the implementation. It uses fseek (which takes long as offset parameter - which is not an immediate problem here since it's called with 0 and SEEK_END, but it's a bad sign). This eventually calls SetFilePointer passing NULL for the third parameter (so it has no way to pass a 64-bit offset, nor capture the resulting 64-bit position). For a file larger than 4G (and possibly 2G, I haven't tried), this fails with ERROR_INVALID_PARAMETER, just as the documentation says it would ("If lpDistanceToMoveHigh is NULL and the new file position does not fit in a 32-bit value, the function fails").

    On the other hand, if I open without ios::ate and then use myfile.seekg(0, ios::end), this calls _fseeki64 and correctly handles 64-bit offsets.

    So there's a bug in the implementation of ios::ate; using an explicit seekg is a workaround.


    Igor Tandetnik

    • Marked as answer by PixPush Tuesday, April 02, 2013 3:16 PM
    Tuesday, April 02, 2013 12:34 PM
  • I sent an email to Stephan T. Lavavej, Visual C++ Libraries
    Developer, and asked him to check this thread when he has a
    spare moment or two.

    - Wayne
    Tuesday, April 02, 2013 1:38 PM
  • Thanks everyone!
    I found an answer on StackOverflow, using streambuf tho, could it be it !?
    That is, re-compiling libs for x64 might solve the issue !?

    http://stackoverflow.com/questions/293672/reading-files-larger-than-4gb-using-c-stl

    Thanks!
    Tuesday, April 02, 2013 3:13 PM

  • On the other hand, if I open without ios::ate and then use myfile.seekg(0, ios::end), this calls _fseeki64 and correctly handles 64-bit offsets.

    So there's a bug in the implementation of ios::ate; using an explicit seekg is a workaround.


    Igor Tandetnik


    Oops, sorry Igor, didn't noticed the solution! :D
    Thanks a lot, I'm going to try that...but by the sound of it, it should work!
    Tuesday, April 02, 2013 3:16 PM

  • On the other hand, if I open without ios::ate and then use myfile.seekg(0, ios::end), this calls _fseeki64 and correctly handles 64-bit offsets.

    So there's a bug in the implementation of ios::ate; using an explicit seekg is a workaround.


    Igor Tandetnik

    I can confirm the same behavior in Visual Studio 2013 Express Desktop

    Thursday, December 19, 2013 12:58 PM