none
How to address storage that is outside the 32 bit address space? RRS feed

  • Question

  • I just finished reviewing a blog post by Eric Lippert (http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx) talking about how we can request as much memory from Windows as we want (say 5GB) but can only address 4GB of it directly:

    "So, suppose the 32 bit process requires huge amounts of storage, and it asks for storage many times. Perhaps it requires a total of 5 GB of storage. The operating system finds enough disk space for 5GB in files and tells the process that sure, the storage is available. How does the process then write to that storage? The process only has 32 bit pointers, but uniquely identifying every byte in 5GB worth of storage would require at least 33 bits.

    ...

    The 5GB of storage is split up into chunks, typically 4KB each, called “pages”. The operating system gives the process a 4GB “virtual address space” – over a million pages - which can be addressed by a 32 bit pointer. The process then tells the operating system which pages from the 5GB of on-disk storage should be “mapped” into the 32 bit address space."

    Ok, wonderful. It seems then I can ask for as much storage as I want and, at any given time, have 4GB of my "favorite" memory mapped into my address space and read/write from it using any and all possible 32 bit pointers.  So what happens when I am done with that first 4GB and want to move on to working with the last GB?  In the comments section Eric makes it sound like this is possible:

    "The use case of memory is typically that you want to remember something. Perhaps you have four billion bytes of video that you are processing, and you want to keep the whole video in memory. You could map the bits you're currently editing into virtual memory while leaving the rest unmapped until you're done with what you've got. -- Eric"

    So I assume what Eric is talking about is changing the page table programmatically.  Is it really possible to instruct the CLR to in turn instruct Windows to do this for us?  Can we really decide that we are tired of "logical memory page0" pointing to "physical memory frame7" and change it to frame9? If so, how?  If not, how do I get to that 5th gig of data?

     

     

    Wednesday, August 11, 2010 10:45 PM

Answers

  •  

    Hi,

     

    SetFilePointer function can moves the file pointer of the specified file, and you may image that the function takes a 64bit pointer as parameter:

     

    DWORD WINAPI SetFilePointer(

      __in         HANDLE hFile,

      __in         LONG lDistanceToMove,

      __inout_opt  PLONG lpDistanceToMoveHigh,

      __in         DWORD dwMoveMethod

    );

     

    If lpDistanceToMoveHigh is not NULL, lpDistanceToMoveHigh and lDistanceToMove form a single 64-bit signed value that specifies the distance to move.

     

    So that we can access files which larger than 4GB in our applications, for example, FileStream class internally uses SetFilePointer function.


    Sincerely,
    Eric
    MSDN Subscriber Support in Forum
    If you have any feedback of our support, please contact msdnmg@microsoft.com.
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    • Marked as answer by eryang Friday, August 20, 2010 2:13 AM
    Thursday, August 12, 2010 10:10 AM

All replies

  •  

    Hi,

     

    SetFilePointer function can moves the file pointer of the specified file, and you may image that the function takes a 64bit pointer as parameter:

     

    DWORD WINAPI SetFilePointer(

      __in         HANDLE hFile,

      __in         LONG lDistanceToMove,

      __inout_opt  PLONG lpDistanceToMoveHigh,

      __in         DWORD dwMoveMethod

    );

     

    If lpDistanceToMoveHigh is not NULL, lpDistanceToMoveHigh and lDistanceToMove form a single 64-bit signed value that specifies the distance to move.

     

    So that we can access files which larger than 4GB in our applications, for example, FileStream class internally uses SetFilePointer function.


    Sincerely,
    Eric
    MSDN Subscriber Support in Forum
    If you have any feedback of our support, please contact msdnmg@microsoft.com.
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    • Marked as answer by eryang Friday, August 20, 2010 2:13 AM
    Thursday, August 12, 2010 10:10 AM
  •  

    We temporarily mark a reply, please remember to click "Mark as Answer" on the post that helps you, and to click "Unmark as Answer" if a marked post does not actually answer your question. This can be beneficial to other community members reading the thread.


    Sincerely,
    Eric
    MSDN Subscriber Support in Forum
    If you have any feedback of our support, please contact msdnmg@microsoft.com.
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.
    Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.
    Friday, August 20, 2010 2:13 AM