none
More issues with array size limits RRS feed

  • General discussion

  • Hello!
    There are a lot of discussions about maximum size of an array on the network, but I can not figure it out because I get totally different numbers.
    As far as internet says I am limited by the maximum of the Int32 counter and 2GB of memory.
    In practice this turns out to be about 1.3GB for some reason:

    byte[] test = new byte[1412751264]; //ok
    byte[] test = new byte[1412751265]; //allocation error
    float[] test = new float[352626255]; //ok
    float[] test = new float[352626256]; //allocation error
    byte[,] test = new byte[37763, 37763]; //ok
    byte[,] test = new byte[37764, 37764]; //allocation error
    float[,] test = new float[18881, 18881]; //ok
    float[,] test = new float[18882, 18882]; //allocation error

    More than that, some strange numbers when I use struct:

    public struct SysFloatX4
    {
         public float A;
         public float B;
         public float C;
         public float D;
    }
    
    // next shows 16:
    
    MessageBox.Show(System.Runtime.InteropServices.Marshal.SizeOf(typeof(FloatX4)).ToString());
    
    // so I thought that I will have at least 9440x9440 array, but:
     
    SysFloatX4[,] test2 = new SysFloatX4[2709, 2709]; //ok
    SysFloatX4[,] test2 = new SysFloatX4[2710, 2710]; //Allocation error

    I never encountered this because I did not work with such volumes of data. I need 10000x10000x4xfloat maximum. It's okay to split input data into smaller arrays, as I will pass that to GPU anyway. But I still need some alternative for struct, as I get data from GPU and tricks with 4 separate arrays seems bad idea. I am using VS2012 on X64 machine, framework version 4.5. Tested it in empty default project. There is about 6GB of free memory on machine.
    Am I missing something obvious?



    • Edited by GIRATOR Monday, June 11, 2018 9:15 AM
    Sunday, June 10, 2018 11:12 PM

All replies

  • Hi,

    you need to compile the program for 64 bit (any CPU, no preferring 32 bit), else you will have the general memory limit for 32 bit apps in .net which is around 1.4 GB.

    Regards,

      Thorsten

    Monday, June 11, 2018 4:04 AM
  • Thank you, now it makes sense, I generally will not have enough memory on a 32-bit system with this approach. Unfortunately, the project must run on 32-bit systems so I'll have to change the model and move even more to the GPU.
    But what about Struct? The array of such struct-s fills correctly with 4-byte blocks of data...
    Does it create something huge and "hidden" for each struct-element?
    Monday, June 11, 2018 9:08 AM
  • Yes, .NET adds overhead to the structures it creates, to support introspection, among other things.  By default, it makes no guarantees at all about the layout of structures and arrays in memory.  If you intend to pass these arrays to unmanaged processes, like GPUs, then you have to do more work.  You might consider using the StructLayout(LayoutKind.Sequential) attribute to see if that changes things.

    > ... maximum of the Int32 counter and 2GB of memory.  In practice this turns out to be about 1.3GB for
    > some reason.

    Your entire PROCESS is limited to 2GB of memory.  That includes your program EXE, the .NET runtime, and all of the system DLLs it requires.


    Tim Roberts, Driver MVP Providenza & Boekelheide, Inc.

    Monday, June 11, 2018 7:51 PM

  • Your entire PROCESS is limited to 2GB of memory.  

    Or 3GB if IMAGE_FILE_LARGE_ADDRESS_AWARE is enabled and 4GT is used.

    Memory Limits for Windows and Windows Server Releases
    https://msdn.microsoft.com/en-us/library/aa366778.aspx

    - Wayne

    Monday, June 11, 2018 11:56 PM
  • "array of such struct-s fills correctly with 4-byte blocks of data" maybe works because

    "C#, Visual Basic, and C++ compilers apply the Sequential layout value to structures by default."

    forum not allows to insert link (with my reputation ?), but I found that article on this site.

    My question about struct was more about why it takes so much memory, 2709x2709 instead of 9440x9440 is like 10 times more for each 16 byte element, isn't it too much for "overhead" ? :(
    Tuesday, June 12, 2018 9:39 AM
  • More than that, some strange numbers when I use struct:

    public struct SysFloatX4
    {
         public float A;
         public float B;
         public float C;
         public float D;
    }
    
    // next shows 16:
    
    MessageBox.Show(System.Runtime.InteropServices.Marshal.SizeOf(typeof(FloatX4)).ToString());
    
    // so I thought that I will have at least 9440x9440 array, but:
     
    SysFloatX4[,] test2 = new SysFloatX4[2709, 2709]; //ok
    SysFloatX4[,] test2 = new SysFloatX4[2710, 2710]; //Allocation error


    >SysFloatX4[,] test2 = new SysFloatX4[2710, 2710]; //Allocation error

    Are you actually seeing an allocation error for that line? 

    If so, were there other allocations done before it?

    If so, were there other allocations deleted before this one was attempted?

    Does that example give an allocation error for you when it is the only 
    allocation in the program?

    Do you see an allocation error from this code? You shouldn't.

    static void Main(string[] args)
    {
        SysFloatX4[,] test2 = new SysFloatX4[9440, 9440];
    }
    

    As Tim pointed out, the memory limits are on available address space for
    the entire process. So if your program has lots of code and/or data then 
    the max size for an allocation will be reduced accordingly.

    Factors that can lead to allocation errors include:

    Overall process memory usage is high.

    Available heap is insufficient due to other active allocations.

    Heap fragmentation.

    Garbage collection has not yet reclaimed allocated heap from prior
    deleted (or no longer referenced) allocations.

    Heap corruption due to program bugs can also cause allocation failures.

    - Wayne

    Tuesday, June 12, 2018 12:06 PM
  • Do you see an allocation error from this code? You shouldn't.

    static void Main(string[] args)
    {
        SysFloatX4[,] test2 = new SysFloatX4[9440, 9440];
    }

    hm... yes I don't get error here... I "Tested it in empty default project." but that was WindowsForms, not console. Now I think I'm too impatient usingVS (?):

    that example is the only allocation in the program and DON'T give an allocation error only on first launch from VS, or if I starting exe directly from folder.


    Tuesday, June 12, 2018 12:47 PM