none
Allocating more memory for program to use

    Question

  • I have been searching for a few days trying to find a good answer to this question, but to no avail.  So please let me know if it has already been answered in full and link me to the answer please.

    I have created a program which holds a lot of information in memory.  For instance, one process may create multiple large 3-dimensional arrays, and will create other large 2 dimensional arrays that get re-dimensioned along the way.  On a couple of occasions, I have run into the problem where I try to run the process, and the program breaks, saying that there is not enough memory to declare one of the large arrays.  So my question is, how do I allocate more memory to that process so that it can create all of the necessary arrays.

     I have read a number of different posts/blogs referring to the Marshall class, but I could not find enough information on how to apply it correctly considering my situation as described above.

    Secondly, on a conceptual note, can I allocate a certain amount of free memory at program startup and have the program use the memory as it is needed, or do I have to allocate memory to a specific process or even specific declaration?  Any information on this would be helpful.  If there are any good articles I should read on this, please let me know.


    • Edited by Trigg_18 Sunday, February 24, 2013 2:26 AM
    Sunday, February 24, 2013 2:24 AM

Answers

  • Beside what monkeyboy writes you have to deal with the fact that a 32bit application can use maximum use 2Gb. There is an option (outside VB so not really something you want to use) to add another 2Gb but then you are at the absolute maximum.

    With a 64bit application (running of course on a 64bits computer) there is also a limit, but that is so high that it is currently almost never a problem.

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx


    Success
    Cor

    Sunday, February 24, 2013 7:21 AM
  • You may want to read this link

    http://msdn.microsoft.com/en-us/library/b388cb5s(v=vs.90).aspx

    Array size considerations.

    Length Limits

    The length of every dimension of an array is limited to the maximum value of the Integer data type, which is (2 ^ 31) - 1. However, the total size of an array is also limited by the memory available on your system. If you attempt to initialize an array that exceeds the amount of available RAM, the common language runtime throws an OutOfMemoryException exception.


    You've taught me everything I know but not everything you know.

    Sunday, February 24, 2013 2:48 AM
  • To increase the memory allocated to your process you can set the MaxWorking set

    http://msdn.microsoft.com/en-us/library/system.diagnostics.process.maxworkingset.aspx

    this method will work if:

       There is enough RAM 

       The memory is available (Not uses by other process)

       You have not reached the limits that are show in Cor link

       Your application is 64 bits

    When setting your working set, be careful to leave some free memory for the system and other application, otherwise you system performance will go down 

     --- Keep you MinWorkingSet low

     --- Set you MaxWorkingSet to the minimum that will let your application working 

    Sunday, February 24, 2013 10:18 AM
  • The list equivalent of a 2-dimensional array is a list of lists.

    For the type of application you have described I think the two-dimensional array is most suitable.

    Redim can consume memory "If you need to change the size of an array, you must use the ReDim Statement (Visual Basic). When you do this, Visual Basic creates a new array and releases the previous array for disposal. "
    (http://msdn.microsoft.com/en-us/library/vstudio/wak0wfyt.aspx)

    I would guess that means that there can be two copies of the array object while the redim executes and until the GC cuts in.  Can you avoid redimming the array?  If you can't, then a list might overcome the problem of changing the size without creating a copy.

    Tuesday, February 26, 2013 10:53 AM

All replies

  • You may want to read this link

    http://msdn.microsoft.com/en-us/library/b388cb5s(v=vs.90).aspx

    Array size considerations.

    Length Limits

    The length of every dimension of an array is limited to the maximum value of the Integer data type, which is (2 ^ 31) - 1. However, the total size of an array is also limited by the memory available on your system. If you attempt to initialize an array that exceeds the amount of available RAM, the common language runtime throws an OutOfMemoryException exception.


    You've taught me everything I know but not everything you know.

    Sunday, February 24, 2013 2:48 AM
  • Beside what monkeyboy writes you have to deal with the fact that a 32bit application can use maximum use 2Gb. There is an option (outside VB so not really something you want to use) to add another 2Gb but then you are at the absolute maximum.

    With a 64bit application (running of course on a 64bits computer) there is also a limit, but that is so high that it is currently almost never a problem.

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx


    Success
    Cor

    Sunday, February 24, 2013 7:21 AM
  • To increase the memory allocated to your process you can set the MaxWorking set

    http://msdn.microsoft.com/en-us/library/system.diagnostics.process.maxworkingset.aspx

    this method will work if:

       There is enough RAM 

       The memory is available (Not uses by other process)

       You have not reached the limits that are show in Cor link

       Your application is 64 bits

    When setting your working set, be careful to leave some free memory for the system and other application, otherwise you system performance will go down 

     --- Keep you MinWorkingSet low

     --- Set you MaxWorkingSet to the minimum that will let your application working 

    Sunday, February 24, 2013 10:18 AM
  • Thank you all for the responses.  I know about the array size limits, but my arrays don't reach the size limit.  The lengths of my arrays are around 6x10^7, so not quite at the limit (still another two orders of magnitude to go).  Additionally, when looking at how much memory the program is using, it is only around 10 MB, so nowhere near the 2 GB limit for 32 bit application or the 16GB on my computer.  Is there something with VB applications that limits the amount of memory they can access?  Just not sure why it is giving me this error.  If I reduce the size of the array, it works fine, but in order to carry my analysis all the way through, I need the full array. 

    Crazypennie, I will look into what you have posted there.  I have never heard of it before, so will have to read up on it.

    Monday, February 25, 2013 8:59 PM
  • Don't know if this would help or not http://msdn.microsoft.com/en-us/library/hh285054%28v=vs.110%29.aspx

    .Net Framework 4.5 - On 64-bit platforms, enables arrays that are greater than 2 gigabytes (GB) in total size.

    "The lengths of my arrays are around 6x10^7" Is that prior to redimensioning and causing the app to crash?

    And there's this too http://msdn.microsoft.com/en-us/library/hkhhsz9t%28v=vs.90%29.aspx

    "In the previous example, the jagged array saves seven elements (six in a leap year) as compared to a two-dimensional array. In a more extreme case the memory savings could be significant."


    You've taught me everything I know but not everything you know.


    Monday, February 25, 2013 9:15 PM
  • How did you come up with the 10MB figure?  That sounds really low, though we don't know anything about the application... if it is a console app then it could be quite small, but a forms application often has 15MB or more of general overhead before it actually loads any working data.  So first I would want to verify the actual reported memory usage.  You can monitor the value of My.Application.Info.WorkingSet while your routine runs to verify memory usage within your program.

    The next thing would be to check for memory leaks and to optimize your code for memory consumption.

    Generally this error means that you have a memory leak condition or you are trying to work with far too much data at one time.  Check to make sure that you are cleaning up after yourself where required and look at your data analysis routine to determine if it can be broken into more managable chunks.  Modifying the default memory management should be the absolute last resort.  This sounds like the kind of problem that should be solved by improved processing routines rather than by raw processing power, but again we don't know much about the application so maybe not.

    Out of curiosity, how much memory are we talking about here?  How much RAM does the PC running this program have installed?


    Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

    Monday, February 25, 2013 9:24 PM
    Moderator
  • So the array is set up as a public array, because multiple routines need to access it (though not a the same time).  It is an array of stored results values, so each time I decide to run a new analysis, the program re-dimensions the array (without preserving values) so that the array values are cleared.  This is where the error occurs.

    On the amount of memory, I think it was just an estimate, as I was looking at my task manager to see the amount of memory.  I can see what I get when tracking the WorkingSet.

    My computer is 64 bit windows 7, quad core, 16 GB DDR3 ram.  So plenty of memory.

    I do dimension a number of large arrays right at the beginning of the process.  Like I said, I have multiple large 3 dimensional arrays, and about a dozen reasonably large 2-dimensional arrays (3000x20).  The program must keep track of many history variables while at the same time recording analysis values at every increment.


    • Edited by Trigg_18 Monday, February 25, 2013 11:36 PM
    Monday, February 25, 2013 11:33 PM
  • So the array is set up as a public array...

    I presume you mean that it's scoped class-level by that statement.

    Why are you using arrays rather than a List(T)?

    This should be obvious but I'll ask anyway: I (and I suspect all of us in this thread) presume that this is VB Net, not VB 6, correct?


    Please call me Frank :)

    Tuesday, February 26, 2013 12:11 AM
  • Yes, class level array, and yes, VB Net.

    And I have never heard of List(T) before.  From a quick glance at the MSDN help, I'm not sure how it would help me.  I don't know how it is different from arrays, the advantages and disadvantages, etc.  I am using arrays because I need to be using multiple indices.  They are just arrays declared as Double, as the only values that should be going into them are Double values.

    Tuesday, February 26, 2013 1:51 AM
  • Yes, class level array, and yes, VB Net.

    And I have never heard of List(T) before.  From a quick glance at the MSDN help, I'm not sure how it would help me.  I don't know how it is different from arrays, the advantages and disadvantages, etc.  I am using arrays because I need to be using multiple indices.  They are just arrays declared as Double, as the only values that should be going into them are Double values.

    A List(T) actually has a larger memory footprint than an array does; that said though, a List(T) is dynamic.

    It's a type of collection so if you want to add to that collection, you add to it -- if you want to remove from it, you remove from it -- and, finally, if you're done with it, you can clear the elements completly (effectively telling it to "empty itself").

    Anything that you can use an array for, you can use a List(Of Double) for and with a list, you don't have to worry about the number of elements in that list.

    Give it thought...


    Please call me Frank :)

    Tuesday, February 26, 2013 1:57 AM
  • ...so each time I decide to run a new analysis, the program re-dimensions the array (without preserving values) so that the array values are cleared.

    You need to look closely at the code you are using here to make sure that you are not creating a copy of the exiting arrays, thus requiring twice the memory you think you need.

    Tuesday, February 26, 2013 3:12 AM
  • Hmm...it is a class level array that is populated as an analysis runs.  At the beginning of each analysis, I call:

    ReDim StoredValues(MCGrid.GetUpperBound(0), NumPhiPoints)

    I don't believe that this would be creating a copy (?).

    Frank,  thank you for explaining the List(T) to me.  However, I am still a little confused, since the MSDN examples on the help page only show 1-dimensional.  Would I say, add rows to the list?  Or can I add to a list in a similar manner as an array?

    Tuesday, February 26, 2013 5:03 AM
  • The list equivalent of a 2-dimensional array is a list of lists.

    For the type of application you have described I think the two-dimensional array is most suitable.

    Redim can consume memory "If you need to change the size of an array, you must use the ReDim Statement (Visual Basic). When you do this, Visual Basic creates a new array and releases the previous array for disposal. "
    (http://msdn.microsoft.com/en-us/library/vstudio/wak0wfyt.aspx)

    I would guess that means that there can be two copies of the array object while the redim executes and until the GC cuts in.  Can you avoid redimming the array?  If you can't, then a list might overcome the problem of changing the size without creating a copy.

    Tuesday, February 26, 2013 10:53 AM
  • Hmm...it is a class level array that is populated as an analysis runs.  At the beginning of each analysis, I call:

    ReDim StoredValues(MCGrid.GetUpperBound(0), NumPhiPoints)

    I don't believe that this would be creating a copy (?).

    Frank,  thank you for explaining the List(T) to me.  However, I am still a little confused, since the MSDN examples on the help page only show 1-dimensional.  Would I say, add rows to the list?  Or can I add to a list in a similar manner as an array?

    As Acamar said, use a List(List Of(Double)).

    But anytime I see multi-dimensional arrays, I wonder if there's not a better way to go about doing it and [almost] always there is.

    If you'd care to explain more about what you're doing I feel sure the brilliant folks here will have some guiding tips.

    Good luck with it. :)


    Please call me Frank :)

    Tuesday, February 26, 2013 3:23 PM