none
How much memory my list<employee> consumed RRS feed

  • Question

  • My pc has win7 i am using vs2013 and project version 4.5.2

    I am storing large data into list say 200 000 data and getting outofmemory exception

    so how could i detect how much memory my list consumed?  Give me some code example  to detect memory used by list.

    I can not go for 64 setting that is my constraint. So please suggest some good idea to get rid of this situation.

    should i use gc.Collect and gc.Waitforpendingfinalizer?

    Friday, August 23, 2019 7:50 PM

Answers

  • Then you cannot store it all in memory.  That's not unusual; that's the very reason why database managers were invented.  You store the items on disk and work on them a few at a time.  There are many database interfaces for C#, including some that are free.  SQLite is free and quite competent.


    Tim Roberts | Driver MVP Emeritus | Providenza &amp; Boekelheide, Inc.

    • Marked as answer by Sudip_inn Saturday, August 24, 2019 7:43 PM
    Saturday, August 24, 2019 6:37 AM
  • Your problem is out of memory exception, as per my understanding. Whereas, you like to know that how much data we can store in a list. I think you have a solution in your mind, if I am not wrong.

    First of all it's not possible to guess the size of data without any hint, and memory limit is something depend on your system itself.

    You can use GC.GetTotalMemory to get an approx size of your list. You need to check the allocated size before add element into list and check again and again after add new element into list.

    Anyway nothing is a good idea until or unless someone understand that why you want to put 20000 records in a single list. Because, it's sounds like an unconventional way to do such things. 


    Avik Das


    • Edited by D Avik Saturday, August 24, 2019 12:03 PM
    • Marked as answer by Sudip_inn Saturday, August 24, 2019 7:43 PM
    Saturday, August 24, 2019 11:20 AM
  • The cache should solve your out-of-memory problem, if you use a properly coded and well-configured cache. The way in which the cache works is that every time you give it something to store, it checks whether there is enough memory, and if there isn't, it discards something that it thinks is the least likely to be needed and stores instead the new piece of data that you gave it to store.

    Therefore, you will not run out of memory (or at least not due to the cache). BUT: you need to be prepared to lose some data. When you tell to the cache "give me back such and such pieces of information" it can respond "sorry I don't have it" if it is something that it has discarded to make some room for something else. This is OK if you have an alternative way of retrieving such data (for example, if it was originally read from a database, you can go back to the database and read it again). But if your data is unique and irreplaceable then no, the cache is not the solution.

    • Marked as answer by Sudip_inn Sunday, August 25, 2019 5:26 PM
    Saturday, August 24, 2019 2:05 PM
    Moderator

All replies

  • How large are your data items?  If you have 200,000 items and each item takes 2,500 bytes, then you're going to consume half a gigabyte of memory.  Garbage collection doesn't do any good if the memory is still in use.

    It sounds like you may need to consider external storage, like using an SQL database, or storing the items on disk and processing them one at a time.


    Tim Roberts | Driver MVP Emeritus | Providenza &amp; Boekelheide, Inc.

    Friday, August 23, 2019 7:55 PM
  • I am storing large data into list and getting outofmemory exception. If i store data into cache does it solve my problem?

    Please give some suggestions. Thanks

    Friday, August 23, 2019 7:56 PM
  • Visual Studio contains a profiler (even the community edition):
    Measure memory usage in Visual Studio

    Friday, August 23, 2019 9:00 PM
  • sorry i am loading 20,00,000 data. my list is huge data.

    how could i detect how much memory my list consumed by code ? any idea sir.

    Saturday, August 24, 2019 4:33 AM
  • Then you cannot store it all in memory.  That's not unusual; that's the very reason why database managers were invented.  You store the items on disk and work on them a few at a time.  There are many database interfaces for C#, including some that are free.  SQLite is free and quite competent.


    Tim Roberts | Driver MVP Emeritus | Providenza &amp; Boekelheide, Inc.

    • Marked as answer by Sudip_inn Saturday, August 24, 2019 7:43 PM
    Saturday, August 24, 2019 6:37 AM
  • Your problem is out of memory exception, as per my understanding. Whereas, you like to know that how much data we can store in a list. I think you have a solution in your mind, if I am not wrong.

    First of all it's not possible to guess the size of data without any hint, and memory limit is something depend on your system itself.

    You can use GC.GetTotalMemory to get an approx size of your list. You need to check the allocated size before add element into list and check again and again after add new element into list.

    Anyway nothing is a good idea until or unless someone understand that why you want to put 20000 records in a single list. Because, it's sounds like an unconventional way to do such things. 


    Avik Das


    • Edited by D Avik Saturday, August 24, 2019 12:03 PM
    • Marked as answer by Sudip_inn Saturday, August 24, 2019 7:43 PM
    Saturday, August 24, 2019 11:20 AM
  • The cache should solve your out-of-memory problem, if you use a properly coded and well-configured cache. The way in which the cache works is that every time you give it something to store, it checks whether there is enough memory, and if there isn't, it discards something that it thinks is the least likely to be needed and stores instead the new piece of data that you gave it to store.

    Therefore, you will not run out of memory (or at least not due to the cache). BUT: you need to be prepared to lose some data. When you tell to the cache "give me back such and such pieces of information" it can respond "sorry I don't have it" if it is something that it has discarded to make some room for something else. This is OK if you have an alternative way of retrieving such data (for example, if it was originally read from a database, you can go back to the database and read it again). But if your data is unique and irreplaceable then no, the cache is not the solution.

    • Marked as answer by Sudip_inn Sunday, August 25, 2019 5:26 PM
    Saturday, August 24, 2019 2:05 PM
    Moderator
  • Hi Sudip,

    Where is the data coming from? Do have a reason for having to store all of it in memory? You could design some sort of caching mechanism by writing the data to files or in a database.

    We need more info about your situation ...


    ~~Bonnie DeWitt [C# MVP]

    http://geek-goddess-bonnie.blogspot.com

    Sunday, August 25, 2019 1:23 AM
    Moderator
  • Since two threads were merged here, the replies might look a bit strange and out of order. Sorry about that, maybe I shouldn't have merged them?

    ~~Bonnie DeWitt [C# MVP]

    http://geek-goddess-bonnie.blogspot.com

    Sunday, August 25, 2019 1:39 AM
    Moderator