locked
How to store huge data in List<string> RRS feed

  • Question

  • User970350993 posted

    Hi All,

    Could you please help me to optimize  the code which is written to store huge data in list<string> like this

    List<string> msg = new List<string>();
    for (int i = 0; i < 100000000; i++)
    {
    string m = "I am string message";
    msg.Add(m);
    Response.Write(msg[0].ToString());
    }

    Currently this is throwing the out of memory exception.  

    Tuesday, April 24, 2018 6:03 PM

Answers

  • User970350993 posted

    Hi All,

    Thanks for reply. I discussed with business people, I told them, If you will provide the huge input to application, it will generate huge data in memory and we can only store upto 2 GB data.  Even though it will take more than 20 min to read  data.

    Actually that input is not like real input, they are thinking on it.  They are giving the combination of 4 to 5 test case in single statement.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, April 26, 2018 10:48 AM

All replies

  • User-851967432 posted

    That's because List<> has capacity of Int32.

    Your code is impractical any way. There is nothing to optimize.

    Tuesday, April 24, 2018 6:32 PM
  • User753101303 posted

    Hi,

    Try perhaps  https://docs.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcallowverylargeobjects-element and make sure to force 64 bit mode.

    It's around 3.5 Gb of data if I'm not mistaken. Beyond that "brute force" approach, it highly depends on what you are trying to do (for example you could store each unique value and a usage count or use a db etc...).

    Tuesday, April 24, 2018 6:37 PM
  • User303363814 posted

    What are you really trying to do?  What business problem are you trying to solve?

    Let's do some small calculations.  Your string is nineteen character long and you are appending it to the Response 100 million times.  That's 1.9 gigabytes being returned to the client page.  At, say, 2 megabits per second internet connection speed that will take about 10,000 seconds.

    Roughly three hours to load the page.  Hmmm, doesn't seem like a real requirement.

    Tuesday, April 24, 2018 11:34 PM
  • User970350993 posted

    Roughly three hours to load the page.  Hmmm, doesn't seem like a real requirement.

    Hi PaulTheSmith,

    I am fixing the bugs of one  of real life application. There is the logic to generate possible combination on basis of reg-expression. That is generating more than 10000 records, then it being stored in list<string> data type. Then they are doing  some string filtration. The exception is throwing while storing the huge data in list<string>.

    I have to ask with business, is the input is really correct ? But i hope they will tell, my input could be any things.    

    Wednesday, April 25, 2018 12:43 AM
  • User-1716253493 posted

    The bug is storing hug records from database into web server memmory

    Maybe you can find another way to not download 100000000 records to web server

    Maybe find way to move filteration in the database

    Wednesday, April 25, 2018 1:29 AM
  • User753101303 posted

    And it worked previously ? It depends on data found somewhere maybe in a database ? This is not something like a full text search ?

    Wednesday, April 25, 2018 7:40 AM
  • User970350993 posted

    There is no requirement for store in database or fetch from database. They will the give the input parameter, on basis of input there is one method to generate the possible combination on basis of input, Then that possible input then will hold in List<string> and it will display in UI.

    It is working for small input. But when they are giving 30 to 40 input in this scenario it will take more memory. Then it is throwing exception. 

    Wednesday, April 25, 2018 8:01 AM
  • User36583972 posted

    Hi chandradev,

    They will the give the input parameter, on basis of input there is one method to generate the possible combination on basis of input, Then that possible input then will hold in List<string> and it will display in UI.

    It is working for small input. But when they are giving 30 to 40 input in this scenario it will take more memory. Then it is throwing exception. 

    You are generating huge data and store in List<string>, And then filter it in this collection. It is easy to call the out of memory exception.

    As far as I know, there is not a good way to avoid your issue. I suggest you can refer the following ideas.

    1: limit the number of input parameters.

    2: Save the huge data in the database. Then, show the data from the Data Controls(Paging display).


    Best Regards,

    Yong Lu

    Wednesday, April 25, 2018 8:27 AM
  • User-1716253493 posted

    Before filter the list string, maybe you can also filter the data from database to limit the records.

    For exampe you want filter the list based "abc123", then you can limit the records based "abc"

    Wednesday, April 25, 2018 8:37 AM
  • User753101303 posted

    And you can't filter as you go ? You are are doing multiple combinations between all those inputs to generate a huge list and then only you filter ? It's is not possible to filter this list earlier ?

    A higher level view could help (looks like a "full text search", "spell checking" or making a search for "similarities" to deduplicate data ????) but if I you really have no way to optimize your approach by using less data at some point using some form of "db" could be helpfull.

    It could be writing to a file and then readback the file to load only what you need from what you generated... Also I would compute the size I need for the worst case scenario to see what would fit (to avoid trying to solve the case for 3.5 Gb and then later end up in needing 10 Gb etc...)

    Wednesday, April 25, 2018 9:59 AM
  • User303363814 posted

    There is the logic to generate possible combination on basis of reg-expression. That is generating more than 10000 records, then it being stored in list<string> data type. Then they are doing  some string filtration

    Generate one combination, if it does not pass filter then discard otherwise keep.  Repeat until all combinations are generated.

    Thursday, April 26, 2018 2:24 AM
  • User-1716253493 posted

    maybe you limit the list by split it into several lists or using collection

    Thursday, April 26, 2018 3:55 AM
  • User970350993 posted

    Hi All,

    Thanks for reply. I discussed with business people, I told them, If you will provide the huge input to application, it will generate huge data in memory and we can only store upto 2 GB data.  Even though it will take more than 20 min to read  data.

    Actually that input is not like real input, they are thinking on it.  They are giving the combination of 4 to 5 test case in single statement.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Thursday, April 26, 2018 10:48 AM