none
reading large files in c# RRS feed

  • Question

  • i have to read file around 550mb file in c# process it and send to database but it gives memoryoutException

     

    which is best way to do it

     

    thanks

    Wednesday, March 2, 2011 5:17 AM

Answers

  • If you use a StreamReader that means that file is probably a text file. That makes things worse in terms of memory usage because a text file of 550 megabytes can result in a string that needs double the memory (1.1 gigabytes) do to the fact that .NET strings are UTF16 (2 byte per character) strings. That's huge.

    How to avoid this really depends on what you mean by "send to database".

    • Marked as answer by Paul Zhou Wednesday, March 9, 2011 2:38 AM
    Wednesday, March 2, 2011 11:51 AM
    Moderator

All replies

  • Can you post the code related to how you're reading it?  I assume you're reading it all in at once...if that's true, can you process the file in chunks instead?  Need more info :)

    ShaneB

    Wednesday, March 2, 2011 5:23 AM
  • thanks for reply,

    yes currently i am reading as a whole but what is best way to broke it into chunks

     

    using (StreamReader sr = File.OpenText(@"C:\test\main.txt"))

    {

    s =

    "";

     

    s = sr.ReadToEnd();

    }

     

    Wednesday, March 2, 2011 5:43 AM
  • More questions :)

    1) What type of data is in the file?  Just a plain text?  Is it line-terminated?  Is it encoded in any way?

    2) Do you need to read it all in at once to do your processing or can you read in smaller amounts? 

    We should start there...

    ShaneB

    Wednesday, March 2, 2011 6:04 AM
  • the file carries data  in particular format .i can break it into smaller format

    but it should have particular ending

     

    thanks

     

     

    Wednesday, March 2, 2011 11:38 AM
  • If you use a StreamReader that means that file is probably a text file. That makes things worse in terms of memory usage because a text file of 550 megabytes can result in a string that needs double the memory (1.1 gigabytes) do to the fact that .NET strings are UTF16 (2 byte per character) strings. That's huge.

    How to avoid this really depends on what you mean by "send to database".

    • Marked as answer by Paul Zhou Wednesday, March 9, 2011 2:38 AM
    Wednesday, March 2, 2011 11:51 AM
    Moderator