none
Memory Allocation in .NET RRS feed

  • Question

  • I wanted to read a xml file of >1 GB. But what i got is out of memory exception. I think it's because of ram size(i've 1.47 GB). Is there any way to use pagefile or virtual memory. or any other way. I want to read whole file in buffer, since it's required by XElement.

    Thanx in advance.

    Guramrit Singh
    Tuesday, September 9, 2008 5:34 AM

Answers

  • OOM has nothing to do with the amount of RAM you have.  You are running out of virtual memory space.  There's a hard limit of 2 GB on 32-bit Windows.  You can extend that with the /3GB boot option but that's troublesome on modern machines.  You'll need to switch to 64-bit Windows for a real fix.  Or a smarter way to handle the XML.
    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Monday, September 15, 2008 6:37 AM
    Tuesday, September 9, 2008 11:18 AM
    Moderator
  • You're refering to System.Xml.Linq.XElement, right? I don't know much about LINQ or LINQ to XML, but my understanding is that LINQ to XML is an in-memory model: your DOM lives entirely within memory, which as you've found can be cost-prohibitive on a 32-bit platform (you've likely run out of address space).

    If you aren't required to use LINQ to XML you might look for a more efficient way to process your large XML document. For example, XmlReader http://msdn.microsoft.com/en-us/library/system.xml.xmlreader(VS.85).aspx (as Hans said, "a smarter way to handle the XML").

    If you need to use LINQ to XML you might be required to switch to 64-bit Windows to make this feasible. Consider that on loading your 1.47 GB XML document suitable data structures must be built to allow traversal and querying of your document. On top of that, if you're loading a UTF-8 XML document of largely ASCII characters your string content will take even more storage space when it gets stored in memory as UTF-16.

    Alternatively, you might whittle down your large XML document into smaller XML documents, process the smaller documents individually, and combine your results when done.


    Curt - http://www.codeneverwritten.com/
    • Marked as answer by Zhi-Xin Ye Monday, September 15, 2008 6:37 AM
    Wednesday, September 10, 2008 8:41 PM

All replies

  • OOM has nothing to do with the amount of RAM you have.  You are running out of virtual memory space.  There's a hard limit of 2 GB on 32-bit Windows.  You can extend that with the /3GB boot option but that's troublesome on modern machines.  You'll need to switch to 64-bit Windows for a real fix.  Or a smarter way to handle the XML.
    Hans Passant.
    • Marked as answer by Zhi-Xin Ye Monday, September 15, 2008 6:37 AM
    Tuesday, September 9, 2008 11:18 AM
    Moderator
  • have you tried looking into file mapping (memory mapped file)?
    Wednesday, September 10, 2008 6:48 PM
  • You're refering to System.Xml.Linq.XElement, right? I don't know much about LINQ or LINQ to XML, but my understanding is that LINQ to XML is an in-memory model: your DOM lives entirely within memory, which as you've found can be cost-prohibitive on a 32-bit platform (you've likely run out of address space).

    If you aren't required to use LINQ to XML you might look for a more efficient way to process your large XML document. For example, XmlReader http://msdn.microsoft.com/en-us/library/system.xml.xmlreader(VS.85).aspx (as Hans said, "a smarter way to handle the XML").

    If you need to use LINQ to XML you might be required to switch to 64-bit Windows to make this feasible. Consider that on loading your 1.47 GB XML document suitable data structures must be built to allow traversal and querying of your document. On top of that, if you're loading a UTF-8 XML document of largely ASCII characters your string content will take even more storage space when it gets stored in memory as UTF-16.

    Alternatively, you might whittle down your large XML document into smaller XML documents, process the smaller documents individually, and combine your results when done.


    Curt - http://www.codeneverwritten.com/
    • Marked as answer by Zhi-Xin Ye Monday, September 15, 2008 6:37 AM
    Wednesday, September 10, 2008 8:41 PM