none
[XML] Huge file manage (> 2GB) RRS feed

  • Question

  • Hi,

    I have an XML file to import into a new database with EntityFramework.

    File looks like this :

    <?xml version="1.0" encoding="utf-8"?>
    <ArrayOfExternAnimal xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <ExternAnimal Breed="Griffon Fauve de Bretagne" Name="Oska" Male="false" Coat="" ChipOrTattoo="BDT567" Lof="149/40" FatherId="0" MotherId="0" SccDatabaseId="900005">
        <BirthDate>1978-07-14T00:00:00</BirthDate>
        <DeathDate>0001-01-01T00:00:00</DeathDate>
        <Quotation>Confirmé</Quotation>
        <Dna />
        <SccHealth />
      </ExternAnimal>
      <ExternAnimal Breed="Griffon Fauve de Bretagne" Name="Panisse" Male="true" Coat="F.Fo." ChipOrTattoo="BAR660" Lof="150/78" FatherId="899881" FatherName="Ludo" MotherId="899791" MotherName="Mascotte" SccDatabaseId="900012">
        <BirthDate>1979-11-30T00:00:00</BirthDate>
        <DeathDate>0001-01-01T00:00:00</DeathDate>
        <Quotation>Confirmé</Quotation>
        <Dna />
        <SccHealth />
      </ExternAnimal>
      <ExternAnimal Breed="Griffon Fauve de Bretagne" Name="Petula" Male="false" Coat="F.Cl." ChipOrTattoo="BAR662" Lof="152/79" FatherId="899881" FatherName="Ludo" MotherId="899791" MotherName="Mascotte" SccDatabaseId="900013">
        <BirthDate>1979-11-30T00:00:00</BirthDate>
        <DeathDate>0001-01-01T00:00:00</DeathDate>
        <Quotation>Confirmé</Quotation>
        <Dna />
        <SccHealth />
      </ExternAnimal>
    </ArrayOfExternAnimal>

    But the DeSerialize is not a good way for a such big file. So, what is the best way to do this ?

    Thanks,

    Vincent

    • Moved by CoolDadTx Wednesday, February 27, 2019 3:03 PM EF related
    Monday, February 25, 2019 5:28 PM

Answers

  • Hi Duvernet Vicent,

    Thank you for posting here.

    For your question, you could refer to the suggestions in StackOverFlow.

    https://stackoverflow.com/questions/5535270/import-data-from-xml-into-database

    And you could download the source file from the MSDN document: How to import/export the XML files using Code First in EF (CSEFStoreXmlFiles).

    https://code.msdn.microsoft.com/CSEFStoreXmlFiles-d58d5702

    Best Regards,

    Wendy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, February 26, 2019 2:23 AM
  • I'm still confused. :/

    the XDocument returned point on ExternAnimal, that's what I'm looking for nice :)

    But for the next step : Is there a way to automap the XDocument to my POCO class ExternAnimal ? (maybe it could be the job of our tool DevArt Entities Developper (https://www.devart.com/entitydeveloper/) ?

    You can use Linq-2-XML and do a projection into a custom type, the POCO.

    https://www.intertech.com/Blog/query-an-xml-document-using-linq-to-xml/

    https://csharp-station.com/Tutorial/Linq/Lesson02

    {
    	var query = from t in xdoc.Descendants("book")
    				where t.Element("genre").Value.ToLower() == genre.ToLower() &&
    					decimal.Parse(t.Element("price").Value) <= maxPrice
    				select new {
    					ID=t.Attribute("id").Value,
    					Title=t.Element("title").Value,
    					Price=t.Element("price").Value };
     
    	Console.WriteLine();
    	Console.WriteLine("Books in '{0}' genre with a maximum price of {1}:",
    					genre, maxPrice);
     
    	foreach (var item in query)
    	{
    		Console.WriteLine("* ID: {0}, Title: {1}, Price: {2}",
    		item.ID, item.Title, item.Price);
    	}

    Thursday, March 21, 2019 10:54 PM

All replies

  • Hi Duvernet Vicent,

    Thank you for posting here.

    For your question, you could refer to the suggestions in StackOverFlow.

    https://stackoverflow.com/questions/5535270/import-data-from-xml-into-database

    And you could download the source file from the MSDN document: How to import/export the XML files using Code First in EF (CSEFStoreXmlFiles).

    https://code.msdn.microsoft.com/CSEFStoreXmlFiles-d58d5702

    Best Regards,

    Wendy


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, February 26, 2019 2:23 AM
  • In order to parse large files, you can use an XmlReader and read the data sequentially, for example:

    using( XmlReader r = XmlReader.Create( @"C:\MyFile.xml" ) )
    {
        if( r.Read() )
        {
            r.ReadStartElement( "ArrayOfExternAnimal" );
            r.MoveToContent();
            if( r.IsStartElement( "ExternAnimal" ) )
            {
                do
                {
                    using( var sr = r.ReadSubtree() )
                    {
                        var xdoc = XDocument.Load( sr ); // "ExternAnimal"
                        // TODO: extract and process data
                        // . . .
                    }
    
                } while( r.ReadToNextSibling( "ExternAnimal" ) );
            }
        }
    }
    

    Tuesday, February 26, 2019 6:38 AM
  • I'm still confused. :/

    the XDocument returned point on ExternAnimal, that's what I'm looking for nice :)

    But for the next step : Is there a way to automap the XDocument to my POCO class ExternAnimal ? (maybe it could be the job of our tool DevArt Entities Developper (https://www.devart.com/entitydeveloper/) ?

    Thursday, March 21, 2019 11:09 AM
  • I'm still confused. :/

    the XDocument returned point on ExternAnimal, that's what I'm looking for nice :)

    But for the next step : Is there a way to automap the XDocument to my POCO class ExternAnimal ? (maybe it could be the job of our tool DevArt Entities Developper (https://www.devart.com/entitydeveloper/) ?

    You can use Linq-2-XML and do a projection into a custom type, the POCO.

    https://www.intertech.com/Blog/query-an-xml-document-using-linq-to-xml/

    https://csharp-station.com/Tutorial/Linq/Lesson02

    {
    	var query = from t in xdoc.Descendants("book")
    				where t.Element("genre").Value.ToLower() == genre.ToLower() &&
    					decimal.Parse(t.Element("price").Value) <= maxPrice
    				select new {
    					ID=t.Attribute("id").Value,
    					Title=t.Element("title").Value,
    					Price=t.Element("price").Value };
     
    	Console.WriteLine();
    	Console.WriteLine("Books in '{0}' genre with a maximum price of {1}:",
    					genre, maxPrice);
     
    	foreach (var item in query)
    	{
    		Console.WriteLine("* ID: {0}, Title: {1}, Price: {2}",
    		item.ID, item.Title, item.Price);
    	}

    Thursday, March 21, 2019 10:54 PM
  • 2 very interesting links :) I understand more things now.

    But in my case, it don't seems to work :

    var query = from t in xdoc.Descendants("ExternAnimal")
                         select t.Value;

    called just after 'var xdoc = XDocument.Load(sr);'

    gives a query.Current = null.

    Here's the debuggeur view : 

    

    Why am I wrong ?

    Friday, March 22, 2019 6:49 PM
  • In order to execute the query, try expanding the Result View subnode.

    Friday, March 22, 2019 7:02 PM
  • Ok, I need to call query.First().

    query has returned all sub nodes text.

    Because the Name property is an attribute, that's why query2 is empty. How can we get the attribute value then ?

    Friday, March 22, 2019 7:22 PM

  • Here's the full working code - thanks for all for help :) :

    private static void LoadXml(string xmlFile)
            {
                using (XmlReader r = XmlReader.Create(xmlFile))
                {
                    if (r.Read())
                    {
                        r.ReadStartElement("ArrayOfExternAnimal");
                        r.MoveToContent();
                        if (r.IsStartElement("ExternAnimal"))
                        {
                            do
                            {
                                using (var sr = r.ReadSubtree())
                                {
                                    var xdoc = XDocument.Load(sr); // "ExternAnimal"
    
                                    var XmlAnimal = from i in xdoc.Descendants("ExternAnimal")
                                                select new ExternAnimal
                                                {
                                                    Name = (string)i.Attribute("Name"),
                                                    Quotation = (string)i.Element("Quotation")
                                                };
    
                                    ExternAnimal animal = XmlAnimal.First();
                                    if (animal.IsValid())
                                    {
                                        Console.WriteLine($"Animal : {animal.Name}");
                                    }
                                }
    
                            } while (r.ReadToNextSibling("ExternAnimal"));
                        }
                    }
                }
            }

    Friday, March 22, 2019 7:43 PM