none
How to speed up IO operation when writing data into xml file RRS feed

  • Question

  • i am saving large data into xml file. which taking long time to save. please tell few good approach which reduce time to save large data into xml file. if possible share code snippet. thanks
    Tuesday, December 31, 2019 6:57 PM

Answers

  • Hi Sudip_inn, 

    Thank you for posting here.

    As William Xifaras suggested, you can use XmlWriter to write the XML.

    XmlWriter uses less memory than XmlDocument, but you will have to write the entire document from scratch each time.

    Here’s an example of using XmlWriter to create the XML file.

            static void Main(string[] args)
            {
                XmlWriter xmlWriter = XmlWriter.Create("file path");
    
                xmlWriter.WriteStartDocument();
                xmlWriter.WriteStartElement("root");
    
                xmlWriter.WriteStartElement("user");
                xmlWriter.WriteAttributeString("age", "42");
                xmlWriter.WriteString("UserA");
                xmlWriter.WriteEndElement();
    
                xmlWriter.WriteStartElement("user");
                xmlWriter.WriteAttributeString("age", "39");
                xmlWriter.WriteString("UserB");
    
                xmlWriter.WriteEndDocument();
                xmlWriter.Close();
            }
    

    Result:

    Besides, if your data stored in database, you can refer to the following references.

    1. Best approach to write huge sql dataset into xml file? [closed]
    2. C# export of large datatbase to XML

    Hope them could be helpful.

    Best Regards,

    Xingyu Zhao



    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Sudip_inn Thursday, January 2, 2020 6:45 PM
    Wednesday, January 1, 2020 8:59 AM
    Moderator
  • i have done the job this way

        List<BrokerInfo> listOfBroker = new List<BrokerInfo>()
        {
    	new BrokerInfo { Section = "TestSec1", Lineitem ="TestLi1" },
    	new BrokerInfo { Section = "TestSec2", Lineitem = "TestLi2" },
    	new BrokerInfo { Section = "TestSec3", Lineitem ="TestLi3" }
        };
    
        var xDoc = new XDocument(new XElement("Engagements",
    	    new XElement("BrokerData",
         from broker in listOfBroker
    
         select new XElement("BrokerInfo",
           new XElement("Section", broker.Section),
           new XElement("When", broker.Lineitem))
        )));
    
        xDoc.Save("D:\\BrokerInfo.xml");
    
        public class BrokerInfo
        {
            public string Section { get; set; }
            public string Lineitem { get; set; }
        }

    • Marked as answer by Sudip_inn Saturday, January 4, 2020 6:25 PM
    Saturday, January 4, 2020 6:25 PM

All replies

  • You are ultimately limited by IO speeds. If you want to speed it up so your main UI doesn't slow down then move the file writing to a secondary thread. If it is just the code itself then profile the code to see how long it is taking. If the profiler says your DB query code is slow then optimize that. If it says the actual write is slow then do the math and see if you're hitting the limits of the IO bandwidth. If it is slow converting your data to XML then consider rewriting that code. XML isn't the fastest format to write as it is.

    Michael Taylor http://www.michaeltaylorp3.net

    Tuesday, December 31, 2019 7:11 PM
    Moderator
  • What are you using to write the XML? You should use an XmlWriter as it is the most performant and uses less memory than other methods. Just keep in mind, you can not parallelize any write operations to the file using the XmlWriter as it is not thread safe.


    william xifaras


    Tuesday, December 31, 2019 8:42 PM
  • Depending on how long and frequency this operation is done consider if possible scheduling the task to perform the entire operation if this can be done unattended which is what large companies do.

    And there are no code snippets for this type of operation.


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Tuesday, December 31, 2019 9:39 PM
    Moderator
  • Hi Sudip_inn, 

    Thank you for posting here.

    As William Xifaras suggested, you can use XmlWriter to write the XML.

    XmlWriter uses less memory than XmlDocument, but you will have to write the entire document from scratch each time.

    Here’s an example of using XmlWriter to create the XML file.

            static void Main(string[] args)
            {
                XmlWriter xmlWriter = XmlWriter.Create("file path");
    
                xmlWriter.WriteStartDocument();
                xmlWriter.WriteStartElement("root");
    
                xmlWriter.WriteStartElement("user");
                xmlWriter.WriteAttributeString("age", "42");
                xmlWriter.WriteString("UserA");
                xmlWriter.WriteEndElement();
    
                xmlWriter.WriteStartElement("user");
                xmlWriter.WriteAttributeString("age", "39");
                xmlWriter.WriteString("UserB");
    
                xmlWriter.WriteEndDocument();
                xmlWriter.Close();
            }
    

    Result:

    Besides, if your data stored in database, you can refer to the following references.

    1. Best approach to write huge sql dataset into xml file? [closed]
    2. C# export of large datatbase to XML

    Hope them could be helpful.

    Best Regards,

    Xingyu Zhao



    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    • Marked as answer by Sudip_inn Thursday, January 2, 2020 6:45 PM
    Wednesday, January 1, 2020 8:59 AM
    Moderator
  • i have done the job this way

        List<BrokerInfo> listOfBroker = new List<BrokerInfo>()
        {
    	new BrokerInfo { Section = "TestSec1", Lineitem ="TestLi1" },
    	new BrokerInfo { Section = "TestSec2", Lineitem = "TestLi2" },
    	new BrokerInfo { Section = "TestSec3", Lineitem ="TestLi3" }
        };
    
        var xDoc = new XDocument(new XElement("Engagements",
    	    new XElement("BrokerData",
         from broker in listOfBroker
    
         select new XElement("BrokerInfo",
           new XElement("Section", broker.Section),
           new XElement("When", broker.Lineitem))
        )));
    
        xDoc.Save("D:\\BrokerInfo.xml");
    
        public class BrokerInfo
        {
            public string Section { get; set; }
            public string Lineitem { get; set; }
        }

    • Marked as answer by Sudip_inn Saturday, January 4, 2020 6:25 PM
    Saturday, January 4, 2020 6:25 PM
  • XDocument is keeping everything in memory (as is XmlDocument),
    so I wonder why you ignored the recommendation by Xingyu and William.
    Your code is NOT the best ways to write a large XML file.

    I wonder why you even asked in this forum, when you ignore the answers anyway.

    Saturday, January 4, 2020 7:29 PM