locked
System out of memory exception due to mismanaged memory using threads RRS feed

  • Question

  • Hello,

    This is the first time I wrote a Multi threaded app and hence I assume I made quite a few grave mistakes which is causing my application to crash.

    Our application recieves a lot of Search - 175,000 queries per day. I collected all these and created 5 xml files each containing 40000 rows of quereis. Then I create an object with these 40000 rows and loop one by one -> Search and Save results. Since I had identify which thread configuration is best, I have to run with 5,10,20,40,80...

    Thus I created the below code but during the App memory increases to above 1 GB and hence eventually crashes. I tried increasing the RAM to 8GB but still application crashes once it reaches 1.5 GB. I know that its due to unmanaged Threading and coding, Any help will be appreciated.

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO;
    //using Microsoft.VisualStudio.TestTools.UnitTesting;
    using System.Configuration;
    using System.Threading;

    using xxxxxxxxxxx.com.model.resource;
    using xxxxxxxxxxx.com.model.resource.metadata;
    using xxxxxxxxxxx.com.model.resource.management;
    using System.Xml;
    using System.Xml.Serialization;
    using System.Xml.Linq;
    using System.Text.RegularExpressions;


    namespace TestMultiThread
    {
        class Program
        {
            static string testPath = ConfigurationManager.AppSettings["TestFilespath"];
            static void Main(string[] args)
            {
                List<int> noOfThreads = ConfigurationManager.AppSettings["NoofThreadsCommaSeperated"].Split(',').Select(x => int.Parse(x)).ToArray().ToList();

                MultiThreadRunQuerySaveResults mtr = new MultiThreadRunQuerySaveResults();

                string inputFilePath = ConfigurationManager.AppSettings["InputFilePath"];
                string inputFileNames = ConfigurationManager.AppSettings["InputFileNames"];
                string[] filePaths = Directory.GetFiles(inputFilePath, inputFileNames + "*.xml", SearchOption.AllDirectories);

                if (!System.IO.Directory.Exists(testPath + "LastRun\\error"))
                {
                    System.IO.Directory.CreateDirectory(testPath + "LastRun\\error");
                }

                foreach (var path in filePaths)
                {
                    foreach (int noOfThread in noOfThreads)
                    {
                        Console.WriteLine(path + ":" + noOfThread.ToString());
                    }
                }

                foreach (var path in filePaths)
                {
                    foreach (int noOfThread in noOfThreads)
                    {
                        Console.WriteLine(path + noOfThread.ToString());
                        mtr.RunMultipleQueries(noOfThread, path);
                        GC.Collect();
                    }
                }

            }
        }

        class MultiThreadRunQuerySaveResults
        {
            static string testPath = ConfigurationManager.AppSettings["TestFilespath"];
            static string OutPutPath = ConfigurationManager.AppSettings["TestFilespath"] + "Templates\\Output\\";
            static string path = testPath + ConfigurationManager.AppSettings["RQLQueryTemplatePath"];

            Queries allQueries = null;

            public string count;
            public QueryResultSet qs;
            public QueriesQuery outQuery;

            static List<QueriesQuery> allOutQuery = new List<QueriesQuery>();

            Queries allOutQueries = new Queries();

            public void ReadRQLQueryTemplateForQueries(string path)
            {
                XmlDocument xmlDoc;
                using (var reader = new XmlTextReader(path)
                {
                    WhitespaceHandling = WhitespaceHandling.None
                })
                {
                    xmlDoc = new XmlDocument();
                    xmlDoc.Load(reader);
                    reader.Close();
                }
                allQueries = (Queries)new XmlSerializer(typeof(Queries)).Deserialize(new MemoryStream(new UTF8Encoding().GetBytes(xmlDoc.InnerXml)));
            }

            public void RunMultipleQueries(int noOfThreads, string path)
            {
                try
                {
                    XmlSerializer ser;
                    FileStream fs;

                    string actualfileName = path.Split('\\')[path.Split('\\').Count() - 1];
                    ReadRQLQueryTemplateForQueries(path);
                    QueriesQuery[] allInQuery = allQueries.Items;

                    int count = 0;
                    Thread[] Threads = null;
                    Threads = new Thread[noOfThreads];
                    int activethreadcount = 0;
                    bool anyactivethreads = true;
                    do
                    {
                        activethreadcount = 0;
                        do
                        {
                            if (null == Threads[activethreadcount])
                            {
                                MultiThreadRunQuerySaveResults runQuerySaveResults = new MultiThreadRunQuerySaveResults();
                                Threads[activethreadcount] = new Thread(runQuerySaveResults.runQuery);
                                runQuerySaveResults.outQuery = allInQuery[count];
                                runQuerySaveResults.count = count.ToString();
                                Threads[activethreadcount].Start();
                                count++;
                            }
                            else if (false == Threads[activethreadcount].IsAlive && count < allInQuery.Count())
                            {
                                MultiThreadRunQuerySaveResults runQuerySaveResults = new MultiThreadRunQuerySaveResults();
                                Threads[activethreadcount] = new Thread(runQuerySaveResults.runQuery);
                                runQuerySaveResults.outQuery = allInQuery[count];
                                runQuerySaveResults.count = count.ToString();
                                Threads[activethreadcount].Start();
                                count++;
                            }
                            activethreadcount++;
                        }
                        while (activethreadcount < noOfThreads);
                        //Thread.Sleep(1000);
                    }
                    while (count < allInQuery.Count());

                    do
                    {
                        anyactivethreads = false;
                        activethreadcount = 0;
                        do
                        {
                            if (Threads[activethreadcount].IsAlive)
                            {
                                anyactivethreads = true;
                            }
                            activethreadcount++;
                        }
                        while (activethreadcount < noOfThreads);
                        //Thread.Sleep(1000);
                    }
                    while (anyactivethreads);


                    allOutQueries.Items = allOutQuery.ToArray();
                    ser = new XmlSerializer(allOutQueries.GetType());
                    fs = File.Open(OutPutPath + noOfThreads.ToString() + "RescoreOutput" + actualfileName + ".xml",
                            FileMode.Create,
                            FileAccess.Write,
                            FileShare.ReadWrite);
                    ser.Serialize(fs, allOutQueries);
                    fs.Close();
                    GC.Collect();

                    //Reset them to 0 to start the new Thread
                    allOutQuery = new List<QueriesQuery>();
                    allOutQueries = new Queries();
                }
                catch (Exception ex)
                {
                    int r = (new Random().Next(0, 100000));
                    Console.WriteLine("Threaded App no:", ex.ToString());
                    string lastrun = testPath + "LastRun\\error\\" + "zzzRunMultipleQueriesErr" + "." + r + ".lastrun";
                    StreamWriter sw = new StreamWriter(lastrun);
                    sw.Write(ex.ToString());
                    sw.Flush();
                    sw.Dispose();
                }
            }



            public void runQuery()
            {
                try
                {
                    outQuery.Status = "Fail";
                    if (null == outQuery.BugId)
                        outQuery.BugId = "Bug ID to be entered";

                    string q = outQuery.RQLQuery;
                    if (q.ToUpper().IndexOf("FOR BROWSE") < 1)
                    {
                        q = q + " For Browse";
                    }

                    //Setting Minimal Paramaters
                    if (null == outQuery.CreatedBy) outQuery.CreatedBy = "igerald";
                    if (null == outQuery.CreationDate) outQuery.CreationDate = DateTime.Now.ToString();
                    if (null == outQuery.OffSet) outQuery.OffSet = "0";
                    if (null == outQuery.PageSize) outQuery.PageSize = "50";
                    if (null == outQuery.MaxLength) outQuery.MaxLength = "5000";
                    if (null == outQuery.Runby) outQuery.Runby = "igerald";
                    if (null == outQuery.RescoreBuild) outQuery.RescoreBuild = "Rescore 1.4.2";
                    ContentServiceClient csvc = new ContentServiceClient();
                    PageSpec pspec = new PageSpec()
                    {
                        Offset = Convert.ToInt32(outQuery.OffSet),
                        PageSize = Convert.ToInt32(outQuery.PageSize)
                    };

                    DateTime starTime = DateTime.Now;
                    qs = csvc.QueryResource(q, pspec);
                    DateTime endTime = DateTime.Now;
                    TimeSpan runTime = endTime.Subtract(starTime);
                    outQuery.PrevRunTime = outQuery.CurrentRunTime;
                    outQuery.CurrentRunTime = runTime.Hours.ToString() + ":" + runTime.Minutes.ToString() + ":" + runTime.Seconds.ToString() + "." + runTime.Milliseconds.ToString();
                    outQuery.LastRunDate = DateTime.Now.ToString();

                    if (null != qs.Results)
                    {
                        outQuery.PrevRunCountField = outQuery.CurrentRunCountField;
                        outQuery.CurrentRunCountField = qs.TotalResultCount.ToString();
                        outQuery.Status = "Pass";
                        outQuery.BugId = "Null";
                        writeResults();
                    }
                    else
                    {
                        outQuery.Status = "NullResult";
                    }
                    Console.WriteLine(outQuery.QueryName + ":" + outQuery.Status + ":" + outQuery.CurrentRunCountField);
                    allOutQuery.Add(outQuery);
                }
                catch (Exception ex)
                {
                    int r = (new Random().Next(0, 100000));
                    Console.WriteLine("Threaded App no:" + ex.ToString());
                    string lastrun = testPath + "LastRun\\error\\" + "zzzrunQueryErr_" + outQuery.QueryName + "." + r + ".lastrun";
                    StreamWriter sw = new StreamWriter(lastrun);
                    sw.Write(ex.ToString());
                    sw.Flush();
                    sw.Dispose();
                    outQuery.Status = "Timed Out";
                    allOutQuery.Add(outQuery);
                }
            }

            public void writeResults()
            {
                try
                {
                    //ResultComparer rs = new ResultComparer();
                    //string s1 = rs.SerializeOnvResultTypes(qs.Results);

                    //int r = (new Random().Next(0, 100000));
                    //string lastrun = testPath + "LastRun\\" + outQuery.QueryName + "." + r + ".lastrun";
                    //WriteFileFromString(lastrun, s1);
                    ////////WriteFileFromString(lastrun + ".reduced", s2);

                    //FileInfo fileInfo = new FileInfo(testPath + "GoodRun\\" + outQuery.QueryName + ".goodrun");
                    //if (!fileInfo.Exists)
                    //{
                    //    WriteFileFromString(testPath + "GoodRun\\" + outQuery.QueryName + ".goodrun", s1);
                    //}
                    //////string good = ReadFileAsString(testPath + "GoodRun\\" + secondQueryName + ".goodrun");
                    //////string good = ReadFileAsString(testPath + "GoodRun\\" + oneQuery.QueryName + ".goodrun");
                }
                catch (Exception ex)
                {
                    int r = (new Random().Next(0, 100000));
                    Console.WriteLine("Threaded App no:", ex.ToString());
                    string lastrun = testPath + "LastRun\\error\\" + "zzztestErrWrite_" + outQuery.QueryName + "." + r + ".lastrun";
                    StreamWriter sw = new StreamWriter(lastrun);
                    sw.Write(ex.ToString());
                    sw.Flush();
                    sw.Dispose();
                }
            }


            public string ReadFileAsString(string filename)
            {
                StreamReader sr = new StreamReader(filename);
                string res = sr.ReadToEnd();
                sr.Dispose();
                return res;
            }

            public void WriteFileFromString(string filename, string data)
            {
                try
                {
                    StreamWriter sw = new StreamWriter(filename);
                    sw.Write(data);
                    sw.Flush();
                    sw.Dispose();
                }
                catch (Exception ex)
                {
                    int r = (new Random().Next(0, 100000));
                    Console.WriteLine("Threaded App no:", ex.ToString());
                    string lastrun = testPath + "LastRun\\error\\" + "zzztestErrWrite_" + outQuery.QueryName + "." + r + ".lastrun";
                    StreamWriter sw = new StreamWriter(lastrun);
                    sw.Write(ex.ToString());
                    sw.Flush();
                    sw.Dispose();
                }
            }

        }
    }

    Thanks,

    Gerald


    - Jerry Test Lead Aditi Tech, Seattle, WA
    Monday, October 24, 2011 11:21 PM

Answers

  • 1) If you compile this application to target x64 instead of x86, it will likely not run out of memory and crash when it reaches 1.5gb.

     

    2) You should consider using Parallel.For/Parallel.ForEach (or at least the ThreadPool) instead of manually handling the threading yourself.  This will dramatically simplify your routine, and potentially fix part of your core problem here.  The framework has a threadpool built-in - and it does a good job of handling this type of scenario on its own.


    Reed Copsey, Jr. - http://reedcopsey.com
    If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
    Tuesday, October 25, 2011 12:06 AM
    Moderator

All replies

  • 1) If you compile this application to target x64 instead of x86, it will likely not run out of memory and crash when it reaches 1.5gb.

     

    2) You should consider using Parallel.For/Parallel.ForEach (or at least the ThreadPool) instead of manually handling the threading yourself.  This will dramatically simplify your routine, and potentially fix part of your core problem here.  The framework has a threadpool built-in - and it does a good job of handling this type of scenario on its own.


    Reed Copsey, Jr. - http://reedcopsey.com
    If a post answers your question, please click "Mark As Answer" on that post and "Mark as Helpful".
    Tuesday, October 25, 2011 12:06 AM
    Moderator
  • Hi Jerry,

    So how is your issue going now with Reed's suggestion?
    Have a nice day,
    Leo Liu [MSFT]
    MSDN Community Support | Feedback to us
    Get or Request Code Sample from Microsoft
    Please remember to mark the replies as answers if they help and unmark them if they provide no help.

    Wednesday, October 26, 2011 3:21 AM
    Moderator