none
Running out of memory RRS feed

  • Question

  • Not sure if this is the right place to post this, but it looks like the most likely candidate, so here goes.

    I have an app that is running that eventually starts to throw System.OutOfMemoryException, however when looking at the task manager My memory usage is less than half of what the machine has.  The machine running this app is dedicated to running ONLY this app.

    What this app does is, it loads up and will scan a specific directory for all the DLL's contained within.  There is a corresponding XML file to each one of these DLL files that tells how often it should be run.  For each DLL file a timer is instantiated.  When the timer elapses for one of these DLL's it will load the assembly into memory, then, using a threadpool it will run it.  When run the DLL will connect to a website and get a webpage, parse it into a XML document where we can XPath the information needed. Once this is finished, the main app will load a dataset with corresponding data and pass it to the assembly.  The assembly will then connect to a website for each row of data in the dataset and compare data against what it has.

    The main app is currently loading about 66 DLL's to run.

    Somewhere along the way, I start getting thrown the out of memory exception and I cannot find out where.  I am beginning to suspect that something is not being collected correctly.

    Any help is appreciated
    -Chris
    Thursday, February 21, 2008 12:00 AM

All replies

  • Its hard to diagnose this without any code.  2 of the most common areas for memory leaks are object not being disposed or closed and event handlers that arent removed which cause objects to hang around in memory.  Either of these could be your issue, or it could be something else.  There are a number of memory profilers out there like the ones from SciTech and Redgate.  I think there might be a free one from MS as well.

     

    Does the stack trace indicate where the memory exception might be coming from?

     

    Thursday, February 21, 2008 2:55 AM
  • These r the diagnostic steps that u can use to find out more about your process -  

    1. Take a process dump  (thru Adplus) and look for objects that are very huge in numbers - see if any of them are out of ordinary range.

    2. Do you use any Winforms or controls that require large number of GDI objects? - Dump would also capture those details -

     

    Later

    -Vani

     

    Thursday, February 21, 2008 11:53 AM
  • OutOfMemoryException and Pinning showed a example a possible cause(memory space slicing) of such kind of issue and the troubleshooting steps. It can help you in this issue.

    And to troubleshoot this issue, we really need the source code and the detailed repro steps to reproduce the problem, so that we can investigate the issue in house. It is not necessary that you send out the complete source of your project. We just need a simplest sample to reproduce the problem. You can remove any confidential information or business logic from it.


    Thanks!
    Friday, February 22, 2008 5:41 AM
  • Here is the service that loads and runs the DLL's

    Code Snippet

    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.Data;
    using System.Diagnostics;
    using System.ServiceProcess;
    using System.Text;
    using System.Timers;
    using Vro.BusinessObject;
    using System.IO;
    using System.Threading;
    using System.Reflection;
    using Vro.ControlLibrary;
    using System.Configuration;
    using Vro.Common;
    using Microsoft.Practices.EnterpriseLibrary.ExceptionHandling;

    namespace MyNameSpace
    {
        public partial class ReplicatorService : ServiceBase
        {
            private static Mutex mut1 = new Mutex();
            private static Mutex mut2 = new Mutex();
            private static List<IPlugin> Assemblies = new List<IPlugin>();
           
            private List<string> accounts = new List<string>(); // Holds the account paths that are currently running - they are removed when not running
            private List<System.Timers.Timer> timers = new List<System.Timers.Timer>();
            private List<FileSystemWatcher> watchers = new List<FileSystemWatcher>();
            private FileSystemWatcher configWatcher = new FileSystemWatcher();

            public ReplicatorService()
            {
                InitializeComponent();

                // Setup Trace Listening to write out the Debug information to a log file
                TextWriterTraceListener tw = new TextWriterTraceListener(ConfigurationManager.AppSettings["LogFile"]);
                Debug.Listeners.Add(tw);
                Debug.AutoFlush = true;

                int maxThreads = 0;
                int maxPortThreads = 0;
                ThreadPool.GetMaxThreads(out maxThreads, out maxPortThreads);
                int newMaxThreads = 50;
                if (int.TryParse(ConfigurationManager.AppSettings["MaxThreads"], out newMaxThreads))
                    if (!ThreadPool.SetMaxThreads(newMaxThreads, maxPortThreads))
                        Debug.WriteLine(DateTime.Now.ToString() + " - FAILED TO SET MAX THREADS - DEFAULTING TO " + newMaxThreads.ToString());
            }

            protected override void OnStart(string[] args)
            {
                Debug.WriteLine(DateTime.Now.ToString() + " - SERVICE STARTED");

                LoadConfiguration();

                // Add a watch to the config file to catch any changes made to it
                FileInfo configFile = new FileInfo(Assembly.GetExecutingAssembly().Location + ".config");
                configWatcher.Path = configFile.DirectoryName;
                configWatcher.Filter = configFile.Name;
                configWatcher.Changed += new FileSystemEventHandler(configWatcher_Changed);
                ((System.ComponentModel.ISupportInitialize)(configWatcher)).BeginInit();
                configWatcher.EnableRaisingEvents = true;
                ((System.ComponentModel.ISupportInitialize)(configWatcher)).EndInit();
            }

            protected override void OnStop()
            {
                // Destroy each timer
                foreach (System.Timers.Timer timer in timers)
                {
                    timer.Enabled = false;
                    timer.Dispose();
                }
                timers.Clear();
                // Destroy each file system watcher
                foreach (FileSystemWatcher watcher in this.watchers)
                {
                    watcher.EnableRaisingEvents = false;
                    watcher.Dispose();
                }
                watchers.Clear();
                // Destroy the config file's file system watcher
                configWatcher.EnableRaisingEvents = false;
                configWatcher.Dispose();

                try
                {
                    // Try to stop each assembly running
                    foreach (IPlugin assembly in Assemblies)
                        assembly.Abort();
                    Assemblies.Clear();

                    // Wait to stop the service until all the current threads stop running
                    int workerThreads = 0;
                    int maxWorkerThreads = 0;
                    int completionPortThreads = 0;
                    int maxCompletionPortThreads = 0;
                    int count = 35;
                    do
                    {
                        ThreadPool.GetAvailableThreads(out workerThreads, out completionPortThreads);
                        ThreadPool.GetMaxThreads(out maxWorkerThreads, out maxCompletionPortThreads);

                        Thread.Sleep(1000);
                    } while (count-- > 0 && workerThreads < maxWorkerThreads - 1); // Max threads minus one to account for the current thread
                }
                catch (Exception ex)
                {
                    ExceptionPolicy.HandleException(ex, Constants.GeneralPolicy);
                }

                Debug.WriteLine(DateTime.Now.ToString() + " - SERVICE STOPPED");
            }

            protected override void OnPause()
            {
                Debug.WriteLine(DateTime.Now.ToString() + " - SERVICE PAUSED");

                base.OnPause();
            }

            protected override void OnContinue()
            {
                Debug.WriteLine(DateTime.Now.ToString() + " - SERVICE CONTINUED");

                base.OnContinue();
            }

            protected void LoadConfiguration()
            {
                try
                {
                    // Clear the current timers and watchers
                    watchers.Clear();
                    timers.Clear();
                    // Add each PMC account declared in the config file
                    PmcAccounts pmcAccounts = PmcAccounts.GetSection();
                    foreach (PmcAccount account in pmcAccounts.Accounts)
                    {
                        if (account.IsPlugin) // For now, just add accounts that are plugins (that is ones that inherit the IPlugin interface)
                        {
                            // Create a file system watcher for the account and add it to the list
                            FileSystemWatcher watcher = new PmcFileSystemWatcher(account);
                            watcher.Created += new FileSystemEventHandler(watcher_Changed);
                            watcher.Changed += new FileSystemEventHandler(watcher_Changed);
                            watcher.Deleted += new FileSystemEventHandler(watcher_Changed);
                            ((System.ComponentModel.ISupportInitialize)(watcher)).BeginInit();
                            watcher.EnableRaisingEvents = true;
                            ((System.ComponentModel.ISupportInitialize)(watcher)).EndInit();
                            watchers.Add(watcher);
                            // Create a timer for the account and add it to the list
                            System.Timers.Timer timer = new PmcTimer(account);
                            (timer as PmcTimer).Elapsed += new PmcElapsedEventHandler(timer_Elapsed);
                            timer.Enabled = account.Enabled && File.Exists((timer as PmcTimer).Account.Path);
                            timers.Add(timer);
                        }
                    }
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(DateTime.Now.ToString() + " - ERROR - " + ex.ToString());

                    if (!System.Diagnostics.EventLog.SourceExists("Replicator"))
                        System.Diagnostics.EventLog.CreateEventSource("Replicator", "Application");
                    System.Diagnostics.EventLog log = new System.Diagnostics.EventLog();
                    log.Source = "Replicator";
                    log.WriteEntry(ex.ToString(), System.Diagnostics.EventLogEntryType.Error);
                    log.Close();

                    ExceptionPolicy.HandleException(ex, Constants.GeneralPolicy);

                    this.OnStop();
                }
            }

            void configWatcher_Changed(object sender, FileSystemEventArgs e)
            {
                this.OnPause();

                LoadConfiguration(); // Reload the configuration to get any changes

                this.OnContinue();
            }

            void watcher_Changed(object sender, FileSystemEventArgs e)
            {
                // Find the timer in the list of timers; if it is found, set its enabled property according to the watcher change type
                PmcTimer timer = (PmcTimer)timers.Find(delegate(System.Timers.Timer t) { return (t as PmcTimer).Account.Path == e.FullPath; });
                if (timer != null)
                    timer.Enabled = e.ChangeType != WatcherChangeTypes.Deleted; // Disable the timer if the file is deleted
            }

            void timer_Elapsed(object sender, PmcElapsedEventArgs e)
            {
                try
                {
                    mut1.WaitOne();
                    // If the assembly doesn't already exist in the list, then proceed to add it to the list and to the thread pool
                    if (!accounts.Contains(e.Account.Path))
                    {
                        accounts.Add(e.Account.Path);
                        WaitCallback callBack = new WaitCallback(RunAssembly);
                        ThreadPool.QueueUserWorkItem(callBack, e.Account.Path + "|" + e.Account.AccountId.ToString());
                    }
                    mut1.ReleaseMutex();
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(DateTime.Now.ToString() + " - ERROR - " + ex.ToString());

                    ex.Data.Add("Assembly Path", e.Account.Path);
                    ex.Data.Add("Account Id", e.Account.AccountId);

                    if (!System.Diagnostics.EventLog.SourceExists("Replicator"))
                        System.Diagnostics.EventLog.CreateEventSource("Replicator", "Application");
                    System.Diagnostics.EventLog log = new System.Diagnostics.EventLog();
                    log.Source = "Replicator";
                    log.WriteEntry(ex.ToString(), System.Diagnostics.EventLogEntryType.Error);
                    log.Close();

                    ExceptionPolicy.HandleException(ex, Constants.GeneralPolicy);
                }
            }

            protected void RunAssembly(object state)
            {
                string path = null;
                int accountId = 0;
                IPlugin assembly = null;

                try
                {
                    string[] states = state.ToString().Split('|');
                    path = states[0];
                    accountId = int.Parse(states[1]);

                    // Load the assembly into memory
                    assembly = LoadAssembly(path);
                    if (assembly != null)
                    {
                        mut2.WaitOne();
                        Assemblies.Add(assembly); // Add the assembly to the list
                        mut2.ReleaseMutex();
                        assembly.Run(accountId); // Run the assembly in memory
                    }
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(DateTime.Now.ToString() + " - ERROR - " + ex.ToString());

                    if (path != null)
                        ex.Data.Add("Assembly Path", path);
                    if (accountId != 0)
                        ex.Data.Add("Account Id", accountId);

                    if (!System.Diagnostics.EventLog.SourceExists("Replicator"))
                        System.Diagnostics.EventLog.CreateEventSource("Replicator", "Application");
                    System.Diagnostics.EventLog log = new System.Diagnostics.EventLog();
                    log.Source = "Replicator";
                    log.WriteEntry(ex.ToString(), System.Diagnostics.EventLogEntryType.Error);
                    log.Close();

                    ExceptionPolicy.HandleException(ex, Constants.GeneralPolicy);
                }
                finally
                {
                    if (assembly != null)
                        assembly.Dispose();

                    mut1.WaitOne();
                    if (path != null && accounts.Contains(path))
                        accounts.Remove(path); // When finished running or upon error, remove the account id from the list
                    mut1.ReleaseMutex();
                    mut2.WaitOne();
                    if (assembly != null && Assemblies.Contains(assembly))
                        Assemblies.Remove(assembly); // When finished running or upon error, remove the assembly from the list
                    mut2.ReleaseMutex();
                }
            }

            protected IPlugin LoadAssembly(string path)
            {
                // If the file doesn't exist, exit
                if (!File.Exists(path))
                    return null;

                try
                {
                    Assembly assembly = null;
                    // Read the file into memory and load the assembly from the memory version
                    using (FileStream fs = File.Open(path, FileMode.Open))
                    {
                        using (MemoryStream ms = new MemoryStream())
                        {
                            byte[] buffer = new byte[1024];
                            int read = 0;
                            while ((read = fs.Read(buffer, 0, 1024)) > 0)
                                ms.Write(buffer, 0, read);
                            assembly = Assembly.Load(ms.ToArray());
                        }
                    }
                    if (assembly != null)
                    {
                        // Find the appropriate type in the assembly, create an instance of it and return it
                        foreach (Type type in assembly.GetExportedTypes())
                        {
                            if (typeof(IPlugin).IsAssignableFrom(type) && !type.IsAbstract)
                            {
                                return (IPlugin)Activator.CreateInstance(type);
                            }
                        }
                    }
                }
                catch (Exception ex)
                {
                    Debug.WriteLine(DateTime.Now.ToString() + " - ERROR - " + ex.ToString());

                    ex.Data.Add("Assembly Path", path);

                    if (!System.Diagnostics.EventLog.SourceExists("Replicator"))
                        System.Diagnostics.EventLog.CreateEventSource("Replicator", "Application");
                    System.Diagnostics.EventLog log = new System.Diagnostics.EventLog();
                    log.Source = "Replicator";
                    log.WriteEntry(ex.ToString(), System.Diagnostics.EventLogEntryType.Error);
                    log.Close();

                    ExceptionPolicy.HandleException(ex, Constants.GeneralPolicy);
                }
                return null;
            }
        }
    }



    Here is an example of the DLL we are loading

    Code Snippet

    using System;
    using System.Collections.Generic;
    using System.Text;
    using Vro.BusinessObject;
    using System.Net;
    using System.Xml;
    using Vro.Common;
    using System.Web;

    namespace Zonder.CalendarAggregators
    {
        public class Scraper : MyScraper
        {
            public Scraper() :
                base("Scraper", "http://www.mywebsite.com/stuff/{0}", null,
                "http://www.mywebsite.com/otherstuff/{0}")
            {
                //this.scrapeForProperties = true;
            }

            /// <summary>
            /// Get the property calendar edit page to obtain page state information.
            /// </summary>
            /// <returns>An array of pmc property ids</returns>
            protected override Dictionary<string, string> GetPropertyIds()
            {
                // Get the request.
                HttpWebRequest request = Utility.GetRequest(this.initialUrl, this.cookies, false, true);

                XmlDocument doc = new XmlDocument();

                // Get the response.
                using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
                {
                    System.IO.StreamReader reader = new System.IO.StreamReader(response.GetResponseStream());
                    string result = reader.ReadToEnd();
                    result = result.Replace("<?xml version=\"1.0\"?>", "");
                    result = result.Replace("<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">", "");
            //Utility simply parses the HTML response into a easy to use XML document
                    doc = Utility.ParseResponse(result, this.parameters);
                }

                // Get the list of propertyIds;
                Dictionary<string, string> propertyIds = new Dictionary<string, string>();
                string ids = "";
                string names = "";

                XmlNodeList properties = doc.SelectNodes(@"//span[contains(@id, '_LblName')]/a");
                foreach (XmlNode property in properties)
                {
                    string id = property.Attributes["href"].Value;
                    id = id.Substring(id.IndexOf("PropertyId=") + 11);
                    id = id.Substring(0, id.IndexOf("&"));

                    ids += id + "\t";
                    string name = property.InnerText.Trim();
                    names += name + "\t";

                    propertyIds.Add(id, name);
                }

                return propertyIds;
            }

          
            protected override DateTime[] GetReservationDays(string propertyId)
            {
                HttpWebRequest request = Utility.GetRequest(String.Format(this.calendarUrl, propertyId), this.cookies, false, true);

                // Get the response.
                List<DateTime> stuffwescrapefor = new List<DateTime>();
                using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
                {
                    XmlDocument doc = Utility.ParseResponse(response, this.parameters);

                    // Scrape for reservations
                    XmlNodeList calendars = doc.DocumentElement.SelectNodes(@"//table[contains(@id, 'stuff')]");
                    foreach (XmlNode calendar in calendars)
                    {
                        string monthYear = calendar.SelectSingleNode(@"./tr/td").InnerText;
                        int month = Utility.GetMonthNumberByName(monthYear.Split(' ')[0]);
                        int year = int.Parse(monthYear.Split(' ')[1]);
                        XmlNodeList days = calendar.SelectNodes(@".//td[@class='Annual']");
                        foreach (XmlNode day in days)
                        {
                            DateTime reservationDay = DateTime.Parse(String.Format("{0}/{1}/{2}", month, day.InnerText, year));
                            stuffwescrapefor.Add(reservationDay);
                        }
                    }
                }
                return stuffwescrapefor.ToArray();
            }
        }
    }


    If needed I can provide an example of the class we inherit from, but it is fairly large (larger than these)

    Friday, February 22, 2008 8:12 PM
  • OOM is always caused by insufficient contiguous virtual address spaces (Virtual Memory). So typically one of following could be the reason:

    1) allocation big chunk of memory (mostly on LOH)

    2) Fragmentation of Heap

    3) memory leak.

    To troubleshoot the issue pleasse follow step -by step mentioned in this article: http://www.codeproject.com/Articles/176031/Out-of-Memory-Exception-A-simple-string-Split-can-.aspx


    Vipin Kumar
    Tuesday, April 5, 2011 8:18 AM
  • Plz check this list... Are you closing your filestream and file you are using in your code.... please check this is important ...

    (XmlDocument has problems in .net cf they will leak memory) Also use RemoveAll() on your xml dom.... doc.RemoveAll(); doc = null;

    Also run a memory profiler it will give very good idea.... (I recommend evaluation version of ants profiler)


    - Rajesh K http://thoughtsontechies.blogspot.com/
    Tuesday, April 5, 2011 9:44 AM