none
Memory usage issues RRS feed

  • Question

  • I created a pretty small WinForms application to just delete files out of a directory that are older than X number of days.  I had this application deleting around a million image files (around 5MB per file) out of a directory and I noticed that the memory usage for the application shown in Task Manger was around 540MB and later jumped up to 580MB while it was deleting the files.  I didn't think much about it while it was processing the files because I figured that was the amount of memory that it needed to store the list of files that needed to be deleted, but that it would clear that up once the deletion processed was finished.  I left the application sitting in memory for a few hours after it completed it's task and the memory the application was using was never released, it was just sitting at 580MB.

    This caught my interest because I have a few server applications that handle a lot of files and they have issues where they eventually start throwing out of memory errors, so I thought that this little application might be an easy way to try to figure out where the memory usage issue is coming from.

    I've tried a few different variations of this small application (changing from lists to arrays, clearing the Lists/Arrays and setting them null before exiting the main function, and tried using a queue object instead of the list/arrays), but it seems like most of the time, the memory used by this is not being released.  Here is the code:

    using System;
    using System.Collections.Generic;
    using System.Data;
    using System.Linq;
    using System.Windows.Forms;
    using System.IO;
    
    namespace DeleteStuff
    {
        public partial class Form1 : Form
        {
            private int fileCount = 0;
            private int curFileNo = 0;
            private bool stopDelete = false;
    
            public Form1()
            {
                InitializeComponent();
            }
    
            private void Form1_Load(object sender, EventArgs e)
            {
    
            }
    
            private void DeleteFilesFromDirectory(DirectoryInfo dir, int dayLimit, bool includeSubDirs)
            {
                DateTime cutoffDate = DateTime.Now.AddDays(-dayLimit);
                List<FileInfo> filesToDelete = dir.GetFiles("*.*").Where(f => f.LastWriteTime <= cutoffDate).ToList();
                fileCount += filesToDelete.Count;
                label3.Text = "Files: " + curFileNo.ToString() + " / " + fileCount.ToString();
                label3.Refresh();
    
                foreach (FileInfo fi in filesToDelete)
                {
                    if (stopDelete) return;
                    curFileNo++;
                    label3.Text = "Files: " + curFileNo.ToString() + " / " + fileCount.ToString();
                    label3.Refresh();
                    Log("Deleting " + fi.FullName + " Last Modified On " + fi.LastWriteTime);
                    try
                    {
                        fi.Delete();
                    }
                    catch (Exception ex)
                    {
                        Log("Failed to delete " + fi.FullName + ", Error: " + ex.Message);
                    }
                }
    
                if (includeSubDirs)
                {
                    List<DirectoryInfo> subDirs = dir.GetDirectories().ToList();
                    foreach (DirectoryInfo di in subDirs)
                    {
                        if (stopDelete) return;
                        Log("Deleting old files from " + di.FullName);
                        DeleteFilesFromDirectory(di, dayLimit, includeSubDirs);
                    }
                }
            }
    
            private void Log(string msg)
            {
                while (listBox1.Items.Count >= 5000)
                {
                    listBox1.Items.RemoveAt(listBox1.Items.Count - 1);
                }
                listBox1.Items.Insert(0, "[" + DateTime.Now.ToString("HH:mm:ss.fff") + "] " + msg);
                listBox1.Refresh();
                Application.DoEvents();
            }
    
            private void button1_Click(object sender, EventArgs e)
            {
                if (button1.Text.Equals("Cancel"))
                {
                    stopDelete = true;
                    return;
                }
                stopDelete = false;
    
                DirectoryInfo dir = new DirectoryInfo(textBox1.Text);
                if (!dir.Exists) return;
                Log("Deleting old files from " + dir.FullName);
                button1.Text = "Cancel";
                fileCount = 0;
                curFileNo = 0;
                Cursor = Cursors.WaitCursor;
                DeleteFilesFromDirectory(dir, Convert.ToInt32(textBox2.Text), checkBox1.Checked);
                Cursor = Cursors.Default;
                button1.Text = "Delete";
                Log("Finished deleting old files");
            }
        }
    }

    The form itself is very basic...  A textbox to enter the path in, a textbox for number of days to keep, a checkbox to include sub folders, a label to show how many files have been processed, a button to start/stop and a listbox to show what was deleted (up to a certain point).  I have also tried not populating the list box.

    I did try adding a GC.Collect() after the delete completes and that does seem to drop the memory usage back down, but that doesn't seem to help my server applications.  

    I'm just wondering if someone could help me understand why the memory usage isn't being cleared once this application finishes it's process.  Any help or thoughts would be appreciated.



    Friday, September 6, 2019 3:26 PM

All replies

  • I left the application sitting in memory for a few hours after it completed it's task

    I do not understand what that means.

    I thought that this little application might be an easy way to try to figure out where the memory usage issue is coming from.

    Probably not. This type of question is asked many times. Windows does not do a cleanup every time there is unused memory. It does a cleanup when it needs more memory. So the fact that tools like the Task Manager shows memory that remains allocated is useless.



    Sam Hobbs
    SimpleSamples.Info

    Friday, September 6, 2019 5:35 PM
  • Probably the problem is caused by large volume of collection returned by dir.GetFiles, and then by another list created by ToList.

    The implementation can be rationalised using a different approach — using EnumerateFiles instead of GetFiles:

    foreach( FileInfo fi in dir.EnumerateFiles( ).Where( f => f.LastWriteTime <= cutoffDate ) )
    {
        . . .
    }

     

    It reads files one-by-one and do not create large lists. However, you do not have an anticipated total count.

    Although, this does not explain the uncleared memory space or the associated false alarm.

    • Edited by Viorel_MVP Friday, September 6, 2019 6:16 PM
    Friday, September 6, 2019 6:10 PM
  • you can use Dispose and finalize methods to release unused objects.
    Friday, September 6, 2019 6:29 PM
  • I left the application sitting in memory for a few hours after it completed it's task

    I do not understand what that means.

    I was just meaning that I left the application running after it finished performing all of the file deletions, so the application was sitting idle for a few hours and the memory it was using wasn't released.  This just caught my attention since I have similar memory usage issues with other applications.


    Monday, September 9, 2019 2:31 PM
  • I figured the Array returned from the .GetFiles and then also converting it to a List is what was consuming the memory to start with (I did try it without converted it to a list and only using the array, but that didn't seem to make much of a difference).  Using the EnumerateFiles is a good Tip, I'll try messing around with that.

    The uncleared memory is what was concerning me though, even when I do force the List object to be cleared, the memory usage from the application doesn't change.  Forcing a GC.Collect() after the process is finished does help, but it seems like the framework should be handling that once the objects fall out of scope.


    Monday, September 9, 2019 2:37 PM
  • List and Array objects don't have Dispose methods though.  I did have an iteration of the code where I perform a .Clear() and set the object to null for the lists, or when I tried it with the arrays, I used the Array.Clear() function to empty the array, then set the array to null and neither of these options helped.
    Monday, September 9, 2019 2:39 PM
  • Hi SteveHamblet,

    Thank you for posting here.

    According to your question, I hope the following references can help you.

    1. C# object array = null, doesn't free memory
    2. How free memory used by a large list in C#?

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Thursday, September 12, 2019 9:35 AM
  • Hi Xingyu,

    Thank you for the additional information, I'll see if either of those help with my issue.

    -Steve

    Friday, September 13, 2019 4:21 PM