Counting Folders on a Drive

Question

I'm trying to do a quick count of all folders on a drive, including all subdirectories.
the Directory.GetDirectories() won't work as there are some files on my PC which have access denied.  Even though i'm the local admin for the pc, some system folders cause this error.

This function won't skip over this type of error and keep going but will instead crash. 

The alternative is to loop through all the folders recursively and use a try catch to continue with the loop when an access denied error is reached.
But recursion takes too long on a large drive and i need the count back quckly.

Does anyone know a better way to cout the folders in a drive ? 
How come windows explorer does this so fast but the .net libraries and even SystemFileObject take so long?

Any help appreciated.

Answer 1

Hello Erick,

As my understand, the following factors may cause the performance slow,

1.Recursion pattern. Recursion pattern can cause a very deep call stacks which can be very very slow if the directory hierachy is very deep!
2.Try/Catch, throwing or logging an exception. This can also make performance down. (If you do not throw or log when meeting the access denied error, you do not worry about it)
3..NET API GetDirectories, find all directories in a folder and then construct an array consists of the result to return. It does not process a file or a directory one by one. That also makes the application slow.(Note this part is improve in .NET 4.0) As to this point, you can read the following two links,
http://www.eggheadcafe.com/software/aspnet/34259448/enumerating-a-directory.aspx
http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=266943

Based on my knowledge, currently, if we want to get a better performance when doing this, calling Windows API FindFirstFile, FindNextFile is a good option,
http://msdn.microsoft.com/en-us/library/aa364418(VS.85).aspx
http://msdn.microsoft.com/en-us/library/aa364428(VS.85).aspx

Have a nice day!
 

Ji Zhou

MSDN Subscriber Support in Forum

If you have any feedback on our support, please contact [e-mail]

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 2

How fast must it be?  You're only listing the directories.  It shouldn't take more than about 1 second using recursion (after the first when you list the excluded directories).

Answer 3

How fast must it be?  You're only listing the directories.  It shouldn't take more than about 1 second using recursion (after the first when you list the excluded directories).

I agree. GetFiles takes a long time. GetDirectories is very fast. Does it have millions of directories?

---

Ganesh Ranganathan
[Please mark the post as answer if it answers your question]
blog.ganeshzone.net

Answer 4

i have two pc's one at work and one at home.  Both have 4cpu's so they are pretty powerfull.  The recursive folder count at home counts my entire c drive of 11,000 folders in under 1 second.  At work it takes 1 minute to count 64,000 folders.  The work D drive which has about the same as my home C drive takes 9 seconds.  What i found out is that certain folders at work produce an access denied error when getDirectories tries to read them.

For some reason my windows app does not have the same prevledges as I do.  When it hits an access denied error it slows the process down ...a lot.  I'm not sure how this would work with a direct winows api call.  Perhaps the priveledge issue would still persist.

Anyone come accross this before ?

Thanks for everyones help though.

Answer 5

Hello Erick,

Yes, I can understand the performance down when a lot of folders report the Access Denied error. As you say, the windows app does not have the same privilege as you do. Then, we may consider to do a UAC elevation when the application starts,

You can check the example from the codefx samples collection,
http://cfx.codeplex.com/Release/ProjectReleases.aspx?ReleaseId=27931


Thanks,
Ji Zhou

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 6

If you wanted greater control over the directory traversal, you could use P/Invoke to acces FindFirst and FindNext.

Here's some sample code:

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Runtime.InteropServices;
using System.Security.Permissions;
using Microsoft.Win32.SafeHandles;

namespace ConsoleApplication1
{
    static class Program
    {
        static void Main()
        {
            int count = 0;

            foreach (var item in RecursiveFolderContents("C:\\", null, FolderContentType.FoldersOnly))
            {
                ++count;

                if ((count % 1000) == 0)
                {
                    Console.WriteLine(count);
                }
            }

            Console.WriteLine("Number of directories = " + count);
        }

        /// <summary>Used with to specify the type of items to be returned.</summary>

        public enum FolderContentType
        {
            /// <summary>Files and folders returned.</summary>

            FilesAndFolders,

            /// <summary>Only folders returned.</summary>

            FoldersOnly,

            /// <summary>Only files returned.</summary>

            FilesOnly
        }

        /// <summary>
        /// Recursively lists the items in a folder that have a certain suffix (wildcards are not supported).
        /// without having to return them all at once via an array.
        /// This is useful if there are a very large number of files in a folder.
        /// </summary>
        /// <param name="folderName">The name of a folder.</param>
        /// <param name="fileSuffix">
        /// A file suffix to match, or "" or null to match all files.
        /// This is used only for files; if folders are being returned, all folders are returned regardless of their suffixes.
        /// </param>
        /// <param name="contentType">Type of the content to be returned.</param>
        /// <returns>An enumerator for all the items.</returns>

        public static IEnumerable<FolderItem> RecursiveFolderContents
        (
            string            folderName,
            string            fileSuffix,
            FolderContentType contentType
        )
        {
            if (string.IsNullOrEmpty(folderName))
            {
                throw new ArgumentNullException("folderName");
            }

            if (!string.IsNullOrEmpty(fileSuffix))
            {
                if (fileSuffix[0] != '.')
                {
                    fileSuffix = "." + fileSuffix;
                }
            }

            return recursiveFolderContents(folderName, fileSuffix, contentType);
        }

        /// <summary>
        /// Recursively lists the items in a folder that have a certain suffix (wildcards are not supported).
        /// without having to return them all at once via an array.
        /// This is useful if there are a very large number of files in a folder.
        /// </summary>
        /// <param name="folderName">The name of a folder.</param>
        /// <param name="fileSuffix">
        /// A file suffix to match, or "" or null to match all files.
        /// This is used only for files; if folders are being returned, all folders are returned regardless of their suffixes.
        /// </param>
        /// <param name="contentType">Type of the content to be returned.</param>
        /// <returns>An enumerator for all the items.</returns>

        private static IEnumerable<FolderItem> recursiveFolderContents
        (
            string            folderName,
            string            fileSuffix,
            FolderContentType contentType
        )
        {
            bool wantAllFiles = string.IsNullOrEmpty(fileSuffix);

            foreach (FolderItem item in FolderContents(folderName, null, FolderContentType.FilesAndFolders))
            {
                if (item.IsFolder)  // Visit all items in subfolders.
                {
                    foreach (FolderItem recursedItem in recursiveFolderContents(item.Name, fileSuffix, contentType))
                    {
                        yield return recursedItem;
                    }

                    if (contentType != FolderContentType.FilesOnly)
                    {
                        yield return item;
                    }
                }
                else  // It's a file.
                {
                    if (contentType != FolderContentType.FoldersOnly)
                    {
                        if (wantAllFiles || string.Equals(Path.GetExtension(item.Name), fileSuffix, StringComparison.CurrentCultureIgnoreCase))
                        {
                            yield return item;
                        }
                    }
                }
            }
        }

        /// <summary>
        /// Lists the items in a folder that match a specification, without having to return them all at once via an array.
        /// This is useful if there are a very large number of files in a folder.
        /// </summary>
        /// <param name="folderName">The name of a folder.</param>
        /// <param name="itemSpec">
        /// An item spec which may contain wildcards or which can be "", null, "*" or "*.*" for all items.
        /// </param>
        /// <param name="contentType">Type of the content to be returned.</param>
        /// <returns>An enumerator for all the items.</returns>

        public static IEnumerable<FolderItem> FolderContents(string folderName, string itemSpec, FolderContentType contentType)
        {
            if (string.IsNullOrEmpty(folderName))
            {
                throw new ArgumentNullException("folderName");
            }

            if (!Directory.Exists(folderName))
            {
                throw new DirectoryNotFoundException("Directory not found: " + folderName);
            }

            string spec;

            if (string.IsNullOrEmpty(itemSpec))
            {
                spec = Path.Combine(folderName, "*");
            }
            else
            {
                spec = Path.Combine(folderName, itemSpec);
            }

            WIN32_FIND_DATA findData;

            using (SafeFindHandle findHandle = FindFirstFile(spec, out findData))
            {
                if (!findHandle.IsInvalid)
                {
                    do
                    {
                        if ((findData.cFileName != ".") && (findData.cFileName != ".."))  // Ignore special "." and ".." folders.
                        {
                            switch (contentType)
                            {
                                case FolderContentType.FilesAndFolders:
                                {
                                    yield return new FolderItem(findData, folderName);
                                    break;
                                }

                                case FolderContentType.FilesOnly:
                                {
                                    if ((findData.dwFileAttributes & FileAttributes.Directory) == 0)
                                    {
                                        yield return new FolderItem(findData, folderName);
                                    }

                                    break;
                                }

                                case FolderContentType.FoldersOnly:
                                {
                                    if ((findData.dwFileAttributes & FileAttributes.Directory) != 0)
                                    {
                                        yield return new FolderItem(findData, folderName);
                                    }

                                    break;
                                }

                                default:
                                {
                                    throw new ArgumentOutOfRangeException("contentType", contentType, "contentType is not one of the allowed values.");
                                }
                            }
                        }
                    }
                    while (FindNextFile(findHandle, out findData));
                }
                else
                {
                    Debug.WriteLine("Cannot find files in " + spec, "Dmr.Common.IO.FileSystem.FolderContents()");
                }
            }
        }

        /// <summary>Used to safely wrap the handle used by FindFirstFile()/FindNextFile()/FindClose().</summary>

        internal sealed class SafeFindHandle: SafeHandleZeroOrMinusOneIsInvalid
        {
            [SecurityPermission(SecurityAction.LinkDemand, UnmanagedCode=true)]
            public SafeFindHandle()
                : base(true)
            {
                // Nothing to do.
            }

            protected override bool ReleaseHandle()
            {
                if (!IsInvalid && !IsClosed)
                {
                    return FindClose(this);
                }

                return (IsInvalid || IsClosed);
            }

            protected override void Dispose(bool disposing)
            {
                if (!IsInvalid && !IsClosed)
                {
                    FindClose(this);
                }

                base.Dispose(disposing);
            }
        }

        [StructLayout(LayoutKind.Sequential, CharSet=CharSet.Unicode)]
        internal struct WIN32_FIND_DATA
        {
            public FileAttributes dwFileAttributes;
            public FILETIME ftCreationTime;
            public FILETIME ftLastAccessTime;
            public FILETIME ftLastWriteTime;
            public int nFileSizeHigh;
            public int nFileSizeLow;
            public int dwReserved0;
            public int dwReserved1;

            [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_PATH)]
            public string cFileName;

            [MarshalAs(UnmanagedType.ByValTStr, SizeConst=MAX_ALTERNATE)]
            public string cAlternate;
        }

        private const int MAX_PATH = 260;
        private const int MAX_ALTERNATE = 14;

        [DllImport("kernel32.dll", SetLastError=true, CharSet=CharSet.Unicode)]
        private static extern SafeFindHandle FindFirstFile(string lpFileName, out WIN32_FIND_DATA lpFindFileData);

        [DllImport("kernel32.dll", SetLastError=true, CharSet=CharSet.Unicode)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool FindNextFile(SafeHandle hFindFile, out WIN32_FIND_DATA lpFindFileData);

        [DllImport("kernel32.dll", SetLastError=true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool FindClose(SafeHandle hFindFile);
    }

    [StructLayout(LayoutKind.Sequential)]
    internal struct FILETIME
    {
        public uint dwLowDateTime;
        public uint dwHighDateTime;

        public long ToLong()
        {
            return dwLowDateTime + ((long)dwHighDateTime) << 32;
        }
    };

    /// <summary>Information about a file or folder obtained from FolderContents.</summary>

    public class FolderItem
    {

        /// <summary>Constructor.</summary>
        /// <param name="data">Data returned from FindFirstFile()/FindNextFile().</param>
        /// <param name="folder">The folder in which the file resides.</param>

        internal FolderItem(Program.WIN32_FIND_DATA data, string folder)
        {
            _data   = data;
            _folder = folder;
        }

        /// <summary>The full path + filename of the item.</summary>

        public string Name
        {
            get
            {
                return Path.Combine(_folder, _data.cFileName);
            }
        }

        /// <summary>The full path + short 8.3 name of the item.</summary>

        public string ShortName
        {
            get
            {
                return Path.Combine(_folder, _data.cAlternate);
            }
        }

        /// <summary>Creation time of the item.</summary>

        public DateTime CreationTime
        {
            get
            {
                return DateTime.FromFileTime(_data.ftCreationTime.ToLong());
            }
        }

        /// <summary>Last access time of the item.</summary>

        public DateTime LastAccessTime
        {
            get
            {
                return DateTime.FromFileTime(_data.ftLastAccessTime.ToLong());
            }
        }

        /// <summary>Last write time of the item.</summary>

        public DateTime LastWriteTime
        {
            get
            {
                return DateTime.FromFileTime(_data.ftLastWriteTime.ToLong());
            }
        }

        /// <summary>Size of the item. Note: May exceed 32 bits in size.</summary>

        public long Size
        {
            get
            {
                return _data.nFileSizeLow + ((long)_data.nFileSizeHigh) << 32;
            }
        }

        /// <summary>Is the item's archive bit set?</summary>

        public bool IsArchive
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Archive) != 0);
            }
        }

        /// <summary>
        /// Is the item compressed?
        /// For a file, this means that all of the data in the file is compressed.
        /// For a directory, this means that compression is the default for newly created files and subdirectories.
        /// </summary>

        public bool IsCompressed
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Compressed) != 0);
            }
        }

        /// <summary>Reserved; do not use.</summary>

        public bool IsDevice
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Device) != 0);
            }
        }

        /// <summary>Is the item a folder? (If this is false, the item must be a file.)</summary>

        public bool IsFolder
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Directory) != 0);
            }
        }

        /// <summary>
        /// Is the item encrypted?
        /// For a file, this means that all data in the file is encrypted.
        /// For a directory, this means that encryption is the default for newly created files and subdirectories.
        /// </summary>

        public bool IsEncrypted
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Encrypted) != 0);
            }
        }

        /// <summary>Is the item hidden?</summary>

        public bool IsHidden
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Hidden) != 0);
            }
        }

        /// <summary>
        /// Only applicable to files.
        /// Is the file not to be indexed by the content indexing service?
        /// </summary>

        public bool IsNotContentIndexed
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.NotContentIndexed) != 0);
            }
        }

        /// <summary>
        /// Is the file data not available immediately?
        /// This attribute indicates that the file data is physically moved to offline storage.
        /// This attribute is used by Remote Storage, which is the hierarchical storage management software.
        /// </summary>

        public bool IsOffline
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Offline) != 0);
            }
        }

        /// <summary>Is the item read-only?</summary>

        public bool IsReadOnly
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.ReadOnly) != 0);
            }
        }

        /// <summary>Has the item got an associated reparse point?</summary>

        public bool IsReparsePoint
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.ReparsePoint) != 0);
            }
        }

        /// <summary>Only applicable to files. Is it a sparse file?</summary>

        public bool IsSparseFile
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.SparseFile) != 0);
            }
        }

        /// <summary>Is the item part of the operating system, or does the operating system use it exclusively?</summary>

        public bool IsSystem
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.System) != 0);
            }
        }

        /// <summary>
        /// Is The file being used for temporary storage?
        /// File systems attempt to keep all of the data in memory for quick access, rather than flushing it back to mass storage.
        /// </summary>

        public bool IsTemporary
        {
            get
            {
                return ((_data.dwFileAttributes & FileAttributes.Temporary) != 0);
            }
        }

        /// <summary>Only applicable to files. Is it a virtual file?</summary>

        public bool IsVirtual
        {
            get
            {
                return (((int)_data.dwFileAttributes & 0x10000) != 0);
            }
        }

        /// <summary>Convert to string - just returns the item's Name.</summary>
        /// <returns>The item's Name.</returns>

        public override string ToString()
        {
            return this.Name;
        }

        private Program.WIN32_FIND_DATA _data;
        private readonly string _folder;
    }
}

Answer 7

thanks everyone..i should be able to get something working with everyones ideas