none
Fastest way to get size of a directory including all files and sub folders RRS feed

  • Question

  • Hi

    I know there are lots of different methods you can use in VB and .NET to get the size of files and folders, and that some of them especially on larger directories can be quite slow.

    Speed is the priority for me here, so what methods do you recommend to get the total size, and a count of files / folders for a specified location e.g. C:\Users.

    This is to be used in an app I am creating which I hope will act in a similar way to TreeSize etc so that I can sort by size and easily see at a glance which locations are using the most disk space.

    Anything else you need me to clarify then please ask.

    Many thanks.


    Darren Rose

    Thursday, September 17, 2020 6:36 PM

Answers

  • For anyone reading this post I have included below links to all the different methods I tried and sample projects I used etc

    NtQueryDirectoryFile

    From Castorix31 on forum - https://social.msdn.microsoft.com/Forums/vstudio/en-US/91c76b90-805a-45d7-bd0b-ad032dd584fd/fastest-way-to-get-size-of-a-directory-including-all-files-and-sub-folders?forum=vbgeneral 

    My.Computer.FileSystem.GetFiles

    Reference: https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualbasic.fileio.filesystem.getfiles?view=netframework-4.8 

    Sample: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net  

    System.IO.Directory.GetFiles / GetDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directory?view=netframework-4.8 

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net and https://stackoverflow.com/questions/2979432/directory-file-size-calculation-how-to-make-it-faster 

    System.IO.DirectoryInfo.GetFiles / GetDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directoryinfo?view=netframework-4.8

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net and https://www.freevbcode.com/ShowCode.asp?ID=4287 and https://stackoverflow.com/questions/2979432/directory-file-size-calculation-how-to-make-it-faster 

    System.IO.DirectoryInfo.EnumerateFiles / EnumerateDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directoryinfo?view=netframework-4.8

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net  and https://social.msdn.microsoft.com/Forums/en-US/f943818c-e48a-476f-8e45-39230337bba0/function-to-enumerate-filesfolderssize-in-a-given-path?forum=vbgeneral  and https://social.msdn.microsoft.com/Forums/vstudio/en-US/91c76b90-805a-45d7-bd0b-ad032dd584fd/fastest-way-to-get-size-of-a-directory-including-all-files-and-sub-folders?forum=vbgeneral Microsoft Scripting Runtime

    Sample: https://stackoverflow.com/questions/2869561/what-is-the-fastest-way-to-calculate-a-windows-folders-size 

    Custom extension to System.IO.DirectoryInfo.GetFiles / GetDirectories

    https://stackoverflow.com/questions/468119/whats-the-best-way-to-calculate-the-size-of-a-directory-in-net 

    Custom Class using FindFirstFile/FindNextFile (which is what  Directory.GetFiles uses internally but with improvements)

    https://www.codeproject.com/Articles/38959/A-Faster-Directory-Enumerator and latest code https://sourceforge.net/projects/fastfileinfo/

    NTFS MFT

    Article: https://blogs.msmvps.com/bsonnino/2019/03/22/super-fast-file-enumeration-with-ntfs-structures/

    Latest Code: https://github.com/bsonnino/NtfsFileEnum

    IN SUMMARY

    The final method above (NTFS MFT) is most definitely the quickest way to get information about files/folders - using same tests as above it literally takes just seconds to return information on the whole C: drive.

    I also found the custom class mentioned above which uses FindFirstFile/FindNextFile very useful as it returns results quickly and avoids all the usual issues of permissions and errors being thrown, results also quite accurate.

    Failing those I would use Castorix31's NtQueryDirectoryFile method above.

    Hope this summary of resources proves useful for anyone else researching quickest ways to enumerate files/folders etc


    Darren Rose

    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Thursday, September 24, 2020 6:11 PM
  • A test I did (add a Button for the click)

    (I needed Wow64 redirection otherwise I don't get same results as Explorer...)

     =>

    Option Strict On
    Imports System.Runtime.InteropServices
    
    Public Class Form1
    
        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)>
        Public Structure UNICODE_STRING
            Public Length As UShort
            Public MaximumLength As UShort
            <MarshalAs(UnmanagedType.LPWStr)>
            Public Buffer As String
        End Structure
    
        <StructLayout(LayoutKind.Sequential)>
        Public Structure OBJECT_ATTRIBUTES
            Public Length As UInt32
            Public RootDirectory As IntPtr
            'Public ObjectName As UNICODE_STRING
            Public ObjectName As IntPtr
            Public Attributes As UInt32
            Public SecurityDescriptor As IntPtr
            Public SecurityQualityOfService As IntPtr
        End Structure
    
        <StructLayout(LayoutKind.Sequential)>
        Public Structure IO_STATUS_BLOCK
            Public status As UInt32
            Public information As IntPtr
        End Structure
    
        Public Const FILE_LIST_DIRECTORY = &H1
        Public Const SYNCHRONIZE As Integer = &H100000
    
        Public Const FILE_SHARE_READ As Integer = 1
        Public Const FILE_SHARE_WRITE As Integer = 2
        Public Const FILE_SHARE_DELETE As Integer = 4
        Public Const FILE_ATTRIBUTE_NORMAL As Integer = &H80
        Public Const FILE_ATTRIBUTE_DIRECTORY As Integer = &H10
        Public Const FILE_ATTRIBUTE_REPARSE_POINT As Integer = &H400
    
        Public Const FILE_DIRECTORY_FILE = &H1
        Public Const FILE_SYNCHRONOUS_IO_NONALERT = &H20
        Public Const FILE_OPEN_FOR_BACKUP_INTENT = &H4000
    
        Public Const OBJ_CASE_INSENSITIVE = &H40
    
        Public Enum FILE_INFORMATION_CLASS
            FileDirectoryInformation = 1
            FileFullDirectoryInformation   ' 2
            FileBothDirectoryInformation   ' 3
            FileBasicInformation           ' 4  wdm
            FileStandardInformation        ' 5  wdm
            FileInternalInformation        ' 6
            FileEaInformation              ' 7
            FileAccessInformation          ' 8
            FileNameInformation            ' 9
            FileRenameInformation          ' 10
            FileLinkInformation            ' 11
            FileNamesInformation           ' 12
            FileDispositionInformation     ' 13
            FilePositionInformation        ' 14 wdm
            FileFullEaInformation          ' 15
            FileModeInformation            ' 16
            FileAlignmentInformation       ' 17
            FileAllInformation             ' 18
            FileAllocationInformation      ' 19
            FileEndOfFileInformation       ' 20 wdm
            FileAlternateNameInformation   ' 21
            FileStreamInformation          ' 22
            FilePipeInformation            ' 23
            FilePipeLocalInformation       ' 24
            FilePipeRemoteInformation      ' 25
            FileMailslotQueryInformation   ' 26
            FileMailslotSetInformation     ' 27
            FileCompressionInformation     ' 28
            FileObjectIdInformation        ' 29
            FileCompletionInformation      ' 30
            FileMoveClusterInformation     ' 31
            FileQuotaInformation           ' 32
            FileReparsePointInformation    ' 33
            FileNetworkOpenInformation     ' 34
            FileAttributeTagInformation    ' 35
            FileTrackingInformation        ' 36
            FileIdBothDirectoryInformation ' 37
            FileIdFullDirectoryInformation ' 38
            FileValidDataLengthInformation ' 39
            FileShortNameInformation       ' 40
            FileMaximumInformation
        End Enum
    
        <StructLayout(LayoutKind.Explicit)>
        Public Structure LARGE_INTEGER
            <FieldOffset(0)>
            Public LowPart As Integer
            <FieldOffset(4)>
            Public HighPart As Integer
            <FieldOffset(0)>
            Public QuadPart As Long
        End Structure
    
        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)>
        Public Structure FILE_BOTH_DIR_INFORMATION
            Public NextEntryOffset As UInt32
            Public FileIndex As UInt32
            Public CreationTime As LARGE_INTEGER
            Public LastAccessTime As LARGE_INTEGER
            Public LastWriteTime As LARGE_INTEGER
            Public ChangeTime As LARGE_INTEGER
            Public EndOfFile As LARGE_INTEGER
            Public AllocationSize As LARGE_INTEGER
            Public FileAttributes As UInt32
            Public FileNameLength As UInt32
            Public EaSize As UInt32
            Public ShortNameLength As Char
            '<MarshalAs(UnmanagedType.ByValTStr, SizeConst:=12)> Public ShortName As Byte()
            '<MarshalAs(UnmanagedType.ByValTStr, SizeConst:=1)> Public FileName As Byte()
            <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=12)> Public ShortName As String
            <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=260)> Public FileName As String
        End Structure
    
        Public Const STATUS_NO_MORE_FILES = &H80000006
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtOpenFile(ByRef FileHandle As IntPtr, DesiredAccess As Integer, ByRef ObjectAttributes As OBJECT_ATTRIBUTES, ByRef IoStatusBlock As IO_STATUS_BLOCK, ShareAccess As UInt32, OpenOptions As UInt32) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtQueryDirectoryFile(FileHandle As IntPtr, pEvent As IntPtr,  'PIO_APC_ROUTINE        ApcRoutine,
                                                    ApcRoutine As IntPtr, ApcContext As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, FileInformation As IntPtr,
                                                    Length As Integer, FileInformationClass As FILE_INFORMATION_CLASS, ReturnSingleEntry As Boolean, ByRef FileName As UNICODE_STRING, RestartScan As Boolean) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtQueryInformationFile(FileHandle As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, FileInformation As IntPtr, Length As Integer, FileInformationClass As FILE_INFORMATION_CLASS) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function NtClose(ByVal handle As IntPtr) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Sub RtlInitUnicodeString(ByRef DestinationString As UNICODE_STRING, SourceString As IntPtr)
        End Sub
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function IsWow64Process(ByVal hProcess As IntPtr, <Out> ByRef Wow64Process As Boolean) As Boolean
        End Function
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function Wow64DisableWow64FsRedirection(<Out> ByRef OldValue As IntPtr) As Boolean
        End Function
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function Wow64RevertWow64FsRedirection(OldValue As IntPtr) As Boolean
        End Function
    
        Private Function InitializeObjectAttributes(ByVal objectName As UNICODE_STRING, Attributes As UInt32) As OBJECT_ATTRIBUTES
            Dim objectAttributes As OBJECT_ATTRIBUTES = New OBJECT_ATTRIBUTES()
            objectAttributes.RootDirectory = IntPtr.Zero
            'objectAttributes.ObjectName = objectName
            objectAttributes.ObjectName = Marshal.AllocHGlobal(Marshal.SizeOf(objectName))
            Marshal.StructureToPtr(objectName, objectAttributes.ObjectName, False)
            objectAttributes.SecurityDescriptor = IntPtr.Zero
            objectAttributes.SecurityQualityOfService = IntPtr.Zero
            objectAttributes.Attributes = Attributes
            objectAttributes.Length = CUInt(Marshal.SizeOf(objectAttributes))
            Return objectAttributes
        End Function
    
        Public nNbFiles As Integer = 0
        'Public nNbFilesSearch As Integer = 0
        Public nTotalSize As Long = 0
    
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    
        End Sub
    
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            nNbFiles = 0
            nTotalSize = 0
            Dim btd As New System.Threading.Thread(AddressOf BackgroundThread)
            btd.IsBackground = True
            btd.Name = "Background Thread"
            btd.Start()
        End Sub
    
        Private Sub ScanDirectory(sDirectory As String)
            Dim RootDirectoryName As UNICODE_STRING = New UNICODE_STRING()
            Dim sPath As String = sDirectory
            Dim sBuffer As String = String.Format("\??\{0}\", sPath)
            Dim pBuffer As IntPtr = Marshal.StringToHGlobalUni(sBuffer)
            RtlInitUnicodeString(RootDirectoryName, pBuffer)
            Dim RootDirectoryAttributes As OBJECT_ATTRIBUTES = New OBJECT_ATTRIBUTES()
            RootDirectoryAttributes = InitializeObjectAttributes(RootDirectoryName, OBJ_CASE_INSENSITIVE)
    
            Dim IoStatusBlock As IO_STATUS_BLOCK = New IO_STATUS_BLOCK()
            Dim hFile As IntPtr = IntPtr.Zero
            ' &HC000000D STATUS_INVALID_PARAMETER 
            Dim NtStatus As Integer = NtOpenFile(hFile, FILE_LIST_DIRECTORY Or SYNCHRONIZE, RootDirectoryAttributes, IoStatusBlock,
                                                 FILE_SHARE_READ Or FILE_SHARE_WRITE, FILE_DIRECTORY_FILE Or FILE_SYNCHRONOUS_IO_NONALERT Or FILE_OPEN_FOR_BACKUP_INTENT)
            Marshal.FreeHGlobal(RootDirectoryAttributes.ObjectName)
            If (NtStatus = 0) Then
                Dim DirectoryInfo As New FILE_BOTH_DIR_INFORMATION()
                'Dim nBufferSize As Integer = (2 * 260) + Marshal.SizeOf(DirectoryInfo)
                Dim nBufferSize As Integer = Marshal.SizeOf(DirectoryInfo)
                Dim pDirectoryInfo As IntPtr = IntPtr.Zero
                pDirectoryInfo = Marshal.AllocHGlobal(nBufferSize)
                Marshal.StructureToPtr(DirectoryInfo, pDirectoryInfo, False)
                Dim nSize As Integer = Marshal.SizeOf(DirectoryInfo)
                Dim pDirectoryInfoRet As FILE_BOTH_DIR_INFORMATION = New FILE_BOTH_DIR_INFORMATION()
                Dim b As Boolean = True
                While (True)
                    NtStatus = NtQueryDirectoryFile(hFile, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero, IoStatusBlock, pDirectoryInfo, nBufferSize,
                            FILE_INFORMATION_CLASS.FileBothDirectoryInformation, True, Nothing, b)
                    If (NtStatus = STATUS_NO_MORE_FILES) Then
                        NtStatus = 0
                        Exit While
                    End If
                    If (NtStatus <> 0) Then
                        Exit While
                    End If
                    pDirectoryInfoRet = CType(Marshal.PtrToStructure(pDirectoryInfo, GetType(FILE_BOTH_DIR_INFORMATION)), FILE_BOTH_DIR_INFORMATION)
                    b = False
                    Dim sFileName As String = pDirectoryInfoRet.FileName.Substring(0, CInt(pDirectoryInfoRet.FileNameLength / 2))
    
                    If ((pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_DIRECTORY) <> FILE_ATTRIBUTE_DIRECTORY) Then
                        'Console.WriteLine("File: {0}", sFileName)
                        'If sFileName.Contains(".avi") Then
                        '    nNbFilesSearch += 1
                        'End If
                        nNbFiles += 1
                        nTotalSize += pDirectoryInfoRet.EndOfFile.LowPart
                    Else
                        'Console.WriteLine("Directory: {0}", sFileName)
                    End If
                    If ((pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_DIRECTORY) = FILE_ATTRIBUTE_DIRECTORY And
                        String.Compare(sFileName, ".") <> 0 And
                        String.Compare(sFileName, "..") <> 0 And
                        (pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_REPARSE_POINT) <> FILE_ATTRIBUTE_REPARSE_POINT) Then
                        Dim sTempPath As String = String.Format("{0}\{1}", sPath, sFileName)
                        ScanDirectory(sTempPath)
                    End If
                End While
                Marshal.FreeHGlobal(pDirectoryInfo)
                NtClose(hFile)
            End If
        End Sub
    
        Public Sub BackgroundThread()
            Dim bWow64 As Boolean = False
            Dim OldValue As IntPtr = IntPtr.Zero
            IsWow64Process(System.Diagnostics.Process.GetCurrentProcess().Handle, bWow64)
            If (bWow64) Then
                Dim bReturn As Boolean = Wow64DisableWow64FsRedirection(OldValue)
            End If
    
            ScanDirectory("e:\toto")
            ' ScanDirectory("c:\windows\system32")
            ' ScanDirectory("c:\windows")
    
            If (bWow64) Then
                Wow64RevertWow64FsRedirection(OldValue)
            End If
            MessageBox.Show(String.Format("Nb files : {0}{1}Size : {2}", nNbFiles, Environment.NewLine, nTotalSize), "Information", MessageBoxButtons.OK, MessageBoxIcon.Information)
        End Sub
    End Class
    


    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Friday, September 18, 2020 7:13 AM
  • I did some tests with big files and it seems to work with those updates =>

        <StructLayout(LayoutKind.Explicit)>
        Public Structure LARGE_INTEGER
            <FieldOffset(0)>
            Public LowPart As UInteger
            <FieldOffset(4)>
            Public HighPart As UInteger
            <FieldOffset(0)>
            Public QuadPart As Long
        End Structure

    Public nTotalSize As ULong = 0

    nTotalSize += (CULng(pDirectoryInfoRet.EndOfFile.HighPart) << 32) + pDirectoryInfoRet.EndOfFile.LowPart

    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Friday, September 18, 2020 8:15 PM

All replies

  • From my tests, NtQueryDirectoryFile

    was faster than other methods I had tested (on Windows 10)

    Thursday, September 17, 2020 6:56 PM
  • From my tests, NtQueryDirectoryFile

    was faster than other methods I had tested (on Windows 10)

    Thanks, I'll take a look - do you have any VB or C# samples of using it as I can't seem to find much online (only had a quick look however)

    Darren Rose

    Thursday, September 17, 2020 10:04 PM
  • A test I did (add a Button for the click)

    (I needed Wow64 redirection otherwise I don't get same results as Explorer...)

     =>

    Option Strict On
    Imports System.Runtime.InteropServices
    
    Public Class Form1
    
        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)>
        Public Structure UNICODE_STRING
            Public Length As UShort
            Public MaximumLength As UShort
            <MarshalAs(UnmanagedType.LPWStr)>
            Public Buffer As String
        End Structure
    
        <StructLayout(LayoutKind.Sequential)>
        Public Structure OBJECT_ATTRIBUTES
            Public Length As UInt32
            Public RootDirectory As IntPtr
            'Public ObjectName As UNICODE_STRING
            Public ObjectName As IntPtr
            Public Attributes As UInt32
            Public SecurityDescriptor As IntPtr
            Public SecurityQualityOfService As IntPtr
        End Structure
    
        <StructLayout(LayoutKind.Sequential)>
        Public Structure IO_STATUS_BLOCK
            Public status As UInt32
            Public information As IntPtr
        End Structure
    
        Public Const FILE_LIST_DIRECTORY = &H1
        Public Const SYNCHRONIZE As Integer = &H100000
    
        Public Const FILE_SHARE_READ As Integer = 1
        Public Const FILE_SHARE_WRITE As Integer = 2
        Public Const FILE_SHARE_DELETE As Integer = 4
        Public Const FILE_ATTRIBUTE_NORMAL As Integer = &H80
        Public Const FILE_ATTRIBUTE_DIRECTORY As Integer = &H10
        Public Const FILE_ATTRIBUTE_REPARSE_POINT As Integer = &H400
    
        Public Const FILE_DIRECTORY_FILE = &H1
        Public Const FILE_SYNCHRONOUS_IO_NONALERT = &H20
        Public Const FILE_OPEN_FOR_BACKUP_INTENT = &H4000
    
        Public Const OBJ_CASE_INSENSITIVE = &H40
    
        Public Enum FILE_INFORMATION_CLASS
            FileDirectoryInformation = 1
            FileFullDirectoryInformation   ' 2
            FileBothDirectoryInformation   ' 3
            FileBasicInformation           ' 4  wdm
            FileStandardInformation        ' 5  wdm
            FileInternalInformation        ' 6
            FileEaInformation              ' 7
            FileAccessInformation          ' 8
            FileNameInformation            ' 9
            FileRenameInformation          ' 10
            FileLinkInformation            ' 11
            FileNamesInformation           ' 12
            FileDispositionInformation     ' 13
            FilePositionInformation        ' 14 wdm
            FileFullEaInformation          ' 15
            FileModeInformation            ' 16
            FileAlignmentInformation       ' 17
            FileAllInformation             ' 18
            FileAllocationInformation      ' 19
            FileEndOfFileInformation       ' 20 wdm
            FileAlternateNameInformation   ' 21
            FileStreamInformation          ' 22
            FilePipeInformation            ' 23
            FilePipeLocalInformation       ' 24
            FilePipeRemoteInformation      ' 25
            FileMailslotQueryInformation   ' 26
            FileMailslotSetInformation     ' 27
            FileCompressionInformation     ' 28
            FileObjectIdInformation        ' 29
            FileCompletionInformation      ' 30
            FileMoveClusterInformation     ' 31
            FileQuotaInformation           ' 32
            FileReparsePointInformation    ' 33
            FileNetworkOpenInformation     ' 34
            FileAttributeTagInformation    ' 35
            FileTrackingInformation        ' 36
            FileIdBothDirectoryInformation ' 37
            FileIdFullDirectoryInformation ' 38
            FileValidDataLengthInformation ' 39
            FileShortNameInformation       ' 40
            FileMaximumInformation
        End Enum
    
        <StructLayout(LayoutKind.Explicit)>
        Public Structure LARGE_INTEGER
            <FieldOffset(0)>
            Public LowPart As Integer
            <FieldOffset(4)>
            Public HighPart As Integer
            <FieldOffset(0)>
            Public QuadPart As Long
        End Structure
    
        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)>
        Public Structure FILE_BOTH_DIR_INFORMATION
            Public NextEntryOffset As UInt32
            Public FileIndex As UInt32
            Public CreationTime As LARGE_INTEGER
            Public LastAccessTime As LARGE_INTEGER
            Public LastWriteTime As LARGE_INTEGER
            Public ChangeTime As LARGE_INTEGER
            Public EndOfFile As LARGE_INTEGER
            Public AllocationSize As LARGE_INTEGER
            Public FileAttributes As UInt32
            Public FileNameLength As UInt32
            Public EaSize As UInt32
            Public ShortNameLength As Char
            '<MarshalAs(UnmanagedType.ByValTStr, SizeConst:=12)> Public ShortName As Byte()
            '<MarshalAs(UnmanagedType.ByValTStr, SizeConst:=1)> Public FileName As Byte()
            <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=12)> Public ShortName As String
            <MarshalAs(UnmanagedType.ByValTStr, SizeConst:=260)> Public FileName As String
        End Structure
    
        Public Const STATUS_NO_MORE_FILES = &H80000006
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtOpenFile(ByRef FileHandle As IntPtr, DesiredAccess As Integer, ByRef ObjectAttributes As OBJECT_ATTRIBUTES, ByRef IoStatusBlock As IO_STATUS_BLOCK, ShareAccess As UInt32, OpenOptions As UInt32) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtQueryDirectoryFile(FileHandle As IntPtr, pEvent As IntPtr,  'PIO_APC_ROUTINE        ApcRoutine,
                                                    ApcRoutine As IntPtr, ApcContext As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, FileInformation As IntPtr,
                                                    Length As Integer, FileInformationClass As FILE_INFORMATION_CLASS, ReturnSingleEntry As Boolean, ByRef FileName As UNICODE_STRING, RestartScan As Boolean) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Function NtQueryInformationFile(FileHandle As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, FileInformation As IntPtr, Length As Integer, FileInformationClass As FILE_INFORMATION_CLASS) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function NtClose(ByVal handle As IntPtr) As Integer
        End Function
    
        <DllImport("NtDll.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Public Shared Sub RtlInitUnicodeString(ByRef DestinationString As UNICODE_STRING, SourceString As IntPtr)
        End Sub
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function IsWow64Process(ByVal hProcess As IntPtr, <Out> ByRef Wow64Process As Boolean) As Boolean
        End Function
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function Wow64DisableWow64FsRedirection(<Out> ByRef OldValue As IntPtr) As Boolean
        End Function
    
        <DllImport("Kernel32.dll", SetLastError:=True, CharSet:=CharSet.Unicode)>
        Private Shared Function Wow64RevertWow64FsRedirection(OldValue As IntPtr) As Boolean
        End Function
    
        Private Function InitializeObjectAttributes(ByVal objectName As UNICODE_STRING, Attributes As UInt32) As OBJECT_ATTRIBUTES
            Dim objectAttributes As OBJECT_ATTRIBUTES = New OBJECT_ATTRIBUTES()
            objectAttributes.RootDirectory = IntPtr.Zero
            'objectAttributes.ObjectName = objectName
            objectAttributes.ObjectName = Marshal.AllocHGlobal(Marshal.SizeOf(objectName))
            Marshal.StructureToPtr(objectName, objectAttributes.ObjectName, False)
            objectAttributes.SecurityDescriptor = IntPtr.Zero
            objectAttributes.SecurityQualityOfService = IntPtr.Zero
            objectAttributes.Attributes = Attributes
            objectAttributes.Length = CUInt(Marshal.SizeOf(objectAttributes))
            Return objectAttributes
        End Function
    
        Public nNbFiles As Integer = 0
        'Public nNbFilesSearch As Integer = 0
        Public nTotalSize As Long = 0
    
        Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load
    
        End Sub
    
        Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
            nNbFiles = 0
            nTotalSize = 0
            Dim btd As New System.Threading.Thread(AddressOf BackgroundThread)
            btd.IsBackground = True
            btd.Name = "Background Thread"
            btd.Start()
        End Sub
    
        Private Sub ScanDirectory(sDirectory As String)
            Dim RootDirectoryName As UNICODE_STRING = New UNICODE_STRING()
            Dim sPath As String = sDirectory
            Dim sBuffer As String = String.Format("\??\{0}\", sPath)
            Dim pBuffer As IntPtr = Marshal.StringToHGlobalUni(sBuffer)
            RtlInitUnicodeString(RootDirectoryName, pBuffer)
            Dim RootDirectoryAttributes As OBJECT_ATTRIBUTES = New OBJECT_ATTRIBUTES()
            RootDirectoryAttributes = InitializeObjectAttributes(RootDirectoryName, OBJ_CASE_INSENSITIVE)
    
            Dim IoStatusBlock As IO_STATUS_BLOCK = New IO_STATUS_BLOCK()
            Dim hFile As IntPtr = IntPtr.Zero
            ' &HC000000D STATUS_INVALID_PARAMETER 
            Dim NtStatus As Integer = NtOpenFile(hFile, FILE_LIST_DIRECTORY Or SYNCHRONIZE, RootDirectoryAttributes, IoStatusBlock,
                                                 FILE_SHARE_READ Or FILE_SHARE_WRITE, FILE_DIRECTORY_FILE Or FILE_SYNCHRONOUS_IO_NONALERT Or FILE_OPEN_FOR_BACKUP_INTENT)
            Marshal.FreeHGlobal(RootDirectoryAttributes.ObjectName)
            If (NtStatus = 0) Then
                Dim DirectoryInfo As New FILE_BOTH_DIR_INFORMATION()
                'Dim nBufferSize As Integer = (2 * 260) + Marshal.SizeOf(DirectoryInfo)
                Dim nBufferSize As Integer = Marshal.SizeOf(DirectoryInfo)
                Dim pDirectoryInfo As IntPtr = IntPtr.Zero
                pDirectoryInfo = Marshal.AllocHGlobal(nBufferSize)
                Marshal.StructureToPtr(DirectoryInfo, pDirectoryInfo, False)
                Dim nSize As Integer = Marshal.SizeOf(DirectoryInfo)
                Dim pDirectoryInfoRet As FILE_BOTH_DIR_INFORMATION = New FILE_BOTH_DIR_INFORMATION()
                Dim b As Boolean = True
                While (True)
                    NtStatus = NtQueryDirectoryFile(hFile, IntPtr.Zero, IntPtr.Zero, IntPtr.Zero, IoStatusBlock, pDirectoryInfo, nBufferSize,
                            FILE_INFORMATION_CLASS.FileBothDirectoryInformation, True, Nothing, b)
                    If (NtStatus = STATUS_NO_MORE_FILES) Then
                        NtStatus = 0
                        Exit While
                    End If
                    If (NtStatus <> 0) Then
                        Exit While
                    End If
                    pDirectoryInfoRet = CType(Marshal.PtrToStructure(pDirectoryInfo, GetType(FILE_BOTH_DIR_INFORMATION)), FILE_BOTH_DIR_INFORMATION)
                    b = False
                    Dim sFileName As String = pDirectoryInfoRet.FileName.Substring(0, CInt(pDirectoryInfoRet.FileNameLength / 2))
    
                    If ((pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_DIRECTORY) <> FILE_ATTRIBUTE_DIRECTORY) Then
                        'Console.WriteLine("File: {0}", sFileName)
                        'If sFileName.Contains(".avi") Then
                        '    nNbFilesSearch += 1
                        'End If
                        nNbFiles += 1
                        nTotalSize += pDirectoryInfoRet.EndOfFile.LowPart
                    Else
                        'Console.WriteLine("Directory: {0}", sFileName)
                    End If
                    If ((pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_DIRECTORY) = FILE_ATTRIBUTE_DIRECTORY And
                        String.Compare(sFileName, ".") <> 0 And
                        String.Compare(sFileName, "..") <> 0 And
                        (pDirectoryInfoRet.FileAttributes And FILE_ATTRIBUTE_REPARSE_POINT) <> FILE_ATTRIBUTE_REPARSE_POINT) Then
                        Dim sTempPath As String = String.Format("{0}\{1}", sPath, sFileName)
                        ScanDirectory(sTempPath)
                    End If
                End While
                Marshal.FreeHGlobal(pDirectoryInfo)
                NtClose(hFile)
            End If
        End Sub
    
        Public Sub BackgroundThread()
            Dim bWow64 As Boolean = False
            Dim OldValue As IntPtr = IntPtr.Zero
            IsWow64Process(System.Diagnostics.Process.GetCurrentProcess().Handle, bWow64)
            If (bWow64) Then
                Dim bReturn As Boolean = Wow64DisableWow64FsRedirection(OldValue)
            End If
    
            ScanDirectory("e:\toto")
            ' ScanDirectory("c:\windows\system32")
            ' ScanDirectory("c:\windows")
    
            If (bWow64) Then
                Wow64RevertWow64FsRedirection(OldValue)
            End If
            MessageBox.Show(String.Format("Nb files : {0}{1}Size : {2}", nNbFiles, Environment.NewLine, nTotalSize), "Information", MessageBoxButtons.OK, MessageBoxIcon.Information)
        End Sub
    End Class
    


    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Friday, September 18, 2020 7:13 AM
  • @Castorix31 - I got a notification saying you had replied so thank you for the code, but for some reason your reply is not appearing here

    EDIT - very odd - your response appears now after I replied!


    Darren Rose


    • Edited by wingers Friday, September 18, 2020 5:36 PM
    Friday, September 18, 2020 4:16 PM
  • @Castorix31

    I can confirm it definitely seems quick, BUT doesn't seem to be getting the correct sizes for some folders

    For example it returns

    Files : 8
    Size : 2302154

    for one particular folder.  The file count of 8 is correct but the size is actually 34,362,040,522 bytes

    see screenshot and dir output below:-

    E:\VirtualBox>dir /s
     Volume in drive E is DATA
     Volume Serial Number is 6E07-4C93
    
     Directory of E:\VirtualBox
    
    29/07/2020  21:59    <DIR>          .
    29/07/2020  21:59    <DIR>          ..
    21/07/2020  20:07    <DIR>          Windows 10
                   0 File(s)              0 bytes
    
     Directory of E:\VirtualBox\Windows 10
    
    21/07/2020  20:07    <DIR>          .
    21/07/2020  20:07    <DIR>          ..
    21/07/2020  19:14    <DIR>          Logs
    20/02/2019  22:14    <DIR>          Snapshots
    21/07/2020  20:07    34,360,786,944 Windows 10 (64-bit).vdi
    21/07/2020  20:07             5,492 Windows 10.vbox
    24/06/2020  13:54             5,492 Windows 10.vbox-prev
                   3 File(s) 34,360,797,928 bytes
    
     Directory of E:\VirtualBox\Windows 10\Logs
    
    21/07/2020  19:14    <DIR>          .
    21/07/2020  19:14    <DIR>          ..
    21/07/2020  20:07           191,177 VBox.log
    24/06/2020  13:54           171,536 VBox.log.1
    11/06/2020  15:56           215,933 VBox.log.2
    25/03/2020  16:26           168,344 VBox.log.3
    21/07/2020  20:07           495,604 VBoxHardening.log
                   5 File(s)      1,242,594 bytes
    
     Directory of E:\VirtualBox\Windows 10\Snapshots
    
    20/02/2019  22:14    <DIR>          .
    20/02/2019  22:14    <DIR>          ..
                   0 File(s)              0 bytes
    
         Total Files Listed:
                   8 File(s) 34,362,040,522 bytes
                  11 Dir(s)  392,579,096,576 bytes free


    Darren Rose



    • Edited by wingers Friday, September 18, 2020 5:48 PM
    Friday, September 18, 2020 5:41 PM
  • I took the size  from LowPart only, but it should be QuadPart for big files (maybe the formula must be changed....)
    • Edited by Castorix31 Friday, September 18, 2020 6:15 PM
    Friday, September 18, 2020 6:15 PM
  • I took the size  from LowPart only, but it should be QuadPart for big files (maybe the formula must be changed....)

    I tried just changing LowPart to QuadPart on " nTotalSize += pDirectoryInfoRet.EndOfFile.LowPart" but that didn't work, so suspect formula needs changing which I would not have a clue how to do.

    I am now testing various other methods and recording how long they take to do a drive for comparison


    Darren Rose

    Friday, September 18, 2020 6:21 PM
  • I did some tests with big files and it seems to work with those updates =>

        <StructLayout(LayoutKind.Explicit)>
        Public Structure LARGE_INTEGER
            <FieldOffset(0)>
            Public LowPart As UInteger
            <FieldOffset(4)>
            Public HighPart As UInteger
            <FieldOffset(0)>
            Public QuadPart As Long
        End Structure

    Public nTotalSize As ULong = 0

    nTotalSize += (CULng(pDirectoryInfoRet.EndOfFile.HighPart) << 32) + pDirectoryInfoRet.EndOfFile.LowPart

    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Friday, September 18, 2020 8:15 PM
  • @Castorix31

    Thank you - I can confirm that works for me as well on all tests I have tried

    I am still trying various other methods, some of which are slightly quicker, when finished testing I will write up all the details of code used and tests done for others to comment on 

    Thanks again


    Darren Rose

    Friday, September 18, 2020 8:50 PM
  • I notice whilst testing that the Wow64 redirection only really applies if you are checking the Windows drive or the \Windows directory. 

    Otherwise it returns correct results even without it for any other drive or any folder other than \Windows, even if app running as x86, x64 or AnyCPU (with Prefer 32-bit turned on or off).

    But to be expected as discussed in previous posts as there are so many files and folders within C:\Windows you can't even easily access with File Explorer logged on as admin. (e.g. C:\Windows\System32\config)

    I suspect same logic applies to some other system folders like System Volume Information as well which you can't access normally.

    Has been very interesting finding and testing every different method I could find and comparing results on different drives, and with different build settings. Times varied between 4 seconds and 31 seconds to do a data drive containing 100Gb, and between 67 seconds and 10 minutes on my C: (Windows) drive!

    Will type up results tomorrow


    Darren Rose


    • Edited by wingers Friday, September 18, 2020 10:03 PM
    Friday, September 18, 2020 10:00 PM

  • Has been very interesting finding and testing every different method I could find and comparing results on different drives, and with different build settings. Times varied between 4 seconds and 31 seconds to do a data drive containing 100Gb, and between 67 seconds and 10 minutes on my C: (Windows) drive!


    While doing comparative testing of different methods, I hope you are keeping 
    in mind the biasing effect of Windows caching. If you do two tests in succession
    reading the same directory then the second test run will almost certainly appear
    faster. The first test executed usually must read the directory data from disk 
    while the second execution (done immediately or very soon after the first) will
    typically be reading the data from memory - in the Windows cache.

    - Wayne

    Friday, September 18, 2020 11:40 PM
  •  I hope you are keeping in mind the biasing effect of Windows caching.

    Yes I am, rebooting between most tests to ensure accurate results....

    Darren Rose

    Saturday, September 19, 2020 12:31 AM
  • Okay I have spent many hours on testing the above solution and many others

    I have been testing with my E: drive (which just contains data) and also on my C: drive which is my main Windows drive to test how they handle permissions etc

    The test is getting size of all directories in root of each drive tested and showing results on a form

    • NtQueryDirectoryFile (Castorix31's solution above) - works fine on both C: (took 52 seconds) and E: (took 3 seconds)

    • My.Computer.FileSystem.GetFiles - E: drive fine (14 seconds so much slower), C: drive times out with "Managed Debugging Assistant 'ContextSwitchDeadlock'" - with this exception disabled ran for 480 seconds but results incorrect

    • System.IO.Directory.GetFiles / GetDirectories - E: drive fine (took 2 seconds), C: drive also fine and took 40 seconds - just need to handle some System.IO.FileNotFoundException exceptions 

    • System.IO.DirectoryInfo.GetFiles / GetDirectories - E: drive fine (took 1 seconds), C: drive also fine and took 22 seconds - again just need to handle some System.IO.FileNotFoundException exceptions 

    • System.IO.DirectoryInfo.EnumerateFiles / EnumerateDirectories - E: drive fine (took 2 seconds), C: drive failed with "Managed Debugging Assistant 'ContextSwitchDeadlock'", and with it turned off took 46 seconds) but results were unreliable and varied each run, some folders completely missed

    • Microsoft Scripting Runtime (using FileSystemObject.GetFolder) - E: drive fine (took 1 second), C: drive took 11 seconds but results incorrect every time

    • FastFileInfo.GetFiles - E: drive took 2 seconds, C: drive took 41 seconds - all results correct - code from https://sourceforge.net/projects/fastfileinfo/

    I rebooted (or cleared cache) between each test to ensure no caching issues etc.  I also tested running app as admin and not as admin, running as x86, x64, and AnyCPU with both Prefer 32-bit enabled and disabled.  Didn't really make a huge amount of difference to results, obviously with C: drive running as admin allowed it to access some locations it couldn't before, but even then still quite a few areas it can't access (and you can't access manually using File Explorer either so make sense).  The only method where running with Prefer 32-bit enabled or disabled was the NtQueryDirectoryFile but again as expected and as covered/explained in a previous discussion.

    So in summary for me - either NtQueryDirectoryFile or System.IO.Directory/DirectoryInfo.Get... are the best options, latter being slightly quicker

    If anyone has any other ideas to improve performance / accuracy or want to see the code I used for each one just let me know.

    UPDATE: 25/09/20 after using more reliable testing methods and improved code


    Darren Rose




    • Edited by wingers Friday, September 25, 2020 4:37 PM
    Saturday, September 19, 2020 2:47 PM
  • I'd recommend reading/processing the NTFS MFT directly, but this raw DotNET code works just a little bit slower than opening File Explorer, selecting all the directories in C:\, right-clicking and then left-clicking Properties to get a count of all the directories and files in the drive.  You won't get the same result when running as Admin as you do from File Explorer, but I don't believe you can get more accurate without reading the MFT itself.

    I realize that you already mentioned using GetDirectories and GetFiles from System.IO.Directory, but I don't see any code showing how you use those tools, so this is how I use those tools.

    Imports System.Collections.Generic
    
    Public Class Form1
    
        Private dirsCount As Long = 0
        Private filesCount As Long = 0
    
        Private Function GetSubDirs(ByVal path As String) As String()
            Dim r As String() = {}
    
            ' Because of unpredictable permission issues in some directories
            ' Plenty of UnauthorizedAccessExceptions occur in all HDDs even 
            ' in Administrator mode (such as with System Volume Information 
            ' directories)
            Try
                r = System.IO.Directory.GetDirectories(path)
            Catch ex As Exception
            End Try
    
            Return r
        End Function
    
        Private Function GetFiles(ByVal path As String) As String()
            Dim r As String() = {}
    
            ' Because of unpredictable permission issues in some directories
            ' Plenty of UnauthorizedAccessExceptions occur in all HDDs even 
            ' in Administrator mode (such as with System Volume Information 
            ' directories)
            Try
                r = System.IO.Directory.GetFiles(path)
            Catch ex As Exception
            End Try
    
            Return r
        End Function
    
        Private Function GetDirectorySize(ByVal path As String) As Long
            Dim r As Long = 0
    
            Me.dirsCount = 0
            Me.filesCount = 0
    
            Dim dirStack As Stack(Of String) = New Stack(Of String)
            dirStack.Push(path)
    
            While dirStack.Count > 0
                Dim thisdir As String = dirStack.Pop()
    
                ' We can get away with this because the permission-safe functions
                ' will return initialized arrays with 0 values where permissions 
                ' fail
                For Each subdir As String In Me.GetSubDirs(thisdir)
                    dirStack.Push(subdir)
    
                    Me.dirsCount += 1
                Next
    
                ' We can get away with this because the permission-safe functions
                ' will return initialized arrays with 0 values where permissions 
                ' fail
                For Each file As String In Me.GetFiles(thisdir)
                    ' You may never run into this problem, but I do clone/backups and 
                    ' retain strange data paths in my HDDs.  So I run into 
                    ' 'PathTooLongException' if I don't ensure that the FQP is under 
                    ' 260 total characters in Windows
                    If (file.Length < 260) Then
                        Dim fi As System.IO.FileInfo = New System.IO.FileInfo(file)
                        r += fi.Length
    
                        Me.filesCount += 1
                    End If
                Next
            End While
    
            Return r
        End Function
    
    
        Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
            '
            ' I run this code after a fresh reboot of my system.
            '
            ' I wait a good long time after logging in to ensure that all background services and foreground 
            ' applications have a chance to finish initializing.
            '
            ' I'm not cheating by closing everything that isn't in use... Firefox is open to this MSDN thread,
            ' Steam is running, Discord is running, Thunderbird is running, etc
            '
            ' My system is a Ryzen 5 2400G with 64GB of HyperX RAM and all my HDDs are WD Enterprise editions
            ' (the smaller ones are 3GB/s while the larger ones are 6GB/s).
            '
            ' I test through the Visual Studio Hosting Process, so that slows things down.
            '
            ' I test using a build for AnyCPU so that slows things down.
            '
            ' I tried cheating JUST A BIT with threading fu, setting 
            ' System.Threading.Thread.CurrentThread.Priority = Threading.ThreadPriority.Highest
            ' This did NOTHING - not sure why.  I was running in Admin Mode.  I didn't research too hard.
            '
            ' Almost every other program on your system would suffer while this app runs with that thread priority anyway.
            '
    
            ' File path and report output
            Dim path, rpt, logpath As String
            ' Size, time in ms, time in seconds
            Dim sz, tm As Long
            ' Time in seconds
            Dim sc As Double
            ' A high performance timer in DotNET
            Dim wtc As Stopwatch
    
            ' Need to initialize rpt to Empty for ease-of-append later on
            rpt = String.Empty
            For Each d As System.IO.DriveInfo In System.IO.DriveInfo.GetDrives()
                '
                ' C:\ is my Windows drive, 2TB with about 1.5TB free space.
                '
                ' D:\ is my Development respository, 1TB with about 630GB free space
                '
                ' E:\ is a 4TB drive in my system, with less than 1TB free space
                '
                ' G:\ is a 4TB drive for games, with about 2.5TB free space
                '
                ' Testing d.IsReady excludes CDROMS with no media inserted
                If (d.IsReady) Then
                    path = d.RootDirectory.Name
    
                    wtc = Stopwatch.StartNew()
                    sz = Me.GetDirectorySize(path)
                    wtc.Stop()
    
                    tm = wtc.ElapsedMilliseconds
                    sc = (CType(tm, Double) / CType(1000, Double))
    
                    If Not String.IsNullOrEmpty(rpt) Then
                        rpt &= System.Environment.NewLine + System.Environment.NewLine
                    End If
    
                    rpt &= "Scanned:  " & path & System.Environment.NewLine & "Calculated " & sz.ToString("N0") & " total bytes in " & Me.dirsCount.ToString("N0") & " directories and " & Me.filesCount.ToString("N0") & " files in " & tm.ToString("N0") & "ms (" & sc.ToString("N0") & "s)"
                End If
            Next
    
            ' Logging to a file in the application directory
            logpath = System.IO.Path.GetDirectoryName(System.Reflection.Assembly.GetExecutingAssembly().Location) & "\driveScan.log"
            If (System.IO.File.Exists(logpath)) Then
                System.IO.File.Delete(logpath)
            End If
            System.IO.File.WriteAllText(logpath, rpt, System.Text.Encoding.UTF8)
        End Sub
    
    
        Private Sub Form1_FormClosing(ByVal sender As System.Object, ByVal e As System.Windows.Forms.FormClosingEventArgs) Handles MyBase.FormClosing
    
        End Sub
    
    
    End Class

    This log was created by running the program through VS Debug Mode, with VS running As Administrator
    
    Scanned:  C:\
    Calculated 537,454,472,338 total bytes in 199,608 directories and 963,428 files in 697,737ms (697.737s)
    
    Scanned:  D:\
    Calculated 324,456,444,028 total bytes in 41,090 directories and 351,280 files in 143,903ms (143.903s)
    
    Scanned:  E:\
    Calculated 2,901,432,661,892 total bytes in 30,055 directories and 289,301 files in 90,629ms (90.629s)
    
    Scanned:  G:\
    Calculated 1,307,539,758,533 total bytes in 38,040 directories and 672,633 files in 302,230ms (302.23s)

    This log was created by running the EXE directly, As Administrator.
    
    Scanned:  C:\
    Calculated 537,448,700,496 total bytes in 199,614 directories and 964,279 files in 813,535ms (813.535s)
    
    Scanned:  D:\
    Calculated 324,456,444,476 total bytes in 41,090 directories and 351,281 files in 136,617ms (136.617s)
    
    Scanned:  E:\
    Calculated 2,901,432,663,952 total bytes in 30,055 directories and 289,301 files in 80,225ms (80.225s)
    
    Scanned:  G:\
    Calculated 1,307,539,803,545 total bytes in 38,040 directories and 672,633 files in 264,522ms (264.522s)


    Before you can learn anything new you have to learn that there's stuff you don't know.



    Saturday, September 19, 2020 6:44 PM
  • @ Andrew B. Painter

    Thanks for your input, just tried your method and my results are below - comparable to other tests above for E: drive, but slower for my C: drive

    Scanned:  C:\
    Calculated 182,129,035,753 total bytes in 156,766 directories and 630,907 files in 120,349ms (120s)

    Scanned:  E:\
    Calculated 106,532,442,668 total bytes in 10,458 directories and 68,666 files in 4,717ms (5s)

    I am now looking into MFT method as well


    Darren Rose

    Saturday, September 19, 2020 7:32 PM
  • Hi wingers,

    You can try the following code to get size of a directory.

    Class FileCounter
        Private _filesCount As Long
        Public _size As Long
    
        Public Sub Count(ByVal rootPath As String)
            Dim filesEnumerated = New DirectoryInfo(rootPath).EnumerateFiles("*", SearchOption.AllDirectories)
            Parallel.ForEach(filesEnumerated, AddressOf GetFileSize)
        End Sub
    
        Private Sub GetFileSize(ByVal fileInfo As FileInfo)
            Interlocked.Increment(_filesCount)
            Interlocked.Add(_size, fileInfo.Length)
        End Sub
    End Class
    
    ...
    
            Dim fcount = New FileCounter()
            fcount.Count("your file path")
            Console.WriteLine(fcount._size)

    Result of a test.

    Code from:What is the fastest way to calculate a Windows folders size?

    Hope it could be helpful.

    Best Regards,

    Xingyu Zhao 


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, September 21, 2020 9:23 AM
    Moderator
  • @ Xingyu Zhao

    Thanks for your reply - have already tried methods using DirectoryInfo.EnumerateFiles as per my response above

    Tested your code and for my e: drive it took 17 seconds so slower than other methods and for my c: drive it took 2 minutes 30 seconds and results returned for some folders where inaccurate


    Darren Rose

    Monday, September 21, 2020 9:50 AM
  • Hi wingers,

    Did you find the fastest way to get size of a directory here? If so, please click the "Mark as Answer" Link at the bottom of the correct post, or you can share your solution here and mark it as answer. It will help other members to find the solution quickly if they face a similar issue. Thanks.

    Best Regards,

    Xingyu Zhao


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Tuesday, September 22, 2020 5:37 AM
    Moderator
  • @Xingyu Zhao

    Yes I know how it works - when I am ready and finished research and reporting back I will do so


    Darren Rose

    Tuesday, September 22, 2020 9:08 AM
  • For anyone reading this post I have included below links to all the different methods I tried and sample projects I used etc

    NtQueryDirectoryFile

    From Castorix31 on forum - https://social.msdn.microsoft.com/Forums/vstudio/en-US/91c76b90-805a-45d7-bd0b-ad032dd584fd/fastest-way-to-get-size-of-a-directory-including-all-files-and-sub-folders?forum=vbgeneral 

    My.Computer.FileSystem.GetFiles

    Reference: https://docs.microsoft.com/en-us/dotnet/api/microsoft.visualbasic.fileio.filesystem.getfiles?view=netframework-4.8 

    Sample: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net  

    System.IO.Directory.GetFiles / GetDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directory?view=netframework-4.8 

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net and https://stackoverflow.com/questions/2979432/directory-file-size-calculation-how-to-make-it-faster 

    System.IO.DirectoryInfo.GetFiles / GetDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directoryinfo?view=netframework-4.8

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net and https://www.freevbcode.com/ShowCode.asp?ID=4287 and https://stackoverflow.com/questions/2979432/directory-file-size-calculation-how-to-make-it-faster 

    System.IO.DirectoryInfo.EnumerateFiles / EnumerateDirectories

    Reference: https://docs.microsoft.com/en-us/dotnet/api/system.io.directoryinfo?view=netframework-4.8

    Samples: https://stackoverflow.com/questions/3208776/what-s-the-best-way-to-calculate-the-size-of-a-directory-in-vb-net  and https://social.msdn.microsoft.com/Forums/en-US/f943818c-e48a-476f-8e45-39230337bba0/function-to-enumerate-filesfolderssize-in-a-given-path?forum=vbgeneral  and https://social.msdn.microsoft.com/Forums/vstudio/en-US/91c76b90-805a-45d7-bd0b-ad032dd584fd/fastest-way-to-get-size-of-a-directory-including-all-files-and-sub-folders?forum=vbgeneral Microsoft Scripting Runtime

    Sample: https://stackoverflow.com/questions/2869561/what-is-the-fastest-way-to-calculate-a-windows-folders-size 

    Custom extension to System.IO.DirectoryInfo.GetFiles / GetDirectories

    https://stackoverflow.com/questions/468119/whats-the-best-way-to-calculate-the-size-of-a-directory-in-net 

    Custom Class using FindFirstFile/FindNextFile (which is what  Directory.GetFiles uses internally but with improvements)

    https://www.codeproject.com/Articles/38959/A-Faster-Directory-Enumerator and latest code https://sourceforge.net/projects/fastfileinfo/

    NTFS MFT

    Article: https://blogs.msmvps.com/bsonnino/2019/03/22/super-fast-file-enumeration-with-ntfs-structures/

    Latest Code: https://github.com/bsonnino/NtfsFileEnum

    IN SUMMARY

    The final method above (NTFS MFT) is most definitely the quickest way to get information about files/folders - using same tests as above it literally takes just seconds to return information on the whole C: drive.

    I also found the custom class mentioned above which uses FindFirstFile/FindNextFile very useful as it returns results quickly and avoids all the usual issues of permissions and errors being thrown, results also quite accurate.

    Failing those I would use Castorix31's NtQueryDirectoryFile method above.

    Hope this summary of resources proves useful for anyone else researching quickest ways to enumerate files/folders etc


    Darren Rose

    • Marked as answer by wingers Thursday, September 24, 2020 6:11 PM
    Thursday, September 24, 2020 6:11 PM
  • Hi

    Can you post a short VB.NET example for the (NTFS MFT) method. It would finalize the thread nicely.


    Regards Les, Livingston, Scotland

    Thursday, September 24, 2020 9:28 PM
  • Hi

    Can you post a short VB.NET example for the (NTFS MFT) method. It would finalize the thread nicely.


    Regards Les, Livingston, Scotland

    Hi Les

    The code I was using (https://github.com/bsonnino/NtfsFileEnum) is in C# but it converts perfectly to VB using Instant VB (https://www.tangiblesoftwaresolutions.com/product_details/csharp-to-vb-converter.html) which is the converter I always use as it does such a great job and has fantastic support

    Update 27/09/20 - there is also a new version for just getting directory sizes - https://github.com/bsonnino/NtfsDirectories

    There are also articles he wrote explaining it:-

    https://blogs.msmvps.com/bsonnino/2019/03/22/super-fast-file-enumeration-with-ntfs-structures/

    and

    https://blogs.msmvps.com/bsonnino/2020/09/26/discover-what-is-consuming-your-disk-with-ntfs-structures/


    Darren Rose


    • Edited by wingers Sunday, September 27, 2020 2:20 PM more information added
    Friday, September 25, 2020 5:29 PM
  • One issue I am seeing with most of the methods is strange results for C:\Users and C:\ProgramData, oddly enough C:\Windows is normally quite accurate.

    If I right click on C:\Users and C:\ProgramData the sizes shown in Windows are:-

    Users = 21,346,640,651 bytes

    ProgramData = 22,842,810,227 bytes

    But some methods such as DirectoryInfo.GetFiles / GetDirectories are showing much larger results e.g.

    Users = 83,146,397,372 bytes

    ProgramData = 62,035,016,515 bytes

    I can replicate these sizes if from admin command prompt I do DIR /A /S

    I know the sizes can't be correct as if I add up all the folder sizes it reports for my C: drive they add up to a value larger than the actual drive itself!


    Darren Rose


    • Edited by wingers Friday, September 25, 2020 5:36 PM
    Friday, September 25, 2020 5:34 PM
  • Update: found cause of incorrect calculation for C:\ProgramData - it was because I had enabled Windows Sandbox in Windows 10 which had created one or two .vhdx files in C:\ProgramData\Microsoft\Windows\Containers and some methods above were searching contents of the .vhdx files hence returning incorrect results

    Darren Rose

    Saturday, September 26, 2020 11:48 AM
  • Update: found cause of different methods showing different results for C:\Users - it was because some methods were excluding anything with 'FileAttributes.ReparsePoint' and some were not, hence the differences :)

    Darren Rose

    Saturday, September 26, 2020 5:32 PM
  • Hi

    Can you post a short VB.NET example for the (NTFS MFT) method. It would finalize the thread nicely.


    Regards Les, Livingston, Scotland

    Further to my last reply

    If you compile the code from https://sourceforge.net/projects/ntfsreader/ or https://github.com/michaelkc/NtfsReader and add the resulting DLL as a reference to your project then getting information directly from NTFS is as easy as code below:-

    Dim driveToAnalyze As New DriveInfo("c")
    
    Dim ntfsReader As New NtfsReader(driveToAnalyze, RetrieveMode.All)
    
    Dim nodes As IEnumerable(Of INode) = ntfsReader.GetNodes(driveToAnalyze.Name).Where(Function(n) (n.Attributes And (Attributes.ReparsePoint Or Attributes.SparseFile)) = 0)
    
    

    To give you an idea of speed, the above was run on my C: drive which contains 144Gb of data:-

    924.579 MB of volume metadata has been read in 3.951 s at 234.021 MB/s
    807870 nodes have been retrieved in 1145.325 ms


    Darren Rose

    Tuesday, September 29, 2020 9:26 PM