none
Reading MFT

    Question

  • Does anyone know how to read the MFT in C#? I have seen a snippet somewhere doing it in C++, but no C#.

    I am trying to read the MFT in order to get a list of SID's for every file on that disk.

    Thanks!
    Friday, June 29, 2007 7:49 PM

Answers

  • I don't think there is any support for reading the MFT (Master File Table) directly from C#, however if you understand the C++ snippets, you might be able to do something with P/Invoke. There is likely some API calls that could return the information your looking for.

    I'd search around http://pinvoke.net
    I know there was a good number of API calls that accepted volume/drive codes to return information about the drive as well as specific calls to get the information.
    Monday, July 02, 2007 7:22 PM

All replies

  • Hi, BottleDude

    Would you please clarify the situation? What is MFT you mentioned?

    Thanks

    Monday, July 02, 2007 6:48 AM
  • I don't think there is any support for reading the MFT (Master File Table) directly from C#, however if you understand the C++ snippets, you might be able to do something with P/Invoke. There is likely some API calls that could return the information your looking for.

    I'd search around http://pinvoke.net
    I know there was a good number of API calls that accepted volume/drive codes to return information about the drive as well as specific calls to get the information.
    Monday, July 02, 2007 7:22 PM
  • Hello BottleDude@discussions.microsoft.com, I'm not sure that specific MFT methods exists for this. Seems that you need to do it manually, starting from the NTFS file structure Read about this there http://www.digit-life.com/articles/ntfs/ --- WBR, Michael Nemtsev [.NET/C# MVP]. My blog: http://spaces.live.com/laflour Team blog: http://devkids.blogspot.com/ "The greatest danger for most of us is not that our aim is too high and we miss it, but that it is too low and we reach it" (c) Michelangelo > Does anyone know how to read the MFT in C#? I have seen a snippet > somewhere doing it in C++, but no C#. > > I am trying to read the MFT in order to get a list of SID's for every > file on that disk. > > Thanks! >
    Friday, July 06, 2007 7:42 AM
  • I was trying to sequentially read files from a volume and grab the SID for each file. I have decided however to use built in functions in C#. The downside to this though is that deleted files will not be read during indexing.
    • Proposed as answer by isaac, Choi Monday, July 28, 2008 4:51 AM
    Saturday, July 07, 2007 12:50 AM
  • I do this all of the time....
    Here is the PInvoke code I wrote:  And below that some sample code I wrote that enumerates the MFT.  That should give you enough to start...

     

    using System;  
    using System.Collections.Generic;  
    using System.Text;  
    using System.Runtime.InteropServices;  
     
    namespace EnumerateVolume  
    {  
        class PInvokeWin32  
        {  
            #region DllImports and Constants  
     
            public const UInt32 GENERIC_READ = 0x80000000;  
            public const UInt32 GENERIC_WRITE = 0x40000000;  
            public const UInt32 FILE_SHARE_READ = 0x00000001;  
            public const UInt32 FILE_SHARE_WRITE = 0x00000002;  
            public const UInt32 FILE_ATTRIBUTE_DIRECTORY = 0x00000010;  
            public const UInt32 OPEN_EXISTING = 3;  
            public const UInt32 FILE_FLAG_BACKUP_SEMANTICS = 0x02000000;  
            public const Int32 INVALID_HANDLE_VALUE = -1;  
            public const UInt32 FSCTL_QUERY_USN_JOURNAL = 0x000900f4;  
            public const UInt32 FSCTL_ENUM_USN_DATA = 0x000900b3;  
            public const UInt32 FSCTL_CREATE_USN_JOURNAL = 0x000900e7;  
     
            [DllImport("kernel32.dll", SetLastError = true)]  
            public static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess,  
                                                      uint dwShareMode, IntPtr lpSecurityAttributes,  
                                                      uint dwCreationDisposition, uint dwFlagsAndAttributes,  
                                                      IntPtr hTemplateFile);  
     
            [DllImport("kernel32.dll", SetLastError = true)]  
            [return: MarshalAs(UnmanagedType.Bool)]  
            public static extern bool GetFileInformationByHandle(IntPtr hFile,  
                                                                         out BY_HANDLE_FILE_INFORMATION lpFileInformation);  
     
            [DllImport("kernel32.dll", SetLastError = true)]  
            [return: MarshalAs(UnmanagedType.Bool)]  
            public static extern bool CloseHandle(IntPtr hObject);  
     
            [DllImport("kernel32.dll", ExactSpelling = trueSetLastError = trueCharSetCharSet = CharSet.Auto)]  
            [return: MarshalAs(UnmanagedType.Bool)]  
            public static extern bool DeviceIoControl(IntPtr hDevice,  
                                                          UInt32 dwIoControlCode,  
                                                          IntPtr lpInBuffer, Int32 nInBufferSize,  
                                                          out USN_JOURNAL_DATA lpOutBuffer, Int32 nOutBufferSize,  
                                                          out uint lpBytesReturned, IntPtr lpOverlapped);  
     
            [DllImport("kernel32.dll", ExactSpelling = trueSetLastError = trueCharSetCharSet = CharSet.Auto)]  
            [return: MarshalAs(UnmanagedType.Bool)]  
            public static extern bool DeviceIoControl(IntPtr hDevice,  
                                                          UInt32 dwIoControlCode,  
                                                          IntPtr lpInBuffer, Int32 nInBufferSize,  
                                                          IntPtr lpOutBuffer, Int32 nOutBufferSize,  
                                                          out uint lpBytesReturned, IntPtr lpOverlapped);  
     
            [DllImport("kernel32.dll")]  
            public static extern void ZeroMemory(IntPtr ptr, Int32 size);  
     
            [StructLayout(LayoutKind.Sequential, Pack = 1)]  
            public struct BY_HANDLE_FILE_INFORMATION  
            {  
                public uint FileAttributes;  
                public FILETIME CreationTime;  
                public FILETIME LastAccessTime;  
                public FILETIME LastWriteTime;  
                public uint VolumeSerialNumber;  
                public uint FileSizeHigh;  
                public uint FileSizeLow;  
                public uint NumberOfLinks;  
                public uint FileIndexHigh;  
                public uint FileIndexLow;  
            }  
     
            [StructLayout(LayoutKind.Sequential, Pack = 1)]  
            public struct FILETIME  
            {  
                public uint DateTimeLow;  
                public uint DateTimeHigh;  
            }  
     
     
            [StructLayout(LayoutKind.Sequential, Pack = 1)]  
            public struct USN_JOURNAL_DATA  
            {  
                public UInt64 UsnJournalID;  
                public Int64 FirstUsn;  
                public Int64 NextUsn;  
                public Int64 LowestValidUsn;  
                public Int64 MaxUsn;  
                public UInt64 MaximumSize;  
                public UInt64 AllocationDelta;  
            }  
     
            [StructLayout(LayoutKind.Sequential, Pack = 1)]  
            public struct MFT_ENUM_DATA  
            {  
                public UInt64 StartFileReferenceNumber;  
                public Int64 LowUsn;  
                public Int64 HighUsn;  
            }  
     
            [StructLayout(LayoutKind.Sequential, Pack = 1)]  
            public struct CREATE_USN_JOURNAL_DATA  
            {  
                public UInt64 MaximumSize;  
                public UInt64 AllocationDelta;  
            }  
     
            public class USN_RECORD  
            {  
                public UInt32 RecordLength;  
                public UInt64 FileReferenceNumber;  
                public UInt64 ParentFileReferenceNumber;  
                public UInt32 FileAttributes;  
                public Int32 FileNameLength;  
                public Int32 FileNameOffset;  
                public string FileName = string.Empty;  
     
                private const int FR_OFFSET = 8;  
                private const int PFR_OFFSET = 16;  
                private const int FA_OFFSET = 52;  
                private const int FNL_OFFSET = 56;  
                private const int FN_OFFSET = 58;  
     
                public USN_RECORD(IntPtr p)  
                {  
                    this.RecordLength = (UInt32)Marshal.ReadInt32(p);  
                    this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET);  
                    this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET);  
                    this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET);  
                    this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET);  
                    this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET);  
                    FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char));  
                }  
            }  
     
            #endregion  
        }  
    }  
     
            public void EnumerateVolume(  
                out Dictionary<UInt64, FileNameAndFrn> files, string[] fileExtensions)  
            {  
                files = new Dictionary<ulong, FileNameAndFrn>();  
                IntPtr medBuffer = IntPtr.Zero;  
                try  
                {  
                    GetRootFrnEntry();  
                    GetRootHandle();  
     
                    CreateChangeJournal();  
     
                    SetupMFT_Enum_DataBuffer(ref medBuffer);  
                    EnumerateFiles(medBuffer, ref files, fileExtensions);  
                }  
                catch (Exception e)  
                {  
                    Log.Info(e.Message, e);  
                    Exception innerException = e.InnerException;  
                    while (innerException != null)  
                    {  
                        Log.Info(innerException.Message, innerException);  
                        innerException = innerException.InnerException;  
                    }  
                    throw new ApplicationException("Error in EnumerateVolume()", e);  
                }  
                finally  
                {  
                    if (_changeJournalRootHandle.ToInt32() != PInvokeWin32.INVALID_HANDLE_VALUE)  
                    {  
                        PInvokeWin32.CloseHandle(_changeJournalRootHandle);  
                    }  
                    if (medBuffer != IntPtr.Zero)  
                    {  
                        Marshal.FreeHGlobal(medBuffer);  
                    }  
                }  
            }  
     
            private void GetRootFrnEntry()  
            {  
                string driveRoot = string.Concat("\\\\.\\", _drive);  
                driveRoot = string.Concat(driveRoot, Path.DirectorySeparatorChar);  
                IntPtr hRoot = PInvokeWin32.CreateFile(driveRoot,   
                    0,  
                    PInvokeWin32.FILE_SHARE_READ | PInvokeWin32.FILE_SHARE_WRITE,   
                    IntPtr.Zero,  
                    PInvokeWin32.OPEN_EXISTING,  
                    PInvokeWin32.FILE_FLAG_BACKUP_SEMANTICS,   
                    IntPtr.Zero);  
     
                if (hRoot.ToInt32() != PInvokeWin32.INVALID_HANDLE_VALUE)  
                {  
                    PInvokeWin32.BY_HANDLE_FILE_INFORMATION fi = new PInvokeWin32.BY_HANDLE_FILE_INFORMATION();  
                    bool bRtn = PInvokeWin32.GetFileInformationByHandle(hRoot, out fi);  
                    if (bRtn)  
                    {  
                        UInt64 fileIndexHigh = (UInt64)fi.FileIndexHigh;  
                        UInt64 indexRoot = (fileIndexHigh <32) | fi.FileIndexLow;  
     
                        FileNameAndFrn f = new FileNameAndFrn(driveRoot, 0);  
                        _directories.Add(indexRoot, f);  
                    }  
                    else  
                    {  
                        throw new IOException("GetFileInformationbyHandle() returned invalid handle",  
                            new Win32Exception(Marshal.GetLastWin32Error()));  
                    }  
                    PInvokeWin32.CloseHandle(hRoot);  
                }  
                else  
                {  
                    throw new IOException("Unable to get root frn entry", new Win32Exception(Marshal.GetLastWin32Error()));  
                }  
            }  
     
            private void GetRootHandle()  
            {  
                string vol = string.Concat("\\\\.\\", _drive);  
                _changeJournalRootHandle = PInvokeWin32.CreateFile(vol,  
                     PInvokeWin32.GENERIC_READ | PInvokeWin32.GENERIC_WRITE,  
                     PInvokeWin32.FILE_SHARE_READ | PInvokeWin32.FILE_SHARE_WRITE,  
                     IntPtr.Zero,  
                     PInvokeWin32.OPEN_EXISTING,  
                     0,  
                     IntPtr.Zero);  
                if (_changeJournalRootHandle.ToInt32() == PInvokeWin32.INVALID_HANDLE_VALUE)  
                {  
                    throw new IOException("CreateFile() returned invalid handle",  
                        new Win32Exception(Marshal.GetLastWin32Error()));  
                }  
            }  
     
            unsafe private void EnumerateFiles(IntPtr medBuffer, ref Dictionary<ulong, FileNameAndFrn> files, string[] fileExtensions)  
            {  
                IntPtr pData = Marshal.AllocHGlobal(sizeof(UInt64) + 0x10000);  
                PInvokeWin32.ZeroMemory(pData, sizeof(UInt64) + 0x10000);  
                uint outBytesReturned = 0;  
     
                while (false != PInvokeWin32.DeviceIoControl(_changeJournalRootHandle, PInvokeWin32.FSCTL_ENUM_USN_DATA, medBuffer,  
                                        sizeof(PInvokeWin32.MFT_ENUM_DATA), pData, sizeof(UInt64) + 0x10000, out outBytesReturned,  
                                        IntPtr.Zero))  
                {  
                    IntPtr pUsnRecord = new IntPtr(pData.ToInt32() + sizeof(Int64));  
                    while (outBytesReturned > 60)  
                    {  
                        PInvokeWin32.USN_RECORD usn = new PInvokeWin32.USN_RECORD(pUsnRecord);  
                        if (0 != (usn.FileAttributes & PInvokeWin32.FILE_ATTRIBUTE_DIRECTORY))  
                        {  
                            //  
                            // handle directories  
                            //  
                            if (!_directories.ContainsKey(usn.FileReferenceNumber))  
                            {  
                                _directories.Add(usn.FileReferenceNumber,  
                                    new FileNameAndFrn(usn.FileName, usn.ParentFileReferenceNumber));  
                            }  
                            else  
                            {   // this is debug code and should be removed when we are certain that  
                                // duplicate frn's don't exist on a given drive.  To date, this exception has  
                                // never been thrown.  Removing this code improves performance....  
                                throw new Exception(string.Format("Duplicate FRN: {0} for {1}",   
                                    usn.FileReferenceNumber, usn.FileName));  
                            }  
                        }  
                        else  
                        {  
                            //   
                            // handle files  
                            //  
                            bool add = true;  
                            if (fileExtensions != null && fileExtensions.Length != 0)  
                            {  
                                add = false;  
                                string s = Path.GetExtension(usn.FileName);  
                                foreach (string extension in fileExtensions)  
                                {  
                                    if (0 == string.Compare(s, extension, true))  
                                    {  
                                        add = true;  
                                        break;  
                                    }  
                                }  
                            }  
                            if (add)  
                            {  
                                if (!files.ContainsKey(usn.FileReferenceNumber))  
                                {  
                                    files.Add(usn.FileReferenceNumber,  
                                        new FileNameAndFrn(usn.FileName, usn.ParentFileReferenceNumber));  
                                }  
                                else  
                                {  
                                    FileNameAndFrn frn = files[usn.FileReferenceNumber];  
                                    if (0 != string.Compare(usn.FileName, frn.Name, true))  
                                    {  
                                        Log.InfoFormat(  
                                            "Attempt to add duplicate file reference number: {0} for file {1}, file from index {2}",  
                                            usn.FileReferenceNumber, usn.FileName, frn.Name);  
                                        throw new Exception(string.Format("Duplicate FRN: {0} for {1}",  
                                            usn.FileReferenceNumber, usn.FileName));  
                                    }  
                                }  
                            }  
                        }  
                        pUsnRecord = new IntPtr(pUsnRecord.ToInt32() + usn.RecordLength);  
                        outBytesReturned -usn.RecordLength;  
                    }  
                    Marshal.WriteInt64(medBuffer, Marshal.ReadInt64(pData, 0));  
                }  
                Marshal.FreeHGlobal(pData);  
            }  
     

     

    • Proposed as answer by StCroixSkipper Wednesday, July 30, 2008 5:41 PM
    • Edited by StCroixSkipper Monday, August 11, 2008 7:19 PM corrected an error in the code.
    Thursday, July 10, 2008 5:31 PM
  • In my tests, reading the mft is about 10x faster then enumerating through the file system with Find First, Find Next...
    • Proposed as answer by isaac, Choi Monday, July 28, 2008 4:53 AM
    Thursday, July 10, 2008 5:33 PM
  • This is something I have been looking for quite a while!  Quick question: FileNameAndFrn... I cannot seem to get the C# compiler to recognize that (seems to be used as a datatype in your sample).  Could you educate me on how to declare and work with it?  Thanks!!
    • Proposed as answer by StCroixSkipper Wednesday, July 23, 2008 6:20 PM
    Sunday, July 20, 2008 8:53 PM
  •  I did omit a simple class, here it is.  Hope this helps.  If not ask another question... 
    p.s.  Sorry about the formatting.  The Add Code didn't format it very well this time...

    public class FileNameAndFrn  

    {  
    #region Properties  
    private string _name;  
    public string Name  
    {  
    get { return _name; }  
    set { _name = value; }  
    }  
     
    private UInt64 _parentFrn;  
    public UInt64 ParentFrn  
    {  
    get { return _parentFrn; }  
    set { _parentFrn = value; }  
    }  
    #endregion  
     
    #region Constructor  
    public FileNameAndFrn(string name, UInt64 parentFrn)  
    {  
    if (name != null && name.Length > 0)  
    {  
    _name = name;  
    }  
    else  
    {  
    throw new ArgumentException("Invalid argument: null or Length = zero", "name");  
    }  
    if (!(parentFrn < 0))  
    {  
    _parentFrn = parentFrn;  
    }  
    else  
    {  
    throw new ArgumentException("Invalid argument: less than zero", "parentFrn");  
    }  
    }  
    #endregion  
    }  
     
    Wednesday, July 23, 2008 6:33 PM
  • Got the class implemented but then the errors really shot up:
    Error 1 The name 'CreateChangeJournal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 29 17 MFT
    Error 2 The name 'SetupMFT_Enum_DataBuffer' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 31 17 MFT
    Error 3 The name 'Log' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 36 17 MFT
    Error 4 The name 'Log' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 40 21 MFT
    Error 5 The name 'innerExceptioninnerException' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 41 21 MFT
    Error 6 The name '_changeJournalRootHandle' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 47 21 MFT
    Error 7 The name '_changeJournalRootHandle' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 49 46 MFT
    Error 8 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 53 21 MFT
    Error 9 The name '_drive' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 63 57 MFT
    Error 10 The name 'Path' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 64 50 MFT
    Error 11 The name '_directories' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 83 21 MFT
    Error 12 The type or namespace name 'IOException' could not be found (are you missing a using directive or an assembly reference?) C:\a\source\C#\MFT\Form1.cs 87 31 MFT
    Error 13 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 88 44 MFT
    Error 14 The type or namespace name 'IOException' could not be found (are you missing a using directive or an assembly reference?) C:\a\source\C#\MFT\Form1.cs 94 27 MFT
    Error 15 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 94 90 MFT
    Error 16 The name '_drive' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 100 51 MFT
    Error 17 The name '_changeJournalRootHandle' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 101 13 MFT
    Error 18 The name '_changeJournalRootHandle' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 108 17 MFT
    Error 19 The type or namespace name 'IOException' could not be found (are you missing a using directive or an assembly reference?) C:\a\source\C#\MFT\Form1.cs 110 27 MFT
    Error 20 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 111 40 MFT
    Error 21 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 118 28 MFT
    Error 22 The name '_changeJournalRootHandle' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 122 58 MFT
    Error 23 The name '_directories' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 135 30 MFT
    Error 24 The name '_directories' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 137 29 MFT
    Error 25 The name 'Path' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 157 40 MFT
    Error 26 The name 'Log' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 179 37 MFT
    Error 27 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 191 17 MFT
    Error 28 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 191 47 MFT
    Error 29 The name 'Marshal' does not exist in the current context C:\a\source\C#\MFT\Form1.cs 193 13 MFT
    Error 30 'System.Runtime.InteropServices.DllImportAttribute' does not contain a definition for 'CharSetCharSet' C:\a\source\C#\MFT\PInvokeWin32.cs 39 79 MFT
    Error 31 'System.Runtime.InteropServices.DllImportAttribute' does not contain a definition for 'CharSetCharSet' C:\a\source\C#\MFT\PInvokeWin32.cs 47 79 MFT
    Sunday, July 27, 2008 9:03 PM
  • You can create a change journal any way you like...  That is the easy part.
    Here is some sample code.

    As for the log statements, simply convert them to Console.WriteLine()...  I use Log4Net, hence the log errors.

    All you need to do is build your own class.  There are some class member variables to define, etc but you should be able to figure out what they are by the code segments.

            unsafe private void CreateChangeJournal()   
            {  
                // This function creates a journal on the volume. If a journal already  
                // exists this function will adjust the MaximumSize and AllocationDelta  
                // parameters of the journal  
                UInt64 MaximumSize = 0x800000;  
                UInt64 AllocationDelta = 0x100000;  
                UInt32 cb;  
                PInvokeWin32.CREATE_USN_JOURNAL_DATA cujd;  
                cujd.MaximumSize = MaximumSize;  
                cujd.AllocationDelta = AllocationDelta;  
     
                int sizeCujd = Marshal.SizeOf(cujd);  
                IntPtr cujdBuffer = Marshal.AllocHGlobal(sizeCujd);  
                PInvokeWin32.ZeroMemory(cujdBuffer, sizeCujd);  
                Marshal.StructureToPtr(cujd, cujdBuffer, true);  
     
                bool fOk = PInvokeWin32.DeviceIoControl(_changeJournalRootHandle, PInvokeWin32.FSCTL_CREATE_USN_JOURNAL,  
                    cujdBuffer, sizeCujd, IntPtr.Zero, 0, out cb, IntPtr.Zero);  
                if (!fOk)  
                {  
                    throw new IOException("DeviceIoControl() returned false", new Win32Exception(Marshal.GetLastWin32Error()));  
                }  
            }  
     
            unsafe private void SetupMFT_Enum_DataBuffer(ref IntPtr medBuffer)  
            {  
                uint bytesReturned = 0;  
                PInvokeWin32.USN_JOURNAL_DATA ujd = new PInvokeWin32.USN_JOURNAL_DATA();  
     
                bool bOk = PInvokeWin32.DeviceIoControl(_changeJournalRootHandle,                           // Handle to drive  
                    PInvokeWin32.FSCTL_QUERY_USN_JOURNAL,   // IO Control Code  
                    IntPtr.Zero,                // In Buffer  
                    0,                          // In Buffer Size  
                    out ujd,                    // Out Buffer  
                    sizeof(PInvokeWin32.USN_JOURNAL_DATA),  // Size Of Out Buffer  
                    out bytesReturned,          // Bytes Returned  
                    IntPtr.Zero);               // lpOverlapped  
                if (bOk)  
                {  
                    PInvokeWin32.MFT_ENUM_DATA med;  
                    med.StartFileReferenceNumber = 0;  
                    med.LowUsn = 0;  
                    med.HighUsn = ujd.NextUsn;  
                    int sizeMftEnumData = Marshal.SizeOf(med);  
                    medBuffer = Marshal.AllocHGlobal(sizeMftEnumData);  
                    PInvokeWin32.ZeroMemory(medBuffer, sizeMftEnumData);  
                    Marshal.StructureToPtr(med, medBuffer, true);  
                }  
                else  
                {  
                    throw new IOException("DeviceIoControl() returned false", new Win32Exception(Marshal.GetLastWin32Error()));  
                }  
            }  
     
     
    Tuesday, July 29, 2008 9:06 PM
  •  Hi StCroixSkipper,

    I tried to compile your code and ran into some problems. Basically I have three variables left undefined. They are:

    _changeJournalRootHandle


    _drive

    _directories

    I can make an educated guess as to how to handle _drive and _directories but with the first one it is more difficult.

    There was also a typo I corrected. This block apparently should look like this:

                    while ( innerException != null )  
                    {  
                        Console.WriteLine ( innerException.Message, innerException );   
                        innerExceptioninnerException = innerException.InnerException;  
                    } 

    I would appreciate your help. Thanks.
    AlexB
    Saturday, August 09, 2008 7:09 PM
  • First, I disagree with your edit...
                    Log.Info(e.Message, e);  
                    Exception innerException = e.InnerException;  

                    while (innerException != null)  
                    {  
                        Log.Info(innerException.Message, innerException);  
                        innerException innerException = innerException.InnerException;  
                    }  
    In .net, each Exception object has an inner exception.  The inner exception object is either another Exception object or null. This code should iterate through all of these nested exceptions until it finds a null inner exception.  The code you editted shouldn't compile...
    There is no type 'innerException' and if there were you couldn't have a variable with a type name..  So I don't understand your edit.

    This code should be:
                    Log.Info(e.Message, e);  
                    Exception innerException = e.InnerException;  

                    while (innerException != null)  
                    {  
                        Log.Info(innerException.Message, innerException);  
                        innerException = innerException.InnerException;  
                    }  
    as it was before your edit!
    I think we shouldn't be editing one another's code.  If you want to make suggestions for edits, fine but I'll edit my own code thanks.  The code you've edited makes no sense and now it has my name on it...


    As for the undefined variables...  You can use whatever you like that works. 

    _directories is a collection of each directory's 64-bit file reference number associated with the directory's  "FileNameAndFrn" object (which should have been called "FileNameAndParentFrn" for clarity which it is in my code. The code I posted was ancient personal code.) I used a dictionary.

    _drive is simply

    public string Drive

    {

    get { return _drive; }

    }

    Hard to figure that one out.


    and _changeJournalRootHandle is

    private IntPtr _changeJournalRootHandle;


    Which I thought was pretty simple to figure out because
            private void GetRootHandle()  
            {  
                string vol = string.Concat("\\\\.\\", _drive);  
                _changeJournalRootHandle = PInvokeWin32.CreateFile(vol,  
                     PInvokeWin32.GENERIC_READ | PInvokeWin32.GENERIC_WRITE,  
                     PInvokeWin32.FILE_SHARE_READ | PInvokeWin32.FILE_SHARE_WRITE,  
                     IntPtr.Zero,  
                     PInvokeWin32.OPEN_EXISTING,  
                     0,  
                     IntPtr.Zero);  
                if (_changeJournalRootHandle.ToInt32() == PInvokeWin32.INVALID_HANDLE_VALUE)  
                {  
                    throw new IOException("CreateFile() returned invalid handle",  
                        new Win32Exception(Marshal.GetLastWin32Error()));  
                }  
            }  

    shows _changeJournalRootHandle as the return from PInvokeWin32.CreateFile() which is defined as 
            [DllImport("kernel32.dll", SetLastError = true)]  
            public static extern IntPtr CreateFile(string lpFileName, uint dwDesiredAccess,  
                                                      uint dwShareMode, IntPtr lpSecurityAttributes,  
                                                      uint dwCreationDisposition, uint dwFlagsAndAttributes,  
                                                      IntPtr hTemplateFile);  
    in the code I provided.

    So, I'm a little disconcerted that you simply didn't try to understand the code I provided.  I didn't intend to write a turnkey,  black box object that would automatically work.  I only wanted to provide the ideas that others could build on to accomplish enumerating the files using the MFT in c#.  Explaining at this level wasn't my intent. 


    Monday, August 11, 2008 7:14 PM
  •  I think we shouldn't be editing one another's code

    I agree but it did not compile. I think there was a typo in there. I mean that particular part gave me trouble.  Anyhow, I don't want to argue. Your code is too valuable for me to aggravate you unnecesserily:) It was ALL my fault from the moment one:)

    I also appreciate your coming back and answering my questions. I will have to look into the details (the place where the devil is, as we know:) and try to compile your code again.

    I also appreciate your explanations. I am sure many others will benefit form your code.

    Thanks much.

    AlexB
    Monday, August 11, 2008 11:21 PM
  •  So, I'm a little disconcerted that you simply didn't try to understand the code I provided.

    Well, this is a dose of Sunday sermon so to speak. Of course I will try to understand your code but the starting point is always to have a code to compile. Otherwise nobody can have any confidence in it or any other code sample. If I post a sample, and I do it a few times daily, I always either make sure that it is absolutely ready and compiles or I will follow up with the person to make sure that all issues are resolved. This is my standard.

    _directories is a collection of each directory's 64-bit file reference number associated with the directory's 

    What do you mean by that? Is it a hash code for each directory? It is too nebulious to say that it is a collection of numbers associated with "FileNameAndFrn" object. I must admit I do lack understanding of your object but the naked fact is that the _directories appears in the code UNDEFINED and caused a compile error. _drive ls the same way. No explanation can help here. There must be a definition for each, preceeding the usage. It is a law in C# and .NET in general. We are not talking about java or vb scripts.  The definition was not there in the first palce. I will see if you introduced it at this stage.

    Ancient or not, it is YOUR code, you posted it, offered it to the public and you either must help people to figure out what it is all about or disavow it and back off right away but to say it clearly. We don't need morality lessons around here. Sorry.

    I hope we will make do with it.

    Thanks.

    AlexB
    Monday, August 11, 2008 11:39 PM
  • OK, Skipper, we are in business. your code compiled. I am glad that I "thrashed" it out of you. It is also a great time saving device. With my IQ of only 45 it would have been hard and long to figure it all out.

    Of course, one day I may take a closer look at your code which I alwasy do and perhaps make some changes. It is a natural process but I prefer to start with a working sample. That day may actually be this week.

    You are a great man for writing that. I hope it will work. I may come back in a day or two asking more stupid questions. Please remember me:)

    Thanks.

    AlexB
    Monday, August 11, 2008 11:47 PM
  •  Code snippets versus compilable code:  This is the last, I promise.
    My intent was to provide code snippets, not compilable code. And I really think that is the more valuable option.  Code snippets provide insight into clever and interesting ways of doing things that may be unfamiliar to others.  If I see a code snippet, I can study it and gain insights how I might make use of a technique to idea.  True, I can do the same with a compilable, executable sample, but my thought process is different.  I prefer sections of code that demonstrate a strategy, technique or idea.

    As for contributions to MSDN forums, I want to encourage others to contribute their code snippets so everyone benefits.  The last thing I want to do discourage contributions.

    I'm not trying to offend anyone but let me raise a couple of examples here in our little thread. 

    One user asked the original participant what he meant by MFT.  Can we not expect of each other that if we didn't know what an MFT was that they would try a simple search like "Microsoft windows MFT" and read through some of the hits. 

    In a like vein, how much effort would it take to determine the types of some of the variables not defined in the code snippets that I provided.  I'm a developer, I write code because I like to and because I get paid to write code.  I have benefited greatly from the ideas of others who have contributed to forums like this.  I think we need to raise the level interaction, the level of the questions, so they reflect a little effort on the part of the participants.  To ask a question, the answer for which we could easily determine for ourselves, diminishes us all.

    I would like someone to take my code snippets and suggest improvements.  I may agree or challenge.  But we would all benefit from the exercise.

    Thanks for the opportunity to express these thoughts.
    • Edited by StCroixSkipper Wednesday, August 13, 2008 11:13 PM spelling error
    Wednesday, August 13, 2008 11:11 PM
  • Thanks for the opportunity to express these thoughts

    You definitely have a knack for doing it.

    There are three things I want to say. #1: I am definitely greatful for the code, #2: I was a bit in a haste to declare that the code compiled. It did not. What happened, being always in a hurry, I forgot to switch the file from "Exclude from Project" to "Include In Project" after I made the changes. However I am not really worried about it. I will definitely compile it later. At this moment I am too busy with other things to finish. What I've got was an assurance that those types are not coming from some class you forgot to post or another serious piece of code. Now I know that it is much simpler.

    And finally, #3. To be honest, I don't like your lecturing. Do what you want but remember, it is your idiosyncratic style. it is a rather peculiar style too. People are different. I for one thing am not interested in deep understanding of your code in detail. It is not my area. it is a utility for me. I feel ruffled that instead of offering help you are forcing me to study a subject I don't need at this point. You goad me and I've got enough on my plate.

    My way of learning is thru practicing, thru using things with as little understanding as possible at the beginning. It saves me time. Understanding usually comes later in bits and pieces. Eventually I will find time to take a look at your code and spend perhaps 30 min analyzing it.

    I post a lot of code for people, the code that serves me well, and I always follow the same principle plus one more: once I posted, I either explain it thoroughly or admit that I do not understand how it works.

    I hope you won't take it as an offense. Thanks again.

    Incidentally, have you ever yourself run it or it is mostly in theoretical and philosophical stage? I am just curious.



    AlexB
    Thursday, August 14, 2008 1:39 PM
  • I've implemented this technique in a couple of commercial products. I've written a photo application which uses it to find all of the 'picture' files on a volume.  I also use the NTFS change journal to identify any new picture files that are added to the volume, with all of the code to ensure that the change journal is valid and revert to a full MFT enumeration if it isn't.  All in C#.

    Jeffrey Cooperstein and Jeffrey Richter wrote an article on the Windows 2000 Change Journal in Sept 1999 that explains the approach in c++

    Here is the link.
    http://technet.microsoft.com/en-us/library/bb742450.aspx
    • Edited by StCroixSkipper Thursday, August 14, 2008 3:55 PM Clarified.
    Thursday, August 14, 2008 3:53 PM
  • Skipper, I appreciate it very much. Now I know, you are a true captain:)
    AlexB
    Thursday, August 14, 2008 4:03 PM
  •  OK, Skipper, I hope you pilot a rescue craft of a good size. I count on you. I ve made some progress but naturally got in trouble as well. Before I begin I want to mention that I think it is odd that this thead is not credited to you as your answer. Someone else got a credit whose rambling seems to be totally irrelevant to the thread as of now. I tried to remedy that and I voted all your posts as helpful. I hope it will work. I will folow up on that and will make sure. I have some conenctions:)

    Anyhow. as much I hated to do that:) I stidued your code yesterday afternoon, made some minor addition (I mean the directory thing) and this morning I tried to run it. Because of my level or perhaps because my specialty is way off I still have some conceptuial difficulty as to how to make a call. One seed parameter is unclear, so I assigned it arbitrarily, ulong, I made it 95. I simply do not understand what this number means and of course I would appreciate if you make a clarification.

    You see, somebody else may try to use your code, will run in the same problem and will be shy enough to be turned away. Thus your magnificent effort (I am only 20% facitious--it is a great code, I'll tell you) will go in vain.

    I am not that personality type. If I need to get down to the bottom of things I always do even at the expense of exposing my uneducated core.

    This is how I called your sea rmonster.

    UInt64 bb = 95;
    string[ ] fileExtensions = new string[ 2 ];
    fileExtensions[ 0 ] = "exe";
    fileExtensions[ 1 ] = "dll";
    Dictionary<UInt64, PInvokeWin32.FileNameAndFrn> dict = new Dictionary<ulong, PInvokeWin32.FileNameAndFrn> ( );
    PInvokeWin32 pInv = new PInvokeWin32 ( );
    pInv.EnumerateVolume ( out dict, fileExtensions );

    I got an exception. Read it:

    Message="Unable to get root frn entry"
      Source="ComeAndGet"
      StackTrace:
           at ComeAndGet.PInvokeWin32.GetRootFrnEntry() in C:\VCsharp_Projects\ComeAndGet\ComeAndGet\MasterRecordRead.cs:line 228
      InnerException: System.ComponentModel.Win32Exception
           Message="The filename, directory name, or volume label syntax is incorrect"
           ErrorCode=-2147467259
           NativeErrorCode=123
           InnerException:

    Is that ulong bb supposed to be a volume label? Does not make much sense though. Anyhow I can get any seed information on any volume with WMI. I just need to know what your sea monster wants for dinner.

    Also I  do not understand what volume we are talking about. There is no volume label for input parameter anywhere. I may be mising something and I am sure when you point it out to me my face will be red. Don't worry. I am used to it.

    Now, this is the exact place where the exception happened. I've never seen anything like this before. This is a fragment of your rmethod, the very few last lines of code:

        else
        {
            throw new IOException ( "GetFileInformationbyHandle() returned invalid handle",
                new Win32Exception ( Marshal.GetLastWin32Error ( ) ) );
        }
            PInvokeWin32.CloseHandle ( hRoot );
        }
        else
        {
                throw new IOException ( "Unable to get root frn entry"
                        new Win32Exception( Marshal.GetLastWin32Error ( ) ) );
        }
    }
    // GetRootFrnEntry  <======== HERE IT WAS. It pointed to the closing curly braket.


    Skipper, my life is hanging by a thread in a balance. You are the only one who can change it.

    Thanks.




    AlexB
    Tuesday, August 26, 2008 8:50 PM
  •  Bump.

    Skipper, is there any hope to get your attention? I just completed a full project and am about to get back to Master Record stuff. Would appreciate your help.

    Thanks.

    AlexB
    Wednesday, September 10, 2008 7:04 PM
  • I just got back from an 8 day hiking trip through Wyoming's Wind Rivers Wilderness area.  So I have just started to catch up on email and other details.  Caught a lot of fish, saw some great country, had some time to myself.

    As for your issue.
    My organization was a little different.  I have a class called EnumerateVolume which uses the class PInvokeWin32.  PInvokeWin32 is simply a class that gives C# the ability to call the low level functions to deal with the MFT.

    That said, GetRootFrnEntry() in my code is in the EnumerateVolume class.  Yours seems to be in the PInvokeWin32 class.  That's fine.  My function uses a member variable called _drive which is of the form \\.\ + _drive.  I'd have to look it up to see what I do to the simple drive letter c, c: to see what the exact format is. I think it is c:.

    The GetRootFrnEntry() function adds the \\.\ and appends the directory separator character (\).  The intent of the function is to "open a file that is the root directory" of the disk volume.  I posted the code for this routine.

    So the problem is to build a dictionary of "File Reference Numbers" associated with their file names and Parent file reference numbers.  The information in the mft contains only file reference numbers for the parents and somehow you have to build a path to the file unless all you want is the file name.  You could call another low level function and given an FRN, get the file name and parent frn. But that would require another io.

    The ulong of the root's parent is zero (0) which stops the recursion through the dictionary at the root.  GetRootFrnEntry() purpose is simply to set up the root's entry in the dictionary.

    That said, I'm not sure I answered your question.  Hope I shed some light on your problem.

    p.s.  I think I mentioned I didn't like the name FileNameAndFrn.  It should have been FileNameAndParentFrn which more precisely defines what the structure/class is about.
    Monday, September 15, 2008 5:07 PM
  • Well, in deference to you, Skipper, and out of profound respect (you've attained a mythical status in my eyes:) I will change the name FileNameAndFrn to FileNameAndParentFrn tomorow, that will make you feel better.

    Thanks for coming back. BTW. My life is like a whirpool in one of those Wisconsin rivers where you caught so many fish and for a time being other problems moved in front of me, thus I kind of stopped "feelng the pain." Nonetheless I want to get it done. It is a great chunk of code and I want to make it work. As I said your name will be forever etched in my heart if we make any sense out of it.

    I don't think you answered my question though, as you suspected. I cannot say for sure at this late hour, give me 15 hours at least and tomorrow morning I will review the code, read your post on a fresh head and we will talk again.

    Many thanks Skipper. I appreciate it.
    AlexB
    Tuesday, September 16, 2008 1:16 AM
  • StCroixSkipper,
       Can I say a big thank you! It took a bit of work, but I came up with the following:

    PInvokeWin32.cs - as described on Page 1 of this thread.
    Log.cs - A quick Log class that writes out a message with the word INFO first. No biggy.
    MFT.cs - I think you call yours EnumerateVolume.cs. Either, the vast majority is listed on page 1.

    A couple of changes that I made:

    Includes:
    using System;  
    using System.Collections;  
    using System.Collections.Generic;  
    using System.Text;  
    using System.Runtime.InteropServices;  
    using System.IO;  
    using System.ComponentModel; 

    Declarations:
    private IntPtr _changeJournalRootHandle;  
    private Dictionary<ulong, FileNameAndFrn> _directories = new Dictionary<ulong,FileNameAndFrn>();  
    private string _drive = "";  
     
    public string Drive  
    {  
        get { return _drive; }  
        set { _drive = value; }  

    Then to call, I did the following:
    static void Main(string[] args)  
    {  
        Dictionary<UInt64, FileNameAndFrn> result;  
        MFT mft = new MFT();  
        mft.Drive = "D:";  
     
        mft.EnumerateVolume(out result, new string[] { ".png" });  
     
        foreach (KeyValuePair<UInt64, FileNameAndFrn> entry in result)  
        {  
            FileNameAndFrn file = (FileNameAndFrn)entry.Value;  
            Console.WriteLine(file.Name);  
        }  
     
        Console.WriteLine("DONE");  
        Console.ReadKey();  

    Works a treat! Many thanks!

    Gaz
    Tuesday, September 16, 2008 10:40 AM
  • Garethbradley, welcome to the fray. I feel you've resolved the last remaining issue I've struggled with but you also introduced a new mystery. What is MFT.cs file with MFT class in it? Is it something you failed to post?

    I looked thru the Skipper's posts and could not see that either. Am I missing something?

    I would really appreciate your clarification: MFT mft = new MFT ( ); // <== wanted.

    Thanks.


    AlexB
    Tuesday, September 16, 2008 6:13 PM
  • Glad it works for you.  I like your name change mft.  It makes sense.  Then next challenge is to use the change journal to identify changes to the volume without doing a findfirst/findnext for the entire volume.  I've think I've taken care of all the edge conditions where the change journal wraps, changes, etc.  The mechanism for reading the change journal is pretty much the same.
    Wednesday, September 17, 2008 4:15 PM
  •  Skipper, I finally got it compiled. Your last comment opened my eyes on the relationships. Now I am getting a runtime stack overflow error on this statement:

    private Dictionary<UInt64, FileNameAndFrn> directories_ = new Dictionary<ulong, FileNameAndFrn> ( );
    public Dictionary<UInt64, FileNameAndFrn> directories
    {
        get { return directories; }   // <== here it is
        set { directories = value; }
    }

    Any thoughts as to how I can handle it?

    Thanks.


    AlexB
    Thursday, September 18, 2008 2:09 PM
  • Just looking at your code ...
    private Dictionary<UInt64, FileNameAndFrn> directories_ = new Dictionary<ulong, FileNameAndFrn> ( );
    public Dictionary<UInt64, FileNameAndFrn> directories
    {
        get { return directories; }   // <== here it is
        set { directories = value; }
    }

    I think it should read
    private Dictionary<UInt64, FileNameAndFrn> directories_ = new Dictionary<ulong, FileNameAndFrn> ( );
    public Dictionary<UInt64, FileNameAndFrn> directories
    {
        get { return directories_; }   // <== here it is
        set { directories_ = value; }
    }

    I've never gotten this error but who knows what would happen when you use directories instead of directories_ in the get and set functions.
    Thursday, September 18, 2008 2:19 PM
  • Thanks Skipper. It's an embarrassing oversight. It was correct at one point but then I refactored it manually and apparently goofed up. Anyhow, now it seems to be working but I am getting an error: invalid handle at GetRootHanlde -- access is denied. It is Vista, so I have to deal with it. I don't think you can help me with it. I pretty much know how to hanlde it. We'll see.

    Thanks a lot.

    AlexB
    Friday, September 19, 2008 12:29 AM
  • Is anyone interested in using the USN Journal from C#?
    Tuesday, December 02, 2008 11:58 PM
  • StCroixSkipper said:

    Is anyone interested in using the USN Journal from C#?

    Sure I am. What do you have in your sleeve?


    AlexB
    Wednesday, December 03, 2008 2:06 AM
  • I think I've figured out how to use it reliably.
    Wednesday, December 03, 2008 3:38 PM
  • Hey guys, great thread and especially great snippets!

    Currently I am working on an application that basically has to be able to undelete data from both NTFS and FAT32 drives.

    I've started working on the NTFS part. So far, your posts were extremely helpful!

    Everything is working nicely... But I am still a bit far away from recovering data...

    Basically, the next step would be selecting those files that are "flagged" as deleted. By now, as you know, the output of EnumerateVolume only stores files that are in use (those files visible to the operating system).

    Could anyone please tell me what changes I should make in order for the previously mentioned thing to happen?

    Thanks!
    Monday, February 16, 2009 7:21 PM
  • Hi StCroixSkipper .

    First of all i would like to thank you for this post. It helped me a lot in my project.
    And I'm really interested in using USN from c#!

    I'm currently investigating this question to, and will notify you with any progress.

    Thanks mate!
    Tuesday, October 13, 2009 11:44 AM
  • I think I've figured out how to use it reliably.
    Have found solution for .NET?
    Tuesday, October 13, 2009 11:49 AM
  • USN_RECORDS return only FileNames this is to reserve as much space possible in the change log. You can map (fileReference) or (ParentFileReference) to complete paths using API. This approach is so much easier to use than keeping a database or enumerating the directories and walking the chain to get the full path.

    Just briefly overlooking the code are you allocating all the directories,fileRef,ParentRef information on the system into a dictionary and then maintaining the dictionary object as an alternative to using a database? This seems like a really great approach but if you use the API you never need to enumerate or maintain this dictionary.

    How effecient would this be for storing such information in a Dictionary object considering a volume can have alot of files/directories present?

    There is some special consideration that needs to be handled when working with (Deleted) items specifically.

    It would be great to see some performance in routines to see which is the better approach to use. That is use the API approach or using the Dictionary approach to walk the chain. It would be great to get some real performance numbers. 

    Let me know if interested.

    Tuesday, January 19, 2010 7:55 AM
  • I've actually done it both ways.  Given a File Reference Number you can to full path.  I used Dictionary in this case because it has the advantage of being faster but it certainly takes up more memory.  Jeffrey Richter used a collection in his example and it seemed appropriate and easy to understand.
    Friday, January 29, 2010 3:15 PM
  • Sorry, I logged in under the wrong username.  I'm actually StCroixSkipper.
    Friday, January 29, 2010 3:16 PM
  • The API seems to be stable with MFT_ENUM_DATA & FSCTL_ENUM_USN_DATA because these entries exists so it can always map the FileReference. However the problem I had when using the API is with FSCTL_READ_USN_JOURNAL it works great for almost every reason but there is some problems using this approach for deleted items because while it returns the FileRef the actual file doesn't exists on disk. I tried to use the ParentRef instead and append the filename from the USN_RECORD. This also doesn't appear to work correctly if deleting an entire directory tree etc.... I am using the NtCreateFile w/ UNICODE_BUFFER so this approach fails horribly because there is no way to map the id to the file because it doesn't exist anymore.

    Did you experience similiar troubles while using the API approach or are you using a different API approach? It appears that even if you use the API approach you will still need a dictionary object anyway to handle such cases. It appears the dictionary approach you have implemented might be the best approach overall.

    I didn't expect any response but I am grateful that you have responded. Thanks
    Friday, January 29, 2010 3:51 PM
  • this is so coooooooooool

    and it works

    thanks a lot StCroixSkipper

    you really make my day

    now i need to get deeper into the MFT in conjunction with the code

    on what OS is this compatible i wonder actually
    the XP's too?

    are the journals needed

    well... i go find out

    so nice this

    menno de ruiter

    signature
    Tuesday, February 02, 2010 8:53 PM
  • The change journal works on all NTFS volumes 3.1 and above this means any NTFS volume on Windows 2000 and above. Enumerating the MFT is good but reading the changes is even better because it makes for a great incremental scan routine.

    There is three changes you need to handle to keep the log up to date ( FileCreate, RenameNewName, FileDelete).

    When a file is created you can keep a log of that files (FileReferenceNumber) these are unique on the system and never change until the file is deleted. Speaking from an incremental backup point of view we don't really need to know the location of the deleted file but we can use the (FileReferenceNumber) to remove that specific record from our incremental log change because it's unique. This keeps the log up to date. When you need to do a scan you can just scan the items in your log because nothing else has changed.

    It should be mentioned that DataOverwrite,DataTruncate should also be used in the checksums to keep up to date information but it really boils down to what you want to keep track of on the system.

    The ONLY problem I have with the journal is it requires administrative rights.. This also applies to reading the MFT but the actual journal doesn't need to be active. You can read MFT without an active journal. Set the HighUsn member to UInt64.MaxValue there shouldn't be anything above this range.

    The best way to handle MFT and the Change Journal is using them with a service otherwise you will need to elevate but you might miss changes as they get purged so a service seems appropriate. If you search around you won't find much or any information regarding the NTFS journal the only article is Keeping Eye on NTFS which is way back in 2000 I belive as far as any real .NET examples I have yet to see anything published outside of a forum of a few code snippets. I have been working with the change journal off and on way back in VB6 before .NET was as appealing as it is now then converting it to VB.NET but C# code is almost identical.

    Thursday, February 04, 2010 2:26 AM
  • Thanks Erik,

    This is highly interesting.

    I am trying to read and manipulate the MFT and thus what i understand from what you wrote, the journal and the logs, from direct raw read/write with use of the disk.sys driver.

    I am trying to sort out of how i can make a tool which optimizes the MFT and defragment the disk.
    a tool like raxco or diskfragmenter

    i might me a long way to go, but i want to give it a shot

    thanks for the explaining of the journal, actually it's thus very crucial
    but suppose the journal and the logs are not their
    does the MFT still reflects the most recent and up to date information about the files on the disk?

    is the journal actually a patch upon the mft data?
    like a filereference in mft should be checked against the journal data if it's not being moved or deleted?

    i am just starting with this project and got info here and their, like winternals book, some good info sheet from ms about all the codes etc. but i try to figure out of how i can access the mft files and map them to raw reading with use of the disk.sys or later on istor or raid or whatever other drivers

    thanks Erik for you post, very helpfull as a root for sorting out more leaves upon this journal issue crucial so to see

    menno de ruiter
    amsterdam
    signature
    Thursday, February 04, 2010 10:57 PM
  • Since WindowsNT when Microsoft first released NTFS, a journaling file system, I've had a fascination with the USN Journal.  I worked on the Primos operating system at Prime Computer and have always been interested in file systems and scheduling, virtual memory management, dynamic linking, ring-oriented security, etc.  In those days, Multics ran on really big machines. When we implemented Primos, we called it 'Multics in a Matchbox'. 

     

    I have been particularly fascinated by the fact that so few commercial products have taken advantage of NTFS's USN Journal.  For backup products especially, since typically less than 10% of the content on a volume changes, you can reduce the time to identify new, deleted or changed files to a fraction of the time it takes to enumerate the entire volume.

     

    As for the Master File Table or MFT, you can enumerate all of the files/directories on a volume in about a tenth of the time it takes to enumerate the volume using FindFirst(), FindNext().  I will qualify the previous statement.  The times to enumerate the volume are about the same if you reboot the system then enumerate. But if the system has been up and running for some time, then my timing studies show that using the MFT takes one tenth of the time of FindFirst(), FindNext().

     

    I've only run into a couple of folks who have really used the USN Journal in a commercial product.  I haven't run into any who have done the work to use the MFT and USN Journal from C#. 

     

    One example of the kinds of performance improvements you can achieve using the USN Journal.  I worked on a de-duplicated file archiving product that discarded old data by enforcing a ‘data retention’ policy’.  You could specify something like ‘keep this data until’ some date.  I wrote a function to purge the data store of all ‘expired’ data.  In an actual customer account, my first attempt at purge took 30+ days to complete.  I wrote a version that used the Master File Table to enumerate the data store and that saved me 90% of a small portion of the time.  It was still unacceptable.  I wrote a service that used the USN Journal to keep all of the data about the data store up-to-date.  The total time it took to purge the data store of ‘stale’ data was reduced from over 30 days to less than 4 hours.

     

    So, here goes. 

     

    The DeviceIoControl function gets you down into the device code of Windows.  And since you don’t want just anyone messing around in the file system, you can only access DeviceIoControl from a process running under ‘administrative’ rights.  You’ll minimally have to use the ‘Run as administrator’ menu item to launch an application that uses the USN Journal.  If you are running on Vista you could write the code to elevate your process but your login must still have admin rights.

     

    Two things help us here, PInvoke and DllImport.  There are some great articles that describe Platform Invocation services or P/Invoke, and the DllImport attribute and how they allow you to access native Win32Api functions that are not currently accessible in .Net.  PInvoke.net is a great resource.

     

    Using the USN Journal requires access to several native Win32 api functions the most notable of which is DeviceIoControl.  This discussion is about the USN Journal and the Master File Table but DeviceIoControl opens the door to many device level operations.  You can see the entire set of FSCTL-control functions and the structures you need to use them in the winioctl.h file.

     

    #define FSCTL_REQUEST_OPLOCK_LEVEL_1    
    #define FSCTL_REQUEST_OPLOCK_LEVEL_2    
    #define FSCTL_REQUEST_BATCH_OPLOCK      
    #define FSCTL_OPLOCK_BREAK_ACKNOWLEDGE  
    #define FSCTL_OPBATCH_ACK_CLOSE_PENDING 
    #define FSCTL_OPLOCK_BREAK_NOTIFY       
    #define FSCTL_LOCK_VOLUME               
    #define FSCTL_UNLOCK_VOLUME             
    #define FSCTL_DISMOUNT_VOLUME           
     
    // decommissioned fsctl value                                              9
    #define FSCTL_IS_VOLUME_MOUNTED         
    #define FSCTL_IS_PATHNAME_VALID         
     
    // decommissioned fsctl value                                             13
    #define FSCTL_QUERY_RETRIEVAL_POINTERS  
    #define FSCTL_GET_COMPRESSION           
    #define FSCTL_SET_COMPRESSION           
    // decommissioned fsctl value                                             17
    // decommissioned fsctl value                                             18
    #define FSCTL_MARK_AS_SYSTEM_HIVE       
    #define FSCTL_OPLOCK_BREAK_ACK_NO_2     
    #define FSCTL_INVALIDATE_VOLUMES        
    #define FSCTL_QUERY_FAT_BPB             
     
    #define FSCTL_FILESYSTEM_GET_STATISTICS 
    #if(_WIN32_WINNT >= 0x0400)
    #define FSCTL_GET_NTFS_VOLUME_DATA      
    #define FSCTL_GET_NTFS_FILE_RECORD      
    #define FSCTL_GET_VOLUME_BITMAP         
    #define FSCTL_GET_RETRIEVAL_POINTERS    
    #define FSCTL_MOVE_FILE                 
    // decomissioned fsctl value                                              31
    #define FSCTL_ALLOW_EXTENDED_DASD_IO    
    #endif /* _WIN32_WINNT >= 0x0400 */
     
    #if(_WIN32_WINNT >= 0x0500)
    // decommissioned fsctl value                                             33
    // decommissioned fsctl value                                             34
    #define FSCTL_FIND_FILES_BY_SID         
    // decommissioned fsctl value                                             36
    // decommissioned fsctl value                                             37
    #define FSCTL_SET_OBJECT_ID             
    #define FSCTL_GET_OBJECT_ID             
    #define FSCTL_DELETE_OBJECT_ID          
    #define FSCTL_SET_REPARSE_POINT         
    #define FSCTL_GET_REPARSE_POINT         
    #define FSCTL_DELETE_REPARSE_POINT      
    #define FSCTL_ENUM_USN_DATA             
    #define FSCTL_SECURITY_ID_CHECK         
     
    #define FSCTL_READ_USN_JOURNAL          
    #define FSCTL_SET_OBJECT_ID_EXTENDED    
    #define FSCTL_CREATE_OR_GET_OBJECT_ID   
     
    #define FSCTL_SET_SPARSE                
     
    #define FSCTL_SET_ZERO_DATA             
    #define FSCTL_QUERY_ALLOCATED_RANGES    
    #define FSCTL_ENABLE_UPGRADE            
    #define FSCTL_SET_ENCRYPTION            
    #define FSCTL_ENCRYPTION_FSCTL_IO       
    #define FSCTL_WRITE_RAW_ENCRYPTED       
    #define FSCTL_READ_RAW_ENCRYPTED        
    #define FSCTL_CREATE_USN_JOURNAL        
    #define FSCTL_READ_FILE_USN_DATA        
    #define FSCTL_WRITE_USN_CLOSE_RECORD    
    #define FSCTL_EXTEND_VOLUME             
    #define FSCTL_QUERY_USN_JOURNAL         
    #define FSCTL_DELETE_USN_JOURNAL        
    #define FSCTL_MARK_HANDLE               
    #define FSCTL_SIS_
    #define FSCTL_SIS_LINK_FILES            
    #define FSCTL_HSM_MSG                   
    // decommissioned fsctl value                                             67
    #define FSCTL_HSM_DATA                  
    #define FSCTL_RECALL_FILE               
    // decommissioned fsctl value                                             70
    #define FSCTL_READ_FROM_PLEX            
    #define FSCTL_FILE_PREFETCH             
    #endif /* _WIN32_WINNT >= 0x0500 */
     
    #if(_WIN32_WINNT >= 0x0600)
    #define FSCTL_MAKE_MEDIA_
    #define FSCTL_SET_DEFECT_MANAGEMENT         
    #define FSCTL_QUERY_SPARING_INFO            
    #define FSCTL_QUERY_ON_DISK_VOLUME_INFO     
    #define FSCTL_SET_VOLUME_COMPRESSION_STATE  
    #define FSCTL_TXFS_MODIFY_RM                
    #define FSCTL_TXFS_QUERY_RM_INFORMATION     
    // decommissioned fsctl value                                                 83
    #define FSCTL_TXFS_ROLLFORWARD_REDO         
    #define FSCTL_TXFS_ROLLFORWARD_UNDO         
    #define FSCTL_TXFS_START_RM                 
    #define FSCTL_TXFS_SHUTDOWN_RM              
    #define FSCTL_TXFS_READ_BACKUP_INFORMATION  
    #define FSCTL_TXFS_WRITE_BACKUP_INFORMATION 
    #define FSCTL_TXFS_CREATE_SECONDARY_RM      
    #define FSCTL_TXFS_GET_METADATA_
    #define FSCTL_TXFS_GET_TRANSACTED_VERSION   
    // decommissioned fsctl value                                                 93
    // decommissioned fsctl value                                                 94
    #define FSCTL_TXFS_CREATE_MINIVERSION       
    // decommissioned fsctl value                                                 96
    // decommissioned fsctl value                                                 97
    // decommissioned fsctl value                                                 98
    #define FSCTL_TXFS_TRANSACTION_ACTIVE       
    #define FSCTL_SET_ZERO_ON_DEALLOCATION      
    #define FSCTL_SET_REPAIR                    
    #define FSCTL_GET_REPAIR                    
    #define FSCTL_WAIT_FOR_REPAIR               
    // decommissioned fsctl value                                                 105
    #define FSCTL_INITIATE_REPAIR               
    #define FSCTL_CSC_INTERNAL                  
    #define FSCTL_SHRINK_VOLUME                 
    #define FSCTL_SET_SHORT_NAME_BEHAVIOR       
    #define FSCTL_DFSR_SET_GHOST_HANDLE_STATE   
     
    //
    //  Values 111 - 119 are reserved for FSRM.
    //
     
    #define FSCTL_TXFS_LIST_TRANSACTION_LOCKED_FILES \
    #define FSCTL_TXFS_LIST_TRANSACTIONS        
    #define FSCTL_QUERY_PAGEFILE_ENCRYPTION     

     

     

    Just a couple of notes:  You'll notice that all the 'DllImport' functions in the Win32Api class you’ll find below  are declared as 'public static' which makes sense since we don't want to create an object just to access these native functions.  All the functions in this class are 'public static' for that reason.

     

    There are some interesting parameters in the [DllImport()] attribute lines.  Many of the native Win32 functions have the ability to set a unique 'last error'.  SetLastError=true tells Platform Invoke Services to 'set last error'.  This allows us to call GetLastError() get more detail if something went wrong.

     

    Many simple data types, such as System.Byte and System.Int32, have a single representation in unmanaged code and do not need their marshaling behavior specified; the common language runtime automatically supplies the correct behavior. UnmanagedType.Bool is the Win32 BOOL type, which is always 4 bytes. [return: MarshalAs(UnmanagedType.Bool)] marshals the unmanaged bool as a managed bool.

     

    On all of the structures you'll see an attribute line of ' [StructLayout(LayoutKind.Sequential, Pack = 1)]'. A 'Pack=1' indicates that data alignment occurs on byte boundaries. There are no gaps between fields with a packing value of 1.

     

    The USN in USN Journal stands for " Update Sequence Number".  The USN Journal provides a persistent log of all changes made to files/directories on a volume.  If a file or directory changes, an entry is placed in the USN Journal denoting the file and the exact change or changes.

     

    Jeffrey Cooperstein and Jeffrey Richter have co-authored a great article http://www.microsoft.com/msj/0999/journal/journal.aspx "Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained." This article discusses using the USN Journal in  C++ and is well worth your time.

     

    Each entry in the USN Journal is assigned a 64 bit number which is the offset into the actual USN Journal file of the entry.  USN Journal entries can be different sizes since at least the filenames are different lengths. 

     

    Each file or directory on a volume has a unique 64 bit File Reference Number.  Files can have the same names excluding the path but no two files or directories can have the same File Reference Number.

     

    My goal here is to share C# code that uses the USN Journal.  I know I can write C++ code that executes faster.  Certainly, having many years of C++ experience helped me to understand how to form the memory for the calling parameters and how to parse through the resultant buffers to process the USN Journal entries.

     

    The DeviceIoControl function [quote]Sends a control code directly to a specified device driver, causing the corresponding device to perform the corresponding operation. Given a device handle, a control code, an input buffer, and an output buffer you can get all kinds of information about the system with DeviceIoControl.  The trouble is that none of it is available to C# code without a little work.

     

    A USN Journal is specific to a given NTFS volume.  The journal when it is started it is essentially an empty 'sparse' file on the volume.  When a change is made to the volume, a record is added to the USN Journal file. 

     

    Quoting from NTFS.com:

    A sparse file has an attribute that causes the I/O subsystem to allocate only meaningful (nonzero) data. Nonzero data is allocated on disk, and non-meaningful data (large strings of data composed of zeros) is not. When a sparse file is read, allocated data is returned as it was stored; non-allocated data is returned, by default, as zeros.

     

    So to free up space on a 'sparse' file you need only to zero out the data.

     

    One of the problems with the USN Journal is that anyone or process with administrative rights can create, start, or stop the USN Journal for any volume. They can remake the volume.  They can do many things that would cause the USN Journal to be ‘invalid’ and some or all of the changes to the volume would be lost.  So the first obstacle to overcome is to determine if there is a USN Journal for a volume and verify that it is valid.  Also, USN Journals can overrun their bounds and become invalid simply because no one has removed the entries.

     

    Each USN Journal has data associated with it that allows us to determine if it is 'valid.' And valid means that we haven't missed any data in the USN Journal and consequently, missed changes to files or directories, it hasn't been stopped and restarted since we've interrogated the USN Journal, and it hasn't overrun its boundaries. When you query the USN Journal you get back a USN_JOURNAL_DATA structure.

     

    typedef struct {
      DWORDLONG UsnJournalID;
      USN       FirstUsn;
      USN       NextUsn;
      USN       LowestValidUsn;
      USN       MaxUsn;
      DWORDLONG MaximumSize;
      DWORDLONG AllocationDelta;
    } USN_JOURNAL_DATA, *PUSN_JOURNAL_DATA;

    The USN type is a 64-bit integer called ‘update sequence number’.  FirstUsn, NextUsn, LowestValidUsn, and MaxUsn are all defined as of USN type.  They represent byte offsets from the beginning of the USN Journal file.  The names are self explanatory.  FirstUsn is the byte offset of the first USN Journal entry.  NextUsn is the byte offset of the first byte of the next USN Journal entry to be written to the USN Journal.

     

    The MaxUsn is the byte offset of the last possible byte in the USN Journal file.  The AllocationDelta is the number of bytes to allocate at the end of the USN Journal file when they are needed.  You can set this number so that you don’t have to allocate data each time you want to add an entry to the Journal.

     

    The trick is to position the file to a byte position, read some data into some buffer and parse out one or more USN Journal Entries.

     

    These are the control codes you’ll need to use the USN Journal.

     

    FSCTL_ENUM_USN_DATA
    FSCTL_READ_USN_JOURNAL
    FSCTL_CREATE_USN_JOURNAL
    FSCTL_QUERY_USN_JOURNAL
    FSCTL_DELETE_USN_JOURNAL

     

     

    FSCTL_CREATE_USN_JOURNAL may be a misnomer because if the USN Journal already exists, this only updates the MaximumSize and AllocationDelta properties if they are larger than those used when the USN Journal was created.  It doesn’t change the UsnJournalID.

     

    Since anyone with admin rights can start or stop the USN Journal or it can exhaust its space or the volume might be reformatted, any application that needs to depend on the USN Journal needs to be able to determine if the USN Journal is ‘valid’.  A ‘valid’ journal has captured all changes to the file system on the volume since the last time we’ve queried it and all of the change entries we need are still available in the USN Journal.

    If we miss even one entry, we have to fall back and determine the state of the USN Journal all over again and then begin to collect changes.  Why is not missing even one USN Journal entry so important?  It depends on the application.  If I am trying to write an application that tracks each and every image file and I miss one USN Journal entry, the worst case is I miss one image file.  But take a backup application that uses the USN Journal to track all new, changed and deleted files.  If it misses one USN Journal entry, it will miss a change to the volume.  The backup application can no longer guarantee that it has protected all the files/directories on the volume.

    So how do we determine if the USN Journal is valid? First we need to query the USN Journal and get its state. The state of the USN Journal is defined by the USN_JOURNAL_DATA data structure defined in the Win32Api.cs file below.  If you call DeviceIoControl with the device handle, the FSCTL_QUERY_USN_JOURNAL control code with an ‘out’ parameter to receive the USN_JOURNAL_DATA you’ll get the UsnJournalID, the FirstUsn, the NextUsn, the LowestValidUsn, the MaxUsn, MaximumSize and AllocationData. 

    Each time you create or start the USN Journal it gets a new UsnJournalID.  If you keep the previous USN_JOURNAL_DATA, you can compare it with the ‘new’ USN_JOURNAL_DATA and determine if you have missed any USN Journal Entries.  Note the LowestValidUsn field. The USN Journal discards ‘old’ change entries as it runs out of space and new entries arrive. If an application doesn’t frequently access the USN Journal and capture the change entries, the USN Journal may free the space occupied by the entries the application needs.  To do this it simply needs to zero out that part of the sparse file. 

    It checks a current USN_JOURNAL_DATA’s UsnJournalID field against a previous USN_JOURNAL_DATA’s UsnJournalID field.  If they are different, the USN Journal is invalid.  The application cannot assume that the USN Journal in this state will provide the change entries for all of the changes that have taken place on the volume.  In fact, you can almost guarantee that change entries will be lost.

    Next it checks if the NextUsn it needs to process from the last time it collected the changes on the volume.  The NextUsn must still be available in the USN Journal.  If both these conditions are met the USN Journal is valid.

    All we need to do is keep a persistent copy of the previous state of the USN Journal around and then we can determine with a quick QueryUsnJournal function whether we can rely on the USN Journal to supply us all changes that have occurred on the volume.

     If you are not careful, the development environment will complain about using native structures.  It will demand that you set the project property ‘Allow unsafe code’ and you’ll have to use the ‘unsafe’ keyword in front of those functions that use the native structures.  If you are careful, you can avoid forcing the ‘Allow unsafe code’ property. The code I’ve provided does not require the ‘Allow unsafe code’ property.

    When you create or query a USN Journal, you can get error codes like: ‘ERROR_JOURNAL_DELETE_IN_PROGRESS’.  It makes more sense to me to handle the error codes instead of throwing exceptions so that if we try to create a USN Journal and it happens to be delete in progress, we should wait until the delete finishes and then create the USN Journal.  The application should not have to deal with these issues.

     

    I’ve added a function, DeleteUsnJournal, which deletes a USN Journal.  You’ll notice that I had to add 4 bytes to the end of the structure in Win32Api.cs to get it to work.  In the documentation, the structure only calls for a UInt64 JournalID and a UInt32 DeleteFlags.  But if you try that structure you’ll get ERROR_INVALID_USER_BUFFER every time.  If you search hard enough and long enough, you’ll find a comment somewhere that says someone got it to work by padding an extra 4 bytes on the end of the structure.

     

    This post is too long to submit,  I'll post the code in a reply... 


    StCroixSkipper
    Friday, April 09, 2010 5:08 PM
  • Here is the code I promised.  First the Win32Api class that has all the DllImports, the constants, etc to access the USN Journal:

    using System;
    using System.Collections.Generic;
    using System.Text;
    using System.IO;
    using System.Runtime.InteropServices;
    using System.Security.Cryptography;
    using Microsoft.Win32;
    using System.ComponentModel;
    
    namespace PInvoke
    {
    	public class Win32Api
        {
            #region enums
            public enum GetLastErrorEnum
            {
                ERROR_SUCCESS = 0,
                ERROR_INVALID_FUNCTION = 1, 
                ERROR_HANDLE_EOF = 38,
                ERROR_INVALID_PARAMETER = 87,
                ERROR_JOURNAL_DELETE_IN_PROGRESS = 1178,
                ERROR_JOURNAL_NOT_ACTIVE  = 1179,
                ERROR_JOURNAL_ENTRY_DELETED = 1181,
                ERROR_INVALID_USER_BUFFER = 1784
            }
    
            public enum UsnJournalDeleteFlags
            {
                USN_DELETE_FLAG_DELETE = 1,
                USN_DELETE_FLAG_NOTIFY = 2
            }
            #endregion
    
            #region constants
            public const UInt32 GENERIC_READ = 0x80000000;
            public const UInt32 GENERIC_WRITE = 0x40000000;
            public const UInt32 FILE_SHARE_READ = 0x00000001;
            public const UInt32 FILE_SHARE_WRITE = 0x00000002;
            public const UInt32 FILE_ATTRIBUTE_DIRECTORY = 0x00000010;
    
            public const UInt32 CREATE_NEW = 1;
            public const UInt32 CREATE_ALWAYS = 2;
            public const UInt32 OPEN_EXISTING = 3;
            public const UInt32 OPEN_ALWAYS = 4;
            public const UInt32 TRUNCATE_EXISTING = 5;
    
            public const UInt32 FILE_ATTRIBUTE_NORMAL = 0x80;
            public const UInt32 FILE_FLAG_BACKUP_SEMANTICS = 0x02000000;
            public const Int32 INVALID_HANDLE_VALUE = -1;
    
            // CTL_CODE( DeviceType, Function, Method, Access ) (((DeviceType) << 16) | ((Access) << 14) | ((Function) << 2) | (Method))
            private const UInt32 FILE_DEVICE_FILE_SYSTEM = 0x00000009;
            private const UInt32 METHOD_NEITHER = 3;
            private const UInt32 METHOD_BUFFERED = 0;
            private const UInt32 FILE_ANY_ACCESS = 0;
            private const UInt32 FILE_SPECIAL_ACCESS = 0;
            private const UInt32 FILE_READ_ACCESS = 1;
            private const UInt32 FILE_WRITE_ACCESS = 2;
    
            public const UInt32 USN_REASON_DATA_OVERWRITE = 0x00000001;
            public const UInt32 USN_REASON_DATA_EXTEND = 0x00000002;
            public const UInt32 USN_REASON_DATA_TRUNCATION = 0x00000004;
            public const UInt32 USN_REASON_NAMED_DATA_OVERWRITE = 0x00000010;
            public const UInt32 USN_REASON_NAMED_DATA_EXTEND = 0x00000020;
            public const UInt32 USN_REASON_NAMED_DATA_TRUNCATION = 0x00000040;
            public const UInt32 USN_REASON_FILE_CREATE = 0x00000100;
            public const UInt32 USN_REASON_FILE_DELETE = 0x00000200;
            public const UInt32 USN_REASON_EA_CHANGE = 0x00000400;
            public const UInt32 USN_REASON_SECURITY_CHANGE = 0x00000800;
            public const UInt32 USN_REASON_RENAME_OLD_NAME = 0x00001000;
            public const UInt32 USN_REASON_RENAME_NEW_NAME = 0x00002000;
            public const UInt32 USN_REASON_INDEXABLE_CHANGE = 0x00004000;
            public const UInt32 USN_REASON_BASIC_INFO_CHANGE = 0x00008000;
            public const UInt32 USN_REASON_HARD_LINK_CHANGE = 0x00010000;
            public const UInt32 USN_REASON_COMPRESSION_CHANGE = 0x00020000;
            public const UInt32 USN_REASON_ENCRYPTION_CHANGE = 0x00040000;
            public const UInt32 USN_REASON_OBJECT_ID_CHANGE = 0x00080000;
            public const UInt32 USN_REASON_REPARSE_POINT_CHANGE = 0x00100000;
            public const UInt32 USN_REASON_STREAM_CHANGE = 0x00200000;
            public const UInt32 USN_REASON_CLOSE = 0x80000000;
    
            public static Int32 GWL_EXSTYLE = -20;
            public static Int32 WS_EX_LAYERED = 0x00080000;
            public static Int32 WS_EX_TRANSPARENT = 0x00000020;
    
            public const UInt32 FSCTL_GET_OBJECT_ID = 0x9009c;
    
            // FSCTL_ENUM_USN_DATA = CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 44,  METHOD_NEITHER, FILE_ANY_ACCESS)
            public const UInt32 FSCTL_ENUM_USN_DATA = (FILE_DEVICE_FILE_SYSTEM << 16) | (FILE_ANY_ACCESS << 14) | (44 << 2) | METHOD_NEITHER;
    
            // FSCTL_READ_USN_JOURNAL = CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 46,  METHOD_NEITHER, FILE_ANY_ACCESS)
            public const UInt32 FSCTL_READ_USN_JOURNAL = (FILE_DEVICE_FILE_SYSTEM << 16) | (FILE_ANY_ACCESS << 14) | (46 << 2) | METHOD_NEITHER;
    
            //  FSCTL_CREATE_USN_JOURNAL        CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 57,  METHOD_NEITHER, FILE_ANY_ACCESS)
            public const UInt32 FSCTL_CREATE_USN_JOURNAL = (FILE_DEVICE_FILE_SYSTEM << 16) | (FILE_ANY_ACCESS << 14) | (57 << 2) | METHOD_NEITHER;
    
            //  FSCTL_QUERY_USN_JOURNAL         CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 61, METHOD_BUFFERED, FILE_ANY_ACCESS)
            public const UInt32 FSCTL_QUERY_USN_JOURNAL = (FILE_DEVICE_FILE_SYSTEM << 16) | (FILE_ANY_ACCESS << 14) | (61 << 2) | METHOD_BUFFERED;
    
            // FSCTL_DELETE_USN_JOURNAL        CTL_CODE(FILE_DEVICE_FILE_SYSTEM, 62, METHOD_BUFFERED, FILE_ANY_ACCESS)
            public const UInt32 FSCTL_DELETE_USN_JOURNAL = (FILE_DEVICE_FILE_SYSTEM << 16) | (FILE_ANY_ACCESS << 14) | (62 << 2) | METHOD_BUFFERED;
    
            #endregion
    
            #region dll imports
    
            /// <summary>
            /// Creates the file specified by 'lpFileName' with desired access, share mode, security attributes,
            /// creation disposition, flags and attributes.
            /// </summary>
            /// <param name="lpFileName">Fully qualified path to a file</param>
            /// <param name="dwDesiredAccess">Requested access (write, read, read/write, none)</param>
            /// <param name="dwShareMode">Share mode (read, write, read/write, delete, all, none)</param>
            /// <param name="lpSecurityAttributes">IntPtr to a 'SECURITY_ATTRIBUTES' structure</param>
            /// <param name="dwCreationDisposition">Action to take on file or device specified by 'lpFileName' (CREATE_NEW,
            /// CREATE_ALWAYS, OPEN_ALWAYS, OPEN_EXISTING, TRUNCATE_EXISTING)</param>
            /// <param name="dwFlagsAndAttributes">File or device attributes and flags (typically FILE_ATTRIBUTE_NORMAL)</param>
            /// <param name="hTemplateFile">IntPtr to a valid handle to a template file with 'GENERIC_READ' access right</param>
            /// <returns>IntPtr handle to the 'lpFileName' file or device or 'INVALID_HANDLE_VALUE'</returns>
            [DllImport("kernel32.dll", SetLastError = true)]
    		public static extern IntPtr CreateFile(string lpFileName, 
                uint dwDesiredAccess,
    			uint dwShareMode, 
                IntPtr lpSecurityAttributes, 
                uint dwCreationDisposition,
    			uint dwFlagsAndAttributes, 
                IntPtr hTemplateFile);
    
            /// <summary>
            /// Closes the file specified by the IntPtr 'hObject'.
            /// </summary>
            /// <param name="hObject">IntPtr handle to a file</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
    		[DllImport("kernel32.dll", SetLastError = true)]
    		[return: MarshalAs(UnmanagedType.Bool)]
    		public static extern bool CloseHandle(
                IntPtr hObject);
    
            /// <summary>
            /// Fills the 'BY_HANDLE_FILE_INFORMATION' structure for the file specified by 'hFile'.
            /// </summary>
            /// <param name="hFile">Fully qualified name of a file</param>
            /// <param name="lpFileInformation">Out BY_HANDLE_FILE_INFORMATION argument</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
            [DllImport("kernel32.dll", SetLastError = true)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool GetFileInformationByHandle(
                IntPtr hFile,
                out BY_HANDLE_FILE_INFORMATION lpFileInformation);
            
            /// <summary>
            /// Deletes the file specified by 'fileName'.
            /// </summary>
            /// <param name="fileName">Fully qualified path to the file to delete</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
    		[DllImport("kernel32.dll", SetLastError = true)]
    		[return: MarshalAs(UnmanagedType.Bool)]
    		public static extern bool DeleteFile(
                string fileName);
    
            /// <summary>
            /// Read data from the file specified by 'hFile'.
            /// </summary>
            /// <param name="hFile">IntPtr handle to the file to read</param>
            /// <param name="lpBuffer">IntPtr to a buffer of bytes to receive the bytes read from 'hFile'</param>
            /// <param name="nNumberOfBytesToRead">Number of bytes to read from 'hFile'</param>
            /// <param name="lpNumberOfBytesRead">Number of bytes read from 'hFile'</param>
            /// <param name="lpOverlapped">IntPtr to an 'OVERLAPPED' structure</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
    		[DllImport("kernel32.dll")]
    		[return: MarshalAs(UnmanagedType.Bool)]
    		public static extern bool ReadFile(
                IntPtr hFile, 
                IntPtr lpBuffer,
    			uint nNumberOfBytesToRead, 
                out uint lpNumberOfBytesRead, 
                IntPtr lpOverlapped);
    
            /// <summary>
            /// Writes the 
            /// </summary>
            /// <param name="hFile">IntPtr handle to the file to write</param>
            /// <param name="bytes">IntPtr to a buffer of bytes to write to 'hFile'</param>
            /// <param name="nNumberOfBytesToWrite">Number of bytes in 'lpBuffer' to write to 'hFile'</param>
            /// <param name="lpNumberOfBytesWritten">Number of bytes written to 'hFile'</param>
            /// <param name="overlapped">IntPtr to an 'OVERLAPPED' structure</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
    		[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
    		[return: MarshalAs(UnmanagedType.Bool)]
    		public static extern bool WriteFile(
                IntPtr hFile, 
                IntPtr bytes,
    			uint nNumberOfBytesToWrite, 
                out uint lpNumberOfBytesWritten,
    			int overlapped);
    
            /// <summary>
            /// Writes the data in 'lpBuffer' to the file specified by 'hFile'.
            /// </summary>
            /// <param name="hFile">IntPtr handle to file to write</param>
            /// <param name="lpBuffer">Buffer of bytes to write to file 'hFile'</param>
            /// <param name="nNumberOfBytesToWrite">Number of bytes in 'lpBuffer' to write to 'hFile'</param>
            /// <param name="lpNumberOfBytesWritten">Number of bytes written to 'hFile'</param>
            /// <param name="overlapped">IntPtr to an 'OVERLAPPED' structure</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
            [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool WriteFile(
                IntPtr hFile,
                byte[] lpBuffer,
                uint nNumberOfBytesToWrite,
                out uint lpNumberOfBytesWritten,
                int overlapped);
    
            /// <summary>
            /// Sends the 'dwIoControlCode' to the device specified by 'hDevice'.
            /// </summary>
            /// <param name="hDevice">IntPtr handle to the device to receive 'dwIoControlCode'</param>
            /// <param name="dwIoControlCode">Device IO Control Code to send</param>
            /// <param name="lpInBuffer">Input buffer if required</param>
            /// <param name="nInBufferSize">Size of input buffer</param>
            /// <param name="lpOutBuffer">Output buffer if required</param>
            /// <param name="nOutBufferSize">Size of output buffer</param>
            /// <param name="lpBytesReturned">Number of bytes returned in output buffer</param>
            /// <param name="lpOverlapped">IntPtr to an 'OVERLAPPED' structure</param>
            /// <returns>'true' if successful, otherwise 'false'</returns>
            [DllImport("kernel32.dll", ExactSpelling = true, SetLastError = true, CharSet = CharSet.Auto)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool DeviceIoControl(
                IntPtr hDevice,
                UInt32 dwIoControlCode,
                IntPtr lpInBuffer, 
                Int32 nInBufferSize,
                out USN_JOURNAL_DATA lpOutBuffer, 
                Int32 nOutBufferSize,
                out uint lpBytesReturned, 
                IntPtr lpOverlapped);
    
            /// <summary>
            /// Sends the control code 'dwIoControlCode' to the device driver specified by 'hDevice'.
            /// </summary>
            /// <param name="hDevice">IntPtr handle to the device to receive 'dwIoControlCode</param>
            /// <param name="dwIoControlCode">Device IO Control Code to send</param>
            /// <param name="lpInBuffer">Input buffer if required</param>
            /// <param name="nInBufferSize">Size of input buffer </param>
            /// <param name="lpOutBuffer">Output buffer if required</param>
            /// <param name="nOutBufferSize">Size of output buffer</param>
            /// <param name="lpBytesReturned">Number of bytes returned</param>
            /// <param name="lpOverlapped">Pointer to an 'OVERLAPPED' struture</param>
            /// <returns></returns>
            [DllImport("kernel32.dll", ExactSpelling = true, SetLastError = true, CharSet = CharSet.Auto)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool DeviceIoControl(
                IntPtr hDevice,
                UInt32 dwIoControlCode,
                IntPtr lpInBuffer, 
                Int32 nInBufferSize,
                IntPtr lpOutBuffer, 
                Int32 nOutBufferSize,
                out uint lpBytesReturned, 
                IntPtr lpOverlapped);
    
            /// <summary>
            /// Sets the number of bytes specified by 'size' of the memory associated with the argument 'ptr' 
            /// to zero.
            /// </summary>
            /// <param name="ptr"></param>
            /// <param name="size"></param>
            [DllImport("kernel32.dll")]
    		public static extern void ZeroMemory(IntPtr ptr, int size);
    
            [DllImport("user32.dll", CharSet = CharSet.Auto)]
            [return: MarshalAs(UnmanagedType.Bool)]
            public static extern bool GetCursorPos(out POINT pt);
    
            [DllImport("user32.dll", CharSet = CharSet.Auto)]
            public static extern Int32 GetWindowLong(IntPtr hWnd, Int32 nIndex);
    
            [DllImport("user32.dll", CharSet = CharSet.Auto)]
            public static extern Int32 SetWindowLong(IntPtr hWnd, Int32 nIndex, Int32 newVal);
    
            #endregion
    
            #region structures
    
            /// <summary>
            /// By Handle File Information structure, contains File Attributes(32bits), Creation Time(FILETIME),
            /// Last Access Time(FILETIME), Last Write Time(FILETIME), Volume Serial Number(32bits),
            /// File Size High(32bits), File Size Low(32bits), Number of Links(32bits), File Index High(32bits),
            /// File Index Low(32bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct BY_HANDLE_FILE_INFORMATION
            {
                public uint FileAttributes;
                public FILETIME CreationTime;
                public FILETIME LastAccessTime;
                public FILETIME LastWriteTime;
                public uint VolumeSerialNumber;
                public uint FileSizeHigh;
                public uint FileSizeLow;
                public uint NumberOfLinks;
                public uint FileIndexHigh;
                public uint FileIndexLow;
            }
    
            /// <summary>
            /// USN Journal Data structure, contains USN Journal ID(64bits), First USN(64bits), Next USN(64bits),
            /// Lowest Valid USN(64bits), Max USN(64bits), Maximum Size(64bits) and Allocation Delta(64bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct USN_JOURNAL_DATA
            {
                public UInt64 UsnJournalID;
                public Int64 FirstUsn;
                public Int64 NextUsn;
                public Int64 LowestValidUsn;
                public Int64 MaxUsn;
                public UInt64 MaximumSize;
                public UInt64 AllocationDelta;
            }
    
            /// <summary>
            /// MFT Enum Data structure, contains Start File Reference Number(64bits), Low USN(64bits),
            /// High USN(64bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct MFT_ENUM_DATA
            {
                public UInt64 StartFileReferenceNumber;
                public Int64 LowUsn;
                public Int64 HighUsn;
            }
    
            /// <summary>
            /// Create USN Journal Data structure, contains Maximum Size(64bits) and Allocation Delta(64(bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct CREATE_USN_JOURNAL_DATA
            {
                public UInt64 MaximumSize;
                public UInt64 AllocationDelta;
            }
    
            /// <summary>
            /// Create USN Journal Data structure, contains Maximum Size(64bits) and Allocation Delta(64(bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct DELETE_USN_JOURNAL_DATA
            {
                public UInt64 UsnJournalID;
                public UInt32 DeleteFlags;
                public UInt32 Reserved;
            }
    
            /// <summary>
            /// Contains the USN Record Length(32bits), USN(64bits), File Reference Number(64bits), 
            /// Parent File Reference Number(64bits), Reason Code(32bits), File Attributes(32bits),
            /// File Name Length(32bits), the File Name Offset(32bits) and the File Name.
            /// </summary>
            public class USN_RECORD
            {
                private const int FR_OFFSET = 8;
                private const int PFR_OFFSET = 16;
                private const int USN_OFFSET = 24;
                private const int REASON_OFFSET = 40;
                public const int FA_OFFSET = 52;
                private const int FNL_OFFSET = 56;
                private const int FN_OFFSET = 58;
    
                public UInt32 RecordLength;
                public UInt64 FileReferenceNumber;
                public UInt64 ParentFileReferenceNumber;
                public Int64 Usn;
                public UInt32 Reason;
                public UInt32 FileAttributes;
                public Int32 FileNameLength;
                public Int32 FileNameOffset;
                public string FileName = string.Empty;
    
                /// <summary>
                /// USN Record Constructor
                /// </summary>
                /// <param name="p">Buffer of bytes representing the USN Record</param>
                public USN_RECORD(IntPtr p)
                {
                    this.RecordLength = (UInt32)Marshal.ReadInt32(p);
                    this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET);
                    this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET);
                    this.Usn = Marshal.ReadInt64(p, USN_OFFSET);
                    this.Reason = (UInt32)Marshal.ReadInt32(p, REASON_OFFSET);
                    this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET);
                    this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET);
                    this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET);
                    FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char));
                }
            }
    
            /// <summary>
            /// Contains the Start USN(64bits), Reason Mask(32bits), Return Only on Close flag(32bits),
            /// Time Out(64bits), Bytes To Wait For(64bits), and USN Journal ID(64bits).
            /// </summary>
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct READ_USN_JOURNAL_DATA
            {
                public Int64 StartUsn;
                public UInt32 ReasonMask;
                public UInt32 ReturnOnlyOnClose;
                public UInt64 Timeout;
                public UInt64 bytesToWaitFor;
                public UInt64 UsnJournalId;
            }
    
            [StructLayout(LayoutKind.Sequential)]
            public struct POINT
            {
                public int X;
                public int Y;
    
                public POINT(int x, int y)
                {
                    this.X = x;
                    this.Y = y;
                }
            }
    
            #endregion
    
            #region functions
    
            /// <summary>
            /// Writes the data in 'text' to the alternate stream ':Description' of the file 'currentFile.
            /// </summary>
            /// <param name="currentfile">Fully qualified path to a file</param>
            /// <param name="text">Data to write to the ':Description' stream</param>
            public static void WriteAlternateStream(string currentfile, string text)
    		{
    			string AltStreamDesc = currentfile + ":Description";
                IntPtr txtBuffer = IntPtr.Zero;
                IntPtr hFile = IntPtr.Zero;
    			DeleteFile(AltStreamDesc);
                string descText = text.TrimEnd(' ');
    
                try
                {
                    hFile = CreateFile(AltStreamDesc, GENERIC_WRITE, 0, IntPtr.Zero,
                                           CREATE_ALWAYS, 0, IntPtr.Zero);
                    if (-1 != hFile.ToInt32())
                    {
                        txtBuffer = Marshal.StringToHGlobalUni(descText);
                        uint nBytes, count;
                        nBytes = (uint)descText.Length;
                        bool bRtn = WriteFile(hFile, txtBuffer, sizeof(char) * nBytes, out count, 0);
                        if (!bRtn)
                        {
                            if ((sizeof(char) * nBytes) != count)
                            {
                                throw new Exception(string.Format("Bytes written {0} should be {1} for file {2}.",
                                    count, sizeof(char) * nBytes, AltStreamDesc));
                            }
                            else
                            {
                                throw new Exception("WriteFile() returned false");
                            }
                        }
                    }
                    else
                    {
                        throw new Win32Exception(Marshal.GetLastWin32Error());
                    }
                }
                catch (Exception exception)
                {
                    string msg = string.Format("Exception caught in WriteAlternateStream()\n  '{0}'\n  for file '{1}'.", 
                        exception.Message, AltStreamDesc);
                    Console.WriteLine(msg);
                }
                finally
                {
                    CloseHandle(hFile);
                    hFile = IntPtr.Zero;
                    Marshal.FreeHGlobal(txtBuffer);
                    GC.Collect();
                }
    		}
    
            /// <summary>
            /// Adds the ':Description' alternate stream name to the argument 'currentFile'.
            /// </summary>
            /// <param name="currentfile">The file whose alternate stream is to be read</param>
            /// <returns>A string value representing the value of the alternate stream</returns>
    		public static string ReadAlternateStream(string currentfile)
    		{
    			string AltStreamDesc = currentfile + ":Description";
    			string returnstring = ReadAlternateStreamEx(AltStreamDesc);
    			return returnstring;
    		}
    
            /// <summary>
            /// Reads the stream represented by 'currentFile'.
            /// </summary>
            /// <param name="currentfile">Fully qualified path including stream</param>
            /// <returns>Value of the alternate stream as a string</returns>
    		public static string ReadAlternateStreamEx(string currentfile)
    		{
    			string returnstring = string.Empty;
                IntPtr hFile = IntPtr.Zero;
                IntPtr buffer = IntPtr.Zero;
                try
                {
                    hFile = CreateFile(currentfile, GENERIC_READ, 0, IntPtr.Zero, OPEN_EXISTING, 0, IntPtr.Zero);
                    if (-1 != hFile.ToInt32())
                    {
                        buffer = Marshal.AllocHGlobal(1000 * sizeof(char));
                        ZeroMemory(buffer, 1000 * sizeof(char));
                        uint nBytes;
                        bool bRtn = ReadFile(hFile, buffer, 1000 * sizeof(char), out nBytes, IntPtr.Zero);
                        if (bRtn)
                        {
                            if (nBytes > 0)
                            {
                                returnstring = Marshal.PtrToStringAuto(buffer);
                                //byte[] byteBuffer = new byte[nBytes];
                                //for (int i = 0; i < nBytes; i++)
                                //{
                                //    byteBuffer[i] = Marshal.ReadByte(buffer, i);
                                //}
                                //returnstring = Encoding.Unicode.GetString(byteBuffer, 0, (int)nBytes);
                            }
                            else
                            {
                                throw new Exception("ReadFile() returned true but read zero bytes");
                            }
                        }
                        else
                        {
                            if (nBytes <= 0)
                            {
                                throw new Exception("ReadFile() read zero bytes.");
                            }
                            else
                            {
                                throw new Exception("ReadFile() returned false");
                            }
                        }
                    }
                    else
                    {
                        Exception excptn = new Win32Exception(Marshal.GetLastWin32Error());
                        if (!excptn.Message.Contains("cannot find the file"))
                        {
                            throw excptn;
                        }
                    }
                }
                catch (Exception exception)
                {
                    string msg = string.Format("Exception caught in ReadAlternateStream(), '{0}'\n  for file '{1}'.", 
                        exception.Message, currentfile);
                    Console.WriteLine(msg);
                    Console.WriteLine(exception.Message);
                }
                finally
                {
                    CloseHandle(hFile);
                    hFile = IntPtr.Zero;
                    if (buffer != IntPtr.Zero)
                    {
                        Marshal.FreeHGlobal(buffer);
                    }
                    GC.Collect();
                }
    			return returnstring;
    		}
    
            /// <summary>
            /// Read the encrypted alternate stream specified by 'currentFile'.
            /// </summary>
            /// <param name="currentfile">Fully qualified path to encrypted alternate stream</param>
            /// <returns>The un-encrypted value of the alternate stream as a string</returns>
    		public static string ReadAlternateStreamEncrypted(string currentfile)
    		{
    			string returnstring = string.Empty;
                IntPtr buffer = IntPtr.Zero;
                IntPtr hFile = IntPtr.Zero;
                try
                {
                    hFile = CreateFile(currentfile, GENERIC_READ, 0, IntPtr.Zero, OPEN_EXISTING, 0, IntPtr.Zero);
                    if (-1 != hFile.ToInt32())
                    {
                        buffer = Marshal.AllocHGlobal(1000 * sizeof(char));
                        ZeroMemory(buffer, 1000 * sizeof(char));
                        uint nBytes;
                        bool bRtn = ReadFile(hFile, buffer, 1000 * sizeof(char), out nBytes, IntPtr.Zero);
                        if (0 != nBytes)
                        {
                            returnstring = DecryptLicenseString(buffer, nBytes);
                        }
                    }
                    else
                    {
                        Exception excptn = new Win32Exception(Marshal.GetLastWin32Error());
                        if (!excptn.Message.Contains("cannot find the file"))
                        {
                            throw excptn;
                        }
                    }
                }
                catch (Exception exception)
                {
                    Console.WriteLine("Exception caught in ReadAlternateStreamEncrypted()");
                    Console.WriteLine(exception.Message);
                }
                finally
                {
                    CloseHandle(hFile);
                    hFile = IntPtr.Zero;
                    if (buffer != IntPtr.Zero)
                    {
                        Marshal.FreeHGlobal(buffer);
                    }
                    GC.Collect();
                }
    			return returnstring;
    		}
    
            /// <summary>
            /// Writes the value of 'LicenseString' as an encrypted stream to the file:stream specified
            /// by 'currentFile'.
            /// </summary>
            /// <param name="currentFile">Fully qualified path to the alternate stream</param>
            /// <param name="LicenseString">The string value to encrypt and write to the alternate stream</param>
    		public static void WriteAlternateStreamEncrypted(string currentFile, string LicenseString)
    		{
    			RC2CryptoServiceProvider rc2 = null;
    			CryptoStream cs = null;
    			MemoryStream ms = null;
    			uint count = 0;
                IntPtr buffer = IntPtr.Zero;
                IntPtr hFile = IntPtr.Zero;
    			try {
    				Encoding enc = Encoding.Unicode;
    
    				byte[] ba = enc.GetBytes(LicenseString);
    				ms = new MemoryStream();
    
    				rc2 = new RC2CryptoServiceProvider();
    				rc2.Key = GetBytesFromHexString("7a6823a42a3a3ae27057c647db812d0");
    				rc2.IV = GetBytesFromHexString("827d961224d99b2d");
    
    				cs = new CryptoStream(ms, rc2.CreateEncryptor(), CryptoStreamMode.Write);
    				cs.Write(ba, 0, ba.Length);
    				cs.FlushFinalBlock();
    
                    buffer = Marshal.AllocHGlobal(1000 * sizeof(char));
    				ZeroMemory(buffer, 1000 * sizeof(char));
    				uint nBytes = (uint)ms.Length;
    				Marshal.Copy(ms.GetBuffer(), 0, buffer, (int)nBytes);
    
    				DeleteFile(currentFile);
    				hFile = CreateFile(currentFile, GENERIC_WRITE, 0, IntPtr.Zero,
    									   CREATE_ALWAYS, 0, IntPtr.Zero);
    				if (-1 != hFile.ToInt32()) 
                    {
    					bool bRtn = WriteFile(hFile, buffer, nBytes, out count, 0);
    				} 
                    else 
                    {
                        Exception excptn = new Win32Exception(Marshal.GetLastWin32Error());
                        if (!excptn.Message.Contains("cannot find the file"))
                        {
                            throw excptn;
                        }
                    }
    			} 
                catch (Exception exception) 
                {
                    Console.WriteLine("WriteAlternateStreamEncrypted()");
                    Console.WriteLine(exception.Message);
    			} 
                finally 
                {
    				CloseHandle(hFile);
    				hFile = IntPtr.Zero;
                    if (cs != null)
                    {
                        cs.Close();
                        cs.Dispose();
                    }
                    
    				rc2 = null;
                    if (ms != null)
                    {
                        ms.Close();
                        ms.Dispose();
                    }
                    if (buffer != IntPtr.Zero)
                    {
                        Marshal.FreeHGlobal(buffer);
                    }
    			}
    		}
    
            /// <summary>
            /// Encrypt the string 'LicenseString' argument and return as a MemoryStream.
            /// </summary>
            /// <param name="LicenseString">The string value to encrypt</param>
            /// <returns>A MemoryStream which contains the encrypted value of 'LicenseString'</returns>
            private static MemoryStream  EncryptLicenseString(string LicenseString)
    		{
    			Encoding enc = Encoding.Unicode;
    
    			byte[] ba = enc.GetBytes(LicenseString);
    			MemoryStream ms = new MemoryStream();
    
    			RC2CryptoServiceProvider rc2 = new RC2CryptoServiceProvider();
    			rc2.Key = GetBytesFromHexString("7a6823a42a3a3ae27057c647db812d0");
    			rc2.IV = GetBytesFromHexString("827d961224d99b2d");
    
    			CryptoStream cs = new CryptoStream(ms, rc2.CreateEncryptor(), CryptoStreamMode.Write);
    			cs.Write(ba, 0, ba.Length);
    
    			cs.Close();
    			cs.Dispose();
    			rc2 = null;
    			return ms;
    		}
    
            /// <summary>
            /// Given an IntPtr to a bufer and the number of bytes, decrypt the buffer and return an 
            /// unencrypted text string.
            /// </summary>
            /// <param name="buffer">An IntPtr to the 'buffer' containing the encrypted string</param>
            /// <param name="nBytes">The number of bytes in 'buffer' to decrypt</param>
            /// <returns></returns>
    		private static string DecryptLicenseString(IntPtr buffer, uint nBytes)
    		{
    			byte[] ba = new byte[nBytes];
    			for( int i=0; i<nBytes; i++) {
    				ba[i] = Marshal.ReadByte(buffer, i);
    			}
    			MemoryStream ms = new MemoryStream(ba);
    
    			RC2CryptoServiceProvider rc2 = new RC2CryptoServiceProvider();
    			rc2.Key = GetBytesFromHexString("7a6823a42a3a3ae27057c647db812d0");
    			rc2.IV = GetBytesFromHexString("827d961224d99b2d");
    
    			CryptoStream cs = new CryptoStream(ms, rc2.CreateDecryptor(), CryptoStreamMode.Read);
    			string licenseString = string.Empty;
    			byte[] ba1 = new byte[4096];
    			int irtn = cs.Read(ba1, 0, 4096);
    			Encoding enc = Encoding.Unicode;
    			licenseString = enc.GetString(ba1, 0, irtn);
    
    			cs.Close();
    			cs.Dispose();
    			ms.Close();
    			ms.Dispose();
    			rc2 = null;
    			return licenseString;
    		}
    
            /// <summary>
            /// Gets the byte array generated from the value of 'hexString'.
            /// </summary>
            /// <param name="hexString">Hexadecimal string</param>
            /// <returns>Array of bytes generated from 'hexString'.</returns>
    		public static byte[] GetBytesFromHexString(string hexString)
    		{
    			int numHexChars = hexString.Length / 2;
    			byte[] ba = new byte[numHexChars];
    			int j = 0;
    			for (int i = 0; i < ba.Length; i++) {
    				string hex = new string(new char[] { hexString[j], hexString[j + 1] });
    				ba[i] = byte.Parse(hex, System.Globalization.NumberStyles.HexNumber);
    				j = j + 2;
    			}
    			return ba;
            }
    
            #endregion
        }
    }
    

     

    Next the FileNameAndParentFrn class:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    
    using PInvoke;
    
    namespace ConsoleUSNJournalProject
    {
        [Serializable]
        public class FileNameAndParentFrn : IEquatable<FileNameAndParentFrn>
        {
            #region Properties
    
            private uint _volSer;
            public uint VolSer
            {
                get { return _volSer; }
            }
    
            private string _name;
            public string Name
            {
                get { return _name; }
                set { _name = value; }
            }
    
            private UInt64 _frn;
            public UInt64 Frn
            {
                get { return _frn; }
                set { _frn = value; }
            }
    
            private UInt64 _parentFrn;
            public UInt64 ParentFrn
            {
                get { return _parentFrn; }
                set { _parentFrn = value; }
            }
    
            private byte[] _md5;
            public byte[] MD5
            {
                get { return _md5; }
                set { _md5 = value; }
            }
    
            #endregion
    
            #region Constructor(s)
    
            public FileNameAndParentFrn(uint volser, string name, UInt64 frn, UInt64 parentFrn)
            {
                if (!string.IsNullOrEmpty(name))
                {
                    _name = name;
                }
                else
                {
                    throw new ArgumentException("Invalid argument: null or Length = zero", "name");
                }
                _volSer = volser;
                _parentFrn = parentFrn;
                _frn = frn;
                _md5 = null;
            }   // end FileNameAndParentFrn() closing bracket
    
            public FileNameAndParentFrn(uint volser, ulong frn)
            {
                _volSer = volser;
                _frn = frn;
            }
    
            public FileNameAndParentFrn(uint volser, Win32Api.USN_RECORD usn)
            {
                if (!string.IsNullOrEmpty(usn.FileName))
                {
                    _name = usn.FileName;
                }
                else
                {
                    throw new ArgumentException("Invalid argument: null or Length = zero", "name");
                }
                _volSer = volser;
                _parentFrn = usn.ParentFileReferenceNumber;
                _frn = usn.FileReferenceNumber;
                _md5 = null;
            }   // end FileNameAndParentFrn() closing bracket
    
            #endregion
    
            #region IEquatable<FileNameAndParentFrn> Members
    
            public bool Equals(FileNameAndParentFrn other)
            {
                bool retval = false;
                if ((object)other != null)
                {
                    if (_frn == other._frn && _volSer == other.VolSer)
                        retval = true;
                }
                return retval;
            }
    
            public override bool Equals(object obj)
            {
                if (obj == null) return base.Equals(obj);
    
                if (!(obj is FileNameAndParentFrn))
                {
                    throw new InvalidCastException("Argument is not a 'FileNameAndParentFrn' type object");
                }
                return base.Equals(obj);
            }
    
            public override int GetHashCode()
            {
                return this._frn.GetHashCode();
            }
    
            public static bool operator ==(FileNameAndParentFrn fnapfrn1, FileNameAndParentFrn fnapfrn2)
            {
                return fnapfrn1.Equals(fnapfrn2);
            }
    
            public static bool operator !=(FileNameAndParentFrn fnapfrn1, FileNameAndParentFrn fnapfrn2)
            {
                return (!fnapfrn1.Equals(fnapfrn2));
            }
    
            #endregion
        }
    }
    

     

    Still not enough room.  The rest of the code is in another reply.

     


    StCroixSkipper
    Friday, April 09, 2010 5:17 PM
  • The final post of source code...

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    
    using System.IO;
    using PInvoke;
    using System.Runtime.InteropServices;
    using System.ComponentModel;
    using System.Xml;
    using System.Threading;
    
    namespace ConsoleUSNJournalProject
    {
        class Program
        {
            private static DriveInfo _driveInfo;
            private static IntPtr _changeJournalRootHandle;
            private static uint _volumeSerialNumber;
            private static ulong _rootFrn = 0;
            private static Win32Api.USN_JOURNAL_DATA _currentUsnState = new Win32Api.USN_JOURNAL_DATA();
            private static Win32Api.USN_JOURNAL_DATA _previousUsnState = new Win32Api.USN_JOURNAL_DATA();
            private static uint _previousVolSer;
    
            private static bool _bFoldersModified = false;
            private static string _mftFoldersFileName = "_Folders.xml";
            private static string _mftFoldersFullPath = Path.Combine(Directory.GetCurrentDirectory(), _mftFoldersFileName);
            private static string _usnJournalStateFileName = "_UsnJournalState.xml";
            private static string _usnJournalStateFullPath = Path.Combine(Directory.GetCurrentDirectory(), _usnJournalStateFileName);
    
    
            private static Dictionary<UInt64, FileNameAndParentFrn> _folders = new Dictionary<ulong, FileNameAndParentFrn>();
    
            static void Main(string[] args)
            {
                //
                // set the _driveInfo to the C-drive
                //
                _driveInfo = new DriveInfo("C");
    
                try
                {
                    InitializeChangeJournal();
    
                    Console.WriteLine();
                    Console.WriteLine("Total Directories found: {0}", _folders.Count);
    
                    Console.WriteLine();
                    Console.WriteLine("Journal state:");
                    Console.WriteLine("    Journal ID: {0}", _previousUsnState.UsnJournalID);
                    Console.WriteLine("   MaximumSize: {0}", _previousUsnState.MaximumSize);
                    Console.WriteLine("        MaxUsn: {0}", _previousUsnState.MaxUsn);
                    Console.WriteLine("      FirstUsn: {0}", _previousUsnState.FirstUsn);
                    Console.WriteLine("       NextUsn:", _previousUsnState.NextUsn);
    
                    Console.WriteLine();
                    Console.WriteLine("Do something that causes the files/folders on C-Drive to change, then hit enter");
                    Console.ReadKey();
                    Win32Api.GetLastErrorEnum lastWin32Error = GatherUsnJournalChanges();
                    Console.WriteLine();
                    Console.WriteLine("Return value from GatherUsnJournalChanges: {0}", lastWin32Error.ToString());
                    
                }
                catch (Exception excptn)
                {
                    DisplayException(excptn);
                }
                Console.Write("Press any key to exit!");
                Console.Read();
            }
    
            private static Win32Api.GetLastErrorEnum CreateUsnJournal()
            {
                Console.WriteLine("CreateUsnJournal() function entered for drive '{0}'", _driveInfo.Name);
    
                // This function creates a journal on the volume. If a journal already
                // exists this function will adjust the MaximumSize and AllocationDelta
                // parameters of the journal
    
                Win32Api.GetLastErrorEnum lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
    
                UInt64 MaximumSize = 0x10000000;
                UInt64 AllocationDelta = 0x100000;
                UInt32 cb;
    
                Win32Api.CREATE_USN_JOURNAL_DATA cujd = new Win32Api.CREATE_USN_JOURNAL_DATA();
                cujd.MaximumSize = MaximumSize;
                cujd.AllocationDelta = AllocationDelta;
    
                int sizeCujd = Marshal.SizeOf(cujd);
                IntPtr cujdBuffer = Marshal.AllocHGlobal(sizeCujd);
                Win32Api.ZeroMemory(cujdBuffer, sizeCujd);
                Marshal.StructureToPtr(cujd, cujdBuffer, true);
                //
                // -1 is an invalid handle and _changeJournalRootHandle is initialized to IntPtr.Zero
                //
                if (_changeJournalRootHandle.ToInt32() == 0 || _changeJournalRootHandle.ToInt32() == Win32Api.INVALID_HANDLE_VALUE)
                {
                    GetRootHandle();
                }
    
                bool fOk = Win32Api.DeviceIoControl(
                    _changeJournalRootHandle,
                    Win32Api.FSCTL_CREATE_USN_JOURNAL,
                    cujdBuffer,
                    sizeCujd,
                    IntPtr.Zero,
                    0,
                    out cb,
                    IntPtr.Zero);
                if (!fOk)
                {
                    lastWin32Error = (Win32Api.GetLastErrorEnum)Marshal.GetLastWin32Error();
                }
                Marshal.FreeHGlobal(cujdBuffer);
                return lastWin32Error;
            }
    
            private static Win32Api.GetLastErrorEnum DeleteUsnJournal()
            {
                //
                // This function deletes a usn journal on the volume. 
                //
                
                Console.WriteLine("DeleteUsnJournal() function entered for drive '{0}'", _driveInfo.Name);
    
                Win32Api.GetLastErrorEnum lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
    
                UInt32 cb;
    
                Win32Api.DELETE_USN_JOURNAL_DATA dujd = new Win32Api.DELETE_USN_JOURNAL_DATA();
                dujd.UsnJournalID = _currentUsnState.UsnJournalID;
                dujd.DeleteFlags = (UInt32)Win32Api.UsnJournalDeleteFlags.USN_DELETE_FLAG_DELETE;
    
                int sizeDujd = Marshal.SizeOf(dujd);
                IntPtr dujdBuffer = Marshal.AllocHGlobal(sizeDujd);
                Win32Api.ZeroMemory(dujdBuffer, sizeDujd);
                Marshal.StructureToPtr(dujd, dujdBuffer, true);
                //
                // -1 is an invalid handle and _changeJournalRootHandle is initialized to IntPtr.Zero
                //
                if (_changeJournalRootHandle.ToInt32() == 0 || _changeJournalRootHandle.ToInt32() == Win32Api.INVALID_HANDLE_VALUE)
                {
                    GetRootHandle();
                }
    
                bool fOk = Win32Api.DeviceIoControl(
                    _changeJournalRootHandle,
                    Win32Api.FSCTL_DELETE_USN_JOURNAL,
                    dujdBuffer,
                    sizeDujd,
                    IntPtr.Zero,
                    0,
                    out cb,
                    IntPtr.Zero);
                if (!fOk)
                {
                    lastWin32Error = (Win32Api.GetLastErrorEnum)Marshal.GetLastWin32Error();
                }
                Marshal.FreeHGlobal(dujdBuffer);
                return lastWin32Error;
            }
    
            private static Win32Api.GetLastErrorEnum QueryUsnJournal()
            {
                Console.WriteLine("QueryUsnJournal() function entered for drive '{0}'", _driveInfo.Name);
                //
                // This function queries the usn journal on the volume. 
                //
                Win32Api.GetLastErrorEnum lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
    
                Win32Api.USN_JOURNAL_DATA ujd = new Win32Api.USN_JOURNAL_DATA();
                int sizeUjd = Marshal.SizeOf(ujd);
                UInt32 cb;
    
                if (_changeJournalRootHandle.ToInt32() == 0)
                {
                    GetRootHandle();
                }
                bool fOk = Win32Api.DeviceIoControl(
                    _changeJournalRootHandle,
                    Win32Api.FSCTL_QUERY_USN_JOURNAL,
                    IntPtr.Zero,
                    0,
                    out ujd,
                    sizeUjd,
                    out cb,
                    IntPtr.Zero);
                if (fOk)
                {
                    _currentUsnState = ujd;
                    Console.WriteLine("  Current usn journal state");
                    Console.WriteLine("       Vol Ser: '{0}'", _volumeSerialNumber);
                    Console.WriteLine("    Journal ID: '{0}'", _currentUsnState.UsnJournalID);
                    Console.WriteLine("     First USN: '{0}'", _currentUsnState.FirstUsn);
                    Console.WriteLine("      Next USN: '{0}'", _currentUsnState.NextUsn);
                    Console.WriteLine("       Max USN: '{0}'", _currentUsnState.MaxUsn);
                }
                else
                {
                    lastWin32Error = (Win32Api.GetLastErrorEnum)Marshal.GetLastWin32Error();
                }
                return lastWin32Error;
            }
    
            private static bool IsUsnJournalValid()
            {
                Console.WriteLine("IsUsnJournalValid() entered for drive '{0}'", _driveInfo.Name);
    
                bool retval = true;
                //
                // is the JournalID from the previous state == JournalID from current state?
                //
                if (_previousUsnState.UsnJournalID == _currentUsnState.UsnJournalID)
                {
                    //
                    // is the next usn to process still available
                    //
                    if (_previousUsnState.NextUsn > _currentUsnState.FirstUsn && _previousUsnState.NextUsn < _currentUsnState.NextUsn)
                    {
                        retval = true;
                    }
                    else
                    {
                        retval = false;
                    }
                }
                else
                {
                    retval = false;
                }
                return retval;
            }
    
            private static void InitializeChangeJournal()
            {
                Console.WriteLine("----------\n  ReInitializing USN Journal for volume '{0}'", _driveInfo.Name);
                DateTime start = DateTime.Now;
    
                bool bSuccess = false;
                Win32Api.GetLastErrorEnum lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
                _folders.Clear();
    
                FileNameAndParentFrn fnapfrn = GetRootFrnEntry(_driveInfo.Name);
    
                if (fnapfrn != null)
                {
                    _folders.Add(fnapfrn.Frn, fnapfrn);
                }
                while(true)
                {
                    lastWin32Error = CreateUsnJournal();
                    if (lastWin32Error == Win32Api.GetLastErrorEnum.ERROR_JOURNAL_DELETE_IN_PROGRESS)
                    {
                        Console.WriteLine("CreateUsnJournal() {0}", lastWin32Error.ToString());
                        Thread.Sleep(100);
                        continue;
                    }
                    else if (lastWin32Error != Win32Api.GetLastErrorEnum.ERROR_SUCCESS)
                    {
                        Console.WriteLine("CreateUsnJournal() encountered an error - {0}", lastWin32Error.ToString());
                        bSuccess = false;
                        break;
                    }
                    else
                    {
                        bSuccess = true;
                        break;
                    }
                }
                if(bSuccess)
                {
                    lastWin32Error = QueryUsnJournal();
                    if (lastWin32Error == Win32Api.GetLastErrorEnum.ERROR_SUCCESS)
                    {
                        GatherNtfsVolumeDirectories();
                        //
                        // write folders out to folder file
                        //
                        UpdateUsnJournalInfo();
                    }
                    else
                    {
                        Console.WriteLine("QueryUsnJournal() encountered an error! {0}", lastWin32Error.ToString());
                    }
                }
                TimeSpan ts = DateTime.Now - start;
                Console.WriteLine("  Time to ReInitialize USN Journal: {0} secs.\n----------", ts.TotalSeconds);
            }
    
            private static FileNameAndParentFrn GetRootFrnEntry(string path)
            {
                Console.WriteLine("GetRootFrnEntry() function entered for drive '{0}'", path);
    
                FileNameAndParentFrn fnapfrn = null;
                string pathRoot = string.Concat("\\\\.\\", path);
                pathRoot = string.Concat(pathRoot, Path.DirectorySeparatorChar);
                IntPtr hRoot = Win32Api.CreateFile(pathRoot,
                    0,
                    Win32Api.FILE_SHARE_READ | Win32Api.FILE_SHARE_WRITE,
                    IntPtr.Zero,
                    Win32Api.OPEN_EXISTING,
                    Win32Api.FILE_FLAG_BACKUP_SEMANTICS,
                    IntPtr.Zero);
    
                if (hRoot.ToInt32() != Win32Api.INVALID_HANDLE_VALUE)
                {
                    Win32Api.BY_HANDLE_FILE_INFORMATION fi = new Win32Api.BY_HANDLE_FILE_INFORMATION();
                    bool bRtn = Win32Api.GetFileInformationByHandle(hRoot, out fi);
                    if (bRtn)
                    {
                        UInt64 fileIndexHigh = (UInt64)fi.FileIndexHigh;
                        UInt64 indexRoot = (fileIndexHigh << 32) | fi.FileIndexLow;
                        _volumeSerialNumber = fi.VolumeSerialNumber;
                        fnapfrn = new FileNameAndParentFrn(_volumeSerialNumber, _driveInfo.Name, indexRoot, 0);
                        Console.WriteLine("  Path root name '{0}' vol ser '{1}' frn '{2}', parent frn '{3}'",
                            fnapfrn.Name, fnapfrn.VolSer, _rootFrn.ToString("x"), fnapfrn.ParentFrn.ToString("x"));
                    }
                    else
                    {
                        throw new IOException("GetFileInformationbyHandle() returned invalid handle",
                            new Win32Exception(Marshal.GetLastWin32Error()));
                    }
                    Win32Api.CloseHandle(hRoot);
                }
                else
                {
                    throw new IOException("Unable to get root frn entry", new Win32Exception(Marshal.GetLastWin32Error()));
                }
                return fnapfrn;
            }
    
            public static void UpdateUsnJournalInfo()
            {
                WriteUsnJournalState();
                _previousUsnState = _currentUsnState;
                if (_bFoldersModified)
                {
                    WriteFoldersFile();
                }
            }
    
            private static void ReadUsnJournalState()
            {
                Console.WriteLine("ReadUsnJournalState() function entered for volume '{0}'", _driveInfo.Name);
    
                if (File.Exists(_usnJournalStateFullPath))
                {
                    FileStream usnJournalStateStream = null;
                    XmlTextReader reader = null;
    
                    try
                    {
                        usnJournalStateStream = File.Open(_usnJournalStateFullPath, FileMode.Open, FileAccess.Read, FileShare.None);
                        reader = new XmlTextReader(usnJournalStateStream);
                        reader.MoveToElement();
                        while (reader.Read())
                        {
                            if (0 != string.Compare(reader.LocalName, "usnjournalstate", true))
                            {
                                continue;
                            }
                            _previousVolSer = Convert.ToUInt32(reader.GetAttribute("volser"));
                            _volumeSerialNumber = _previousVolSer;
                            _previousUsnState.UsnJournalID = Convert.ToUInt64(reader.GetAttribute("journalid"), 16);
                            _previousUsnState.FirstUsn = Convert.ToInt64(reader.GetAttribute("firstusn"), 16);
                            _previousUsnState.NextUsn = Convert.ToInt64(reader.GetAttribute("nextusn"), 16);
                            _previousUsnState.MaxUsn = Convert.ToInt64(reader.GetAttribute("maxusn"), 16);
                            _previousUsnState.MaximumSize = Convert.ToUInt64(reader.GetAttribute("maxsize"), 16);
                            _previousUsnState.AllocationDelta = Convert.ToUInt64(reader.GetAttribute("allocationdelta"), 16);
                            _previousUsnState.LowestValidUsn = Convert.ToInt64(reader.GetAttribute("lowestvalidusn"), 16);
                            Console.WriteLine("  Previous usn journal state");
                            Console.WriteLine("       Vol Ser: '{0}'", _previousVolSer);
                            Console.WriteLine("    Journal ID: '{0}'", _previousUsnState.UsnJournalID);
                            Console.WriteLine("     First USN: '{0}'", _previousUsnState.FirstUsn);
                            Console.WriteLine("      Next USN: '{0}'", _previousUsnState.NextUsn);
                            Console.WriteLine("       Max USN: '{0}'", _previousUsnState.MaxUsn);
                        }
                    }
                    finally
                    {
                        if (reader != null)
                        {
                            reader.Close();
                        }
                        if (usnJournalStateStream != null)
                        {
                            usnJournalStateStream.Close();
                            usnJournalStateStream.Dispose();
                        }
                    }
                }
                else
                {
                    string msg = string.Format("  Usn Journal state file '{0}' does not exist", _usnJournalStateFullPath);
                    Console.WriteLine(msg);
                    throw new Exception(msg.TrimStart(' '));
                }
            }
    
            public static void Serialize()
            {
                //
                // serialize the usn journal state and folder information
                //
                WriteUsnJournalState();
                WriteFoldersFile();
            }
    
            private static void WriteUsnJournalState()
            {
                FileStream usnJournalStateStream = null;
                XmlTextWriter writer = null;
                if (File.Exists(_usnJournalStateFullPath))
                {
                    File.Delete(_usnJournalStateFullPath);
                }
                try
                {
                    usnJournalStateStream = File.Open(
                        _usnJournalStateFullPath,
                        FileMode.OpenOrCreate,
                        FileAccess.Write,
                        FileShare.None);
    
                    using (writer = new XmlTextWriter(usnJournalStateStream, Encoding.UTF8))
                    {
                        writer.Formatting = Formatting.Indented;
                        writer.WriteStartDocument();
                        writer.WriteStartElement("root");
                        writer.WriteStartElement("usnjournalstate");
                        writer.WriteAttributeString("volser", _volumeSerialNumber.ToString());
                        writer.WriteAttributeString("journalid", _currentUsnState.UsnJournalID.ToString("x"));
                        writer.WriteAttributeString("firstusn", _currentUsnState.FirstUsn.ToString("x"));
                        writer.WriteAttributeString("nextusn", _currentUsnState.NextUsn.ToString("x"));
                        writer.WriteAttributeString("maxusn", _currentUsnState.MaxUsn.ToString("x"));
                        writer.WriteAttributeString("maxsize", _currentUsnState.MaximumSize.ToString("x"));
                        writer.WriteAttributeString("allocationdelta", _currentUsnState.AllocationDelta.ToString("x"));
                        writer.WriteAttributeString("lowestvalidusn", _currentUsnState.LowestValidUsn.ToString("x"));
                        writer.WriteEndElement();
                        writer.WriteEndElement();
                        writer.WriteEndDocument();
                    }
                }
                catch (XmlException xmlExcptn)
                {
                    throw new Exception("Exception caught in WriteFoldersFile().", xmlExcptn);
                }
                finally
                {
                    if (writer != null)
                    {
                        writer.Close();
                    }
                    if (usnJournalStateStream != null)
                    {
                        usnJournalStateStream.Close();
                        usnJournalStateStream.Dispose();
                    }
                }
            }
    
            private static void WriteFoldersFile()
            {
                Console.WriteLine("WriteDirectoriesFile() function entered for drive '{0}'", _driveInfo.Name);
    
                FileStream foldersStream = null;
                if (_bFoldersModified)
                {
                    if (File.Exists(_mftFoldersFullPath))
                    {
                        File.Delete(_mftFoldersFullPath);
                    }
                    try
                    {
                        foldersStream = File.Open(
                            _mftFoldersFullPath,
                            FileMode.OpenOrCreate,
                            FileAccess.Write,
                            FileShare.None);
    
                        using (XmlTextWriter writer = new XmlTextWriter(foldersStream, Encoding.UTF8))
                        {
                            writer.Formatting = Formatting.Indented;
                            writer.WriteStartDocument();
                            writer.WriteStartElement("folders");
                            foreach (KeyValuePair<ulong, FileNameAndParentFrn> kvp in _folders)
                            {
                                FileNameAndParentFrn fnapfrn = kvp.Value;
                                writer.WriteStartElement("folder");
                                writer.WriteAttributeString("frn", kvp.Key.ToString("x"));
                                writer.WriteAttributeString("pfrn", fnapfrn.ParentFrn.ToString("x"));
                                writer.WriteAttributeString("nm", fnapfrn.Name);
                                writer.WriteEndElement();
                            }
                            writer.WriteEndElement();
                            writer.WriteEndDocument();
                        }
                    }
                    catch (XmlException xmlExcptn)
                    {
                        throw new Exception("Exception caught in WriteFoldersFile().", xmlExcptn);
                    }
                    Console.WriteLine("Total folders serialized to disk: {0}", _folders.Count);
                    _bFoldersModified = false;
                }
            }
    
            private static Win32Api.GetLastErrorEnum GatherNtfsVolumeDirectories()
            {
                Console.WriteLine("----------\n  Gathering Ntfs Volume Directories for drive '{0}'", _driveInfo.Name);
                DateTime start = DateTime.Now;
    
                Win32Api.GetLastErrorEnum lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
                //
                // set up MFT_ENUM_DATA structure
                //
                Win32Api.MFT_ENUM_DATA med;
                med.StartFileReferenceNumber = 0;
                med.LowUsn = 0;
                med.HighUsn = _currentUsnState.NextUsn;
                Int32 sizeMftEnumData = Marshal.SizeOf(med);
                IntPtr medBuffer = Marshal.AllocHGlobal(sizeMftEnumData);
                Win32Api.ZeroMemory(medBuffer, sizeMftEnumData);
                Marshal.StructureToPtr(med, medBuffer, true);
    
                //
                // set up the data buffer which receives the USN_RECORD data
                //
                int pDataSize = sizeof(UInt64) + 10000;
                IntPtr pData = Marshal.AllocHGlobal(pDataSize);
                Win32Api.ZeroMemory(pData, pDataSize);
                uint outBytesReturned = 0;
                Win32Api.USN_RECORD usn = null;
    
                //
                // Gather up volume's directories
                //
                while (false != Win32Api.DeviceIoControl(
                    _changeJournalRootHandle,
                    Win32Api.FSCTL_ENUM_USN_DATA,
                    medBuffer,
                    sizeMftEnumData,
                    pData,
                    pDataSize,
                    out outBytesReturned,
                    IntPtr.Zero))
                {
                    IntPtr pUsnRecord = new IntPtr(pData.ToInt32() + sizeof(Int64));
                    while (outBytesReturned > 60)
                    {
                        usn = new Win32Api.USN_RECORD(pUsnRecord);
                        //
                        // check for directory entries
                        //
                        if (0 != (usn.FileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
                        {
                            FileNameAndParentFrn directoryEntry =
                                new FileNameAndParentFrn(_volumeSerialNumber, usn.FileName,
                                    usn.FileReferenceNumber,
                                    usn.ParentFileReferenceNumber);
                            _folders.Add(usn.FileReferenceNumber, directoryEntry);
                            _bFoldersModified = true;
                        }
                        pUsnRecord = new IntPtr(pUsnRecord.ToInt32() + usn.RecordLength);
                        outBytesReturned -= usn.RecordLength;
                    }
                    Marshal.WriteInt64(medBuffer, Marshal.ReadInt64(pData, 0));
                }
                Marshal.FreeHGlobal(pData);
    
                Console.WriteLine("  Total Directories gathered from Usn Journal: {0}", _folders.Count);
                TimeSpan elapsedTime = DateTime.Now - start;
                Console.WriteLine("  Elapsed time for reading MFT: {0} secs.\n----------", elapsedTime.TotalSeconds);
    
                lastWin32Error = (Win32Api.GetLastErrorEnum)Marshal.GetLastWin32Error();
                if (lastWin32Error == Win32Api.GetLastErrorEnum.ERROR_SUCCESS || lastWin32Error == Win32Api.GetLastErrorEnum.ERROR_HANDLE_EOF)
                {
                    lastWin32Error = Win32Api.GetLastErrorEnum.ERROR_SUCCESS;
                }
                return lastWin32Error;
            }
    
            private static Win32Api.GetLastErrorEnum GatherUsnJournalChanges()
            {
                string statusMsg = string.Empty;
                DateTime start = DateTime.Now;
                TimeSpan result;
                bool bReadMore = true;
                //
                // update the usn journal state
                //
                Win32Api.GetLastErrorEnum lastWin32Error = QueryUsnJournal();
                if (lastWin32Error == Win32Api.GetLastErrorEnum.ERROR_SUCCESS)
                {
                    //
                    // sequentially process the usn journal looking for image file entries
                    //
                    int pbDataSize = sizeof(UInt64) * 0x4000;
                    IntPtr pbData = Marshal.AllocHGlobal(pbDataSize);
                    Win32Api.ZeroMemory(pbData, pbDataSize);
                    uint outBytesReturned = 0;
    
                    Win32Api.READ_USN_JOURNAL_DATA rujd = new Win32Api.READ_USN_JOURNAL_DATA();
                    rujd.StartUsn = _previousUsnState.NextUsn;
                    rujd.ReasonMask = Win32Api.USN_REASON_CLOSE |
                        Win32Api.USN_REASON_FILE_DELETE |
                        Win32Api.USN_REASON_RENAME_NEW_NAME |
                        Win32Api.USN_REASON_RENAME_OLD_NAME;
                    rujd.ReturnOnlyOnClose = 0;
                    rujd.Timeout = 0;
                    rujd.bytesToWaitFor = 0;
                    rujd.UsnJournalId = _previousUsnState.UsnJournalID;
    
                    int sizeRujd = Marshal.SizeOf(rujd);
                    IntPtr rujdBuffer = Marshal.AllocHGlobal(sizeRujd);
                    Win32Api.ZeroMemory(rujdBuffer, sizeRujd);
                    Marshal.StructureToPtr(rujd, rujdBuffer, true);
    
                    Win32Api.USN_RECORD usn = null;
    
                    //
                    // read usn journal entries
                    //
                    while (bReadMore)
                    {
                        bool bRtn = Win32Api.DeviceIoControl(
                            _changeJournalRootHandle,
                            Win32Api.FSCTL_READ_USN_JOURNAL,
                            rujdBuffer,
                            sizeRujd,
                            pbData,
                            pbDataSize,
                            out outBytesReturned,
                            IntPtr.Zero);
                        if (bRtn)
                        {
                            IntPtr pUsnRecord = new IntPtr(pbData.ToInt32() + sizeof(UInt64));
                            while (outBytesReturned > 60)   // while there are at least one entry in the usn journal
                            {
                                usn = new Win32Api.USN_RECORD(pUsnRecord);
                                if (usn.Usn >= _currentUsnState.NextUsn)      // only read until the current usn points beyond the current state's usn
                                {
                                    bReadMore = false;
                                    break;
                                }
                                if (0 == (usn.FileAttributes & Win32Api.FILE_ATTRIBUTE_DIRECTORY))
                                {
                                    //
                                    // handle files
                                    //
                                    HandleFiles(usn);
                                }
                                else
                                {
                                    //
                                    // handle folders/directories
                                    //
                                    HandleFolders(usn);
                                }
    
                                pUsnRecord = new IntPtr(pUsnRecord.ToInt32() + usn.RecordLength);
                                outBytesReturned -= usn.RecordLength;
    
                            }   // end while (outBytesReturned > 60) closing bracket
    
                        }   // end else closing bracket for if (bRtn)
                        else
                        {
                            lastWin32Error = (Win32Api.GetLastErrorEnum)Marshal.GetLastWin32Error();
                            Console.WriteLine("Error encountered reading USN Journal! {0}", lastWin32Error);
                            break;
                        }
    
                        Int64 nextUsn = Marshal.ReadInt64(pbData, 0);
                        if (nextUsn >= _currentUsnState.NextUsn)
                        {
                            break;
                        }
                        Marshal.WriteInt64(rujdBuffer, nextUsn);
    
                    }   // end while (bReadMore) closing bracket
    
                    Marshal.FreeHGlobal(rujdBuffer);
                    Marshal.FreeHGlobal(pbData);
    
                    result = DateTime.Now - start;
                    Console.WriteLine("  Time to read usn journal entries:{0} seconds", result.TotalSeconds);
                }
                else
                {
                    Console.WriteLine("Error encountered in QueryUsnJournal! {0}", lastWin32Error.ToString());
                }
                return lastWin32Error;
            }   // end GatherUsnJournalChanges() closing bracket
    
            private static void HandleFiles(Win32Api.USN_RECORD usn)
            {
                FileNameAndParentFrn fnapfrn = new FileNameAndParentFrn(_volumeSerialNumber, usn);
                if ((usn.Reason & Win32Api.USN_REASON_FILE_DELETE) != 0 ||
                    (usn.Reason & Win32Api.USN_REASON_RENAME_OLD_NAME) != 0)
                {
                    //
                    // if the reason code is delete or rename old add entry to the deleted image file list 
                    //
                    HandleDeletes(fnapfrn, usn.Reason);
                }
                else if ((usn.Reason & Win32Api.USN_REASON_RENAME_NEW_NAME) != 0)
                {
                    //
                    // if the reason code is 'rename new' handle the new files
                    //
                    HandleAdds(fnapfrn, usn.Reason);
                }
                else if ((usn.Reason & Win32Api.USN_REASON_CLOSE) != 0 &&
                        (usn.Reason & Win32Api.USN_REASON_FILE_CREATE) != 0)
                {
                    //
                    // if an image file has been copied in, we should see reasons for create and close.
                    // Handle as a new image file
                    //
                    HandleAdds(fnapfrn, usn.Reason);
                }
            }
    
            private static void HandleFolders(Win32Api.USN_RECORD usn)
            {
                ulong frn = usn.FileReferenceNumber;
                if ((usn.Reason & Win32Api.USN_REASON_FILE_DELETE) != 0 ||
                    (usn.Reason & Win32Api.USN_REASON_RENAME_OLD_NAME) != 0)
                {
                    _folders.Remove(frn);
                    _bFoldersModified = true;
                }
                else if ((usn.Reason & Win32Api.USN_REASON_RENAME_NEW_NAME) != 0 &&
                    (usn.Reason & Win32Api.USN_REASON_CLOSE) != 0)
                {
                    _folders.Remove(frn);
                    FileNameAndParentFrn fnapfrn =
                        new FileNameAndParentFrn(_volumeSerialNumber, usn.FileName,
                            frn, usn.ParentFileReferenceNumber);
                    _folders.Add(frn, fnapfrn);
                    _bFoldersModified = true;
                }
            }
    
            private static void HandleAdds(FileNameAndParentFrn fnapfrn, uint reason)
            {
                //
                // get the fully qualified path of the image file
                //
                string folder = GetFullyQualifiedPath(fnapfrn.ParentFrn).ToLower();
                //
                // send the FileNameAndParentFrn object to the image search engine to be added
                // to the image index.
                //
                Console.WriteLine("Add request");
                Console.WriteLine("  Path: {0}\n  Filename: {1}\n   frn:{2}\n  pfrn:{3}",
                            folder, fnapfrn.Name, fnapfrn.Frn, fnapfrn.ParentFrn);
    
                DisplayReasonCodes(folder, reason);
            }   // end MonitorFile() closing bracket
    
            private static void HandleDeletes(FileNameAndParentFrn fnapfrn, uint reason)
            {
                //
                // get the fully qualified path of the image file
                //
                string folder = GetFullyQualifiedPath(fnapfrn.ParentFrn).ToLower();
                //
                // send the FileNameAndParentFrn object to the image search engine to be removed 
                // from the image index.
                //
                Console.WriteLine("Delete request");
                Console.WriteLine("  Path: {0}\n  Filename: {1}\n   frn:{2}\n  pfrn:{3}",
                            folder, fnapfrn.Name, fnapfrn.Frn, fnapfrn.ParentFrn);
    
                DisplayReasonCodes(folder, reason);
            }
    
            public static string GetFullyQualifiedPath(UInt64 frn)
            {
                string retval = string.Empty; ;
                FileNameAndParentFrn fnFRN = null;
                if (frn >= 0)
                {
                    if (_folders.TryGetValue(frn, out fnFRN))
                    {
                        retval = fnFRN.Name;
                        while (fnFRN.ParentFrn != 0)
                        {
                            if (_folders.TryGetValue(fnFRN.ParentFrn, out fnFRN))
                            {
                                string name = fnFRN.Name;
                                retval = Path.Combine(name, retval);
                            }
                            else
                            {
                               Console.WriteLine("File or Directory's '{0}' parent frn not found", retval);
                                break;
                            }
                        }
                    }
                }
                else
                {
                    throw new ArgumentException("Invalid argument", "frn");
                }
                return retval;
            }
    
            private static void GetRootHandle()
            {
                Console.WriteLine("GetRootHandle() function entered for drive '{0}'", _driveInfo.Name);
    
                string vol = string.Concat("\\\\.\\", _driveInfo.Name.TrimEnd('\\'));
                _changeJournalRootHandle = Win32Api.CreateFile(vol,
                     Win32Api.GENERIC_READ | Win32Api.GENERIC_WRITE,
                     Win32Api.FILE_SHARE_READ | Win32Api.FILE_SHARE_WRITE,
                     IntPtr.Zero,
                     Win32Api.OPEN_EXISTING,
                     0,
                     IntPtr.Zero);
                if (_changeJournalRootHandle.ToInt32() == Win32Api.INVALID_HANDLE_VALUE)
                {
                    Exception excptn = new Win32Exception(Marshal.GetLastWin32Error());
                    throw new IOException("CreateFile() returned invalid handle", excptn);
                }
            }
    
            private static void DisplayReasonCodes(string fileName, uint reason)
            {
                Console.WriteLine("\n    Reason Code(s)");
                if ((reason & Win32Api.USN_REASON_BASIC_INFO_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_BASIC_INFO_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_CLOSE) != 0)
                {
                    Console.WriteLine("    USN_REASON_CLOSE");
                }
                if ((reason & Win32Api.USN_REASON_COMPRESSION_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_COMPRESSION_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_DATA_EXTEND) != 0)
                {
                    Console.WriteLine("    USN_REASON_DATA_EXTEND");
                }
                if ((reason & Win32Api.USN_REASON_DATA_OVERWRITE) != 0)
                {
                    Console.WriteLine("    USN_REASON_DATA_OVERWRITE");
                }
                if ((reason & Win32Api.USN_REASON_DATA_TRUNCATION) != 0)
                {
                    Console.WriteLine("    USN_REASON_DATA_TRUNCATION");
                }
                if ((reason & Win32Api.USN_REASON_EA_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_EA_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_ENCRYPTION_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_ENCRYPTION_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_FILE_CREATE) != 0)
                {
                    Console.WriteLine("    USN_REASON_FILE_CREATE");
                }
                if ((reason & Win32Api.USN_REASON_FILE_DELETE) != 0)
                {
                    Console.WriteLine("    USN_REASON_FILE_DELETE");
                }
                if ((reason & Win32Api.USN_REASON_HARD_LINK_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_HARD_LINK_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_INDEXABLE_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_INDEXABLE_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_NAMED_DATA_EXTEND) != 0)
                {
                    Console.WriteLine("    USN_REASON_NAMED_DATA_EXTEND");
                }
                if ((reason & Win32Api.USN_REASON_NAMED_DATA_OVERWRITE) != 0)
                {
                    Console.WriteLine("    USN_REASON_NAMED_DATA_OVERWRITE");
                }
                if ((reason & Win32Api.USN_REASON_NAMED_DATA_TRUNCATION) != 0)
                {
                    Console.WriteLine("    USN_REASON_NAMED_DATA_TRUNCATION");
                }
                if ((reason & Win32Api.USN_REASON_OBJECT_ID_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_OBJECT_ID_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_RENAME_NEW_NAME) != 0)
                {
                    Console.WriteLine("    USN_REASON_RENAME_NEW_NAME");
                }
                if ((reason & Win32Api.USN_REASON_RENAME_OLD_NAME) != 0)
                {
                    Console.WriteLine("    USN_REASON_RENAME_OLD_NAME");
                }
                if ((reason & Win32Api.USN_REASON_REPARSE_POINT_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_REPARSE_POINT_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_SECURITY_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_SECURITY_CHANGE");
                }
                if ((reason & Win32Api.USN_REASON_STREAM_CHANGE) != 0)
                {
                    Console.WriteLine("    USN_REASON_STREAM_CHANGE");
                }
    
            }   // DisplayReasonCodes() closing bracket
    
            private static void DisplayException(Exception excptn)
            {
                string padding = string.Empty;
                while (excptn != null)
                {
                    Console.WriteLine("{0}{1}", padding, excptn.Message);
                    padding = "   ";
                    excptn = excptn.InnerException;
                }
            }
        }
    }
    

    What I've provided so far is C# code that accesses and manipulates the USN Journal. I'm in the process of designing a UsnJournal class which uses some of this code but provides a class that can be used to ask the USN Journal for all the changed, deleted and new files.  I'll also write a WPF front end that uses this class and allows you to see exactly what the USN Journal can tell you about the modifications that occur on a volume.


    StCroixSkipper
    Friday, April 09, 2010 5:22 PM
  • Skipper, you are a Juggernaut! One day a monument will be installed on St Creux shore. It will read: In memory of the great navigator who while sailing these waters crushed the most daunting problem in Windows Programming - MFT:)

     


    AlexB - Win_7 Pro64, SqlSer64 WinSer64
    Friday, April 09, 2010 7:12 PM
  • It will be interesting to see who else chimes in... 

    I'd like to get a community of folks who are interested in using the USN Journal an maybe post code and get it reviewed then make it available.


    StCroixSkipper
    Saturday, April 10, 2010 1:42 AM
  • Good Work StCroixSkipper... When you mentioned the padding of the 4bytes for the DELETE_USN_JOURNAL_DATA Structure. I think you may have found my post on the documentation here http://msdn.microsoft.com/en-us/library/aa363928(VS.85).aspx#ctl00_WikiContent_ctl00_History

    I added this information on 12/16/2008 10:27 AM by eriklitze

    We could start a project on MS CodePlex. I would be more than willing to contribute what I have observed over the past years working with the journal.

    Sunday, April 11, 2010 3:51 PM
  • That 4 bytes of padding drove me crazy for a week until I found your post.

    I would welcome any and all input! The more input and review the better. 

    What is the consensus?  Should we create a new thread?  Where should we create it?  Here, CodePlex, Somewhere else? My choice would be the place where we can get the most participation.  I've tried DreamInCode but no one there will even comment to express interest.  I can submit code all day long and I get nada back.


    StCroixSkipper
    Sunday, April 11, 2010 10:11 PM
  • Here are my ideas for the public interface for a UsnJournal object.

    I want to hide the complexity of dealing with the Usn Journal as much as possible without hiding too much.

    I can write the code so that all of the errors calling DeviceIoControl are handled within the object.  I'd rather not throw exceptions.  Instead, I would return success/error codes.

    So the only error codes that makes sense to the user are

            public enum UsnJournalReturnCode
            {
                USN_JOURNAL_SUCCESS = 0,
                USN_JOURNAL_INVALID = 1,
                USN_JOURNAL_NOT_ACTIVE = 2,
                VOLUME_NOT_NTFS = 3
            }
    

    I've thought about calling the object a 'Volume'.  The other alternative is calling it UsnJournal.  I'm only dealing with one kind of volume, NTFS.  The only added value 'Volume' would bring is the data in the DriveInfo object, name, available freespace, total freespace, format, total size, and volume label.  I can expose all the DriveInfo fields by making it the DriveInfo object a member variable of UsnJournal.

    Here is the class:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO;
    using PInvoke;
    
    namespace ConsoleUSNJournalProject
    {
        class NtfsUsnJournal
        {
            #region enum(s)
            public enum UsnJournalReturnCode
            {
                USN_JOURNAL_SUCCESS = 0,
                USN_JOURNAL_INVALID = 1,
                USN_JOURNAL_NOT_ACTIVE = 2,
                VOLUME_NOT_NTFS = 3
            }
            #endregion
    
            #region private member variables
    
            private DriveInfo _driveInfo = null;
            private UInt64 _usnJournalId = 0;
    
            #endregion
    
            #region properties
    
            public string VolumeName
            {
                get { return _driveInfo.Name; }
            }
    
            public long AvailableFreeSpace
            {
                get { return _driveInfo.AvailableFreeSpace; }
            }
    
            public long TotalFreeSpace
            {
                get { return _driveInfo.TotalFreeSpace; }
            }
    
            public string Format
            {
                get { return _driveInfo.DriveFormat; }
            }
    
            public DirectoryInfo RootDirectory
            {
                get { return _driveInfo.RootDirectory; }
            }
    
            public long TotalSize
            {
                get { return _driveInfo.TotalSize; }
            }
    
            public string VolumeLabel
            {
                get { return _driveInfo.VolumeLabel; }
            }
    
            #endregion
    
            #region constructor(s)
    
            public NtfsUsnJournal(DriveInfo driveInfo)
            {
                _driveInfo = driveInfo;
            }
    
            #endregion
    
            #region public methods
    
            public UsnJournalReturnCode CreateUsnJournal(int maxSize, int allocationDelta)
            {
                //
                // caller should be able to change the size and allocation delta...
                //
            }
    
            public UsnJournalReturnCode DeleteUsnJournal()
            {
                //
                // I'm not sure about this one -- what do you think?
                // My take is if they can create it, they should be able to delete it.
                //
            }
    
            public bool IsUsnJournalValid()
            {
            }
    
            public bool IsUsnJournalActive()
            {
            }
    
            public UsnJournalReturnCode GetUsnJournalState(out Win32Api.USN_JOURNAL_DATA usnJournalState)
            {
            }
    
            public UsnJournalReturnCode GetChanges(Win32Api.USN_JOURNAL_DATA previousUsnState, out List<VolumeChangeEntry> changes, out Win32Api.USN_JOURNAL_DATA currentState)
            {
                //
                // each time the user calls for the changes it process usn journal entries from the 
                // previous state up to the current state.  So this info doesn't get misinterpreted
                // I think this should be one function rather than one function to get new files or 
                // directories, one function to get modified files, one function to get deleted 
                // files/directories.  
                //
                // this function would determine that the volume is NTFS,
                // it would read the current state,
                // that depending on the previousUsnState and the current state
                //   it would determine if the usn journal is valid
                // then it would process all the entries from the previous state to the current
                // state to load up the lists of new, modified, and deleted filesystem objects.
                //
                // I've thought about creating a UsnJournalEntry class that has the type of filesystem
                // object (file/folder), the change type (new, modified, deleted), etc., so that I could 
                // pass back one list, i.e. List<UsnJournalEntry>
                //
                // I'm leaning toward the three lists.
                //
            }
    
            #endregion
    
            #region private member functions
    
            #endregion
        }
    }
    

     

    If I returned one list of UsnJournalEntry's here is what that class might look like:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    
    using System.IO;
    
    namespace ConsoleUSNJournalProject
    {
        public class UsnChangeEntry
        {
            #region enums
    
            public enum FolderOrFile
            {
                FOLDER = 0,
                FILE = 1
            }
    
            public enum NewChangedDeleted
            {
                NEW = 0,
                CHANGED = 1,
                DELETED = 2
            }
    
            #endregion
    
            #region properties
    
            private string _name;
            public string Name
            {
                get { return _name; }
            }
    
            private string _path;
            public string Path
            {
                get { return _path; }
            }
    
            public string FullyQualifiedPath
            {
                get { return System.IO.Path.Combine(_path, _name); }
            }
    
            private FolderOrFile _fileType;
            public FolderOrFile FileType
            {
                get { return _fileType; }
            }
    
            private NewChangedDeleted _changeType;
            public NewChangedDeleted ChangeType
            {
                get { return _changeType; }
            }
    
            #endregion
    
            #region private member variables
            #endregion
    
            #region constructor(s)
            #endregion
    
            #region public member functions
            #endregion
    
            #region private member functions
            #endregion
        }
    }
    

     

    Comments anyone? Suggestions, criticisms, alternatives?


    StCroixSkipper
    Sunday, April 11, 2010 10:48 PM
  • Also, I know there is an api to get a filename given a 64-bit file reference number.  I've used it in C++ before.  But for the life of me, I can't find it today.  I really want to get rid of the memory and effort of keeping the list of folders on the volume up-to-date.

    I know I can code for the one problem area.

    Which is if someone deletes a directory, all of the files and the directory are deleted and usn deleted entries are recorded in the usn journal.  But by the time you are asked to process these entries, the files' parent frn in the deleted directory will point to a non-existant folder -- its been deleted!.  So you have to gather all of the deleted directories into a list while you are processing the usn journal entries and resolve these at the end into fully qualified paths.

    So, does anyone know/remember how to get file name of a filesystem object given the frn?


    StCroixSkipper
    Sunday, April 11, 2010 10:59 PM
  • Private Function PathFromFrn(ByVal Id As Long) As String
    
            Dim fOk As Integer
            Dim FileName As String = String.Empty
            Dim UnicodeString As UNICODE_STRING
            Dim ObjAttributes As OBJECT_ATTRIBUTES
            Dim IoStatusBlock As IO_STATUS_BLOCK
            Dim hFile As IntPtr ' out handle 
            Dim Buffer As IntPtr = Marshal.AllocHGlobal(4096) ' Raw buffer
            Dim Refptr As IntPtr = Marshal.AllocHGlobal(8) ' 8 byte FileID
            Dim ObjAtt As IntPtr = Marshal.AllocHGlobal(Marshal.SizeOf(ObjAttributes)) 'pointer to the unicode string struct
    
            ' pointer>>fileid
            Marshal.WriteInt64(Refptr, 0, Id)
    
            ' 8 byte file id
            UnicodeString.Length = 8
            UnicodeString.MaximumLength = 8
            UnicodeString.Buffer = Refptr
    
            ' copy unicode structure to pointer
            Marshal.StructureToPtr(UnicodeString, ObjAtt, True)
    
            ' InitializeObjectAttributes Macro
            ObjAttributes.Length = Marshal.SizeOf(ObjAttributes)
            ObjAttributes.ObjectName = ObjAtt
            ObjAttributes.RootDirectory = m_hCJ
            ObjAttributes.Attributes = OBJ_CASE_INSENSITIVE
    
            fOk = NtCreateFile(hFile, GENERIC_READ, ObjAttributes, IoStatusBlock, 0, 0, _
                               FILE_SHARE_READ Or FILE_SHARE_WRITE, _
                               FILE_OPEN, FILE_OPEN_BY_FILE_ID Or FILE_OPEN_FOR_BACKUP_INTENT, 0, 0)
    
            If fOk <> INVALID_HANDLE_VALUE Then
    
                fOk = NtQueryInformationFile(hFile, IoStatusBlock, Buffer, 4096, FileNameInformationClass)
    
                If fOk = 0 Then
    
                    ' The first 4 bytes is the length
                    Dim FileLength As Integer = Marshal.ReadInt32(Buffer, 0)
                    ' The filename is just after the first 4 bytes.
                    FileName = Marshal.PtrToStringUni(New IntPtr(Buffer.ToInt32() + 4), FileLength / 2)
    
                End If
    
            End If
            ' free allocated memory and handles
            CloseHandle(hFile)
            Marshal.FreeHGlobal(Buffer)
            Marshal.FreeHGlobal(ObjAtt)
            Marshal.FreeHGlobal(Refptr)
    
            Return FileName
    
        End Function
    
    Monday, April 12, 2010 2:55 PM
  •     Private Const FILE_FLAG_BACKUP_SEMANTICS = &H2000000
        Private Const FileNameInformationClass = 9
        Private Const FILE_OPEN_FOR_BACKUP_INTENT = &H4000
        Private Const FILE_OPEN_BY_FILE_ID = &H2000
        Private Const FILE_OPEN = &H1
        Private Const OBJ_CASE_INSENSITIVE = &H40
        'Private Const OBJ_KERNEL_HANDLE = &H200
    
    <DllImport("ntdll.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function NtCreateFile(ByRef FileHandle As IntPtr, ByVal DesiredAccess As Integer, ByRef ObjectAttributes As OBJECT_ATTRIBUTES, ByRef IoStatusBlock As IO_STATUS_BLOCK, ByVal AllocationSize As Integer, ByVal FileAttribs As Integer, ByVal SharedAccess As Integer, ByVal CreationDisposition As Integer, ByVal CreateOptions As Integer, ByVal EaBuffer As Integer, ByVal EaLength As Integer) As Integer
        End Function
    
        <DllImport("ntdll.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function NtQueryInformationFile(ByVal FileHandle As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, ByVal FileInformation As IntPtr, ByVal Length As Integer, ByVal FileInformationClass As Integer) As Integer
        End Function
    
    
        Private Structure IO_STATUS_BLOCK
            Dim Status As Integer
            Dim Information As Integer
        End Structure
    
        Private Structure UNICODE_STRING
            Dim Length As Short
            Dim MaximumLength As Short
            Dim Buffer As IntPtr
        End Structure
    
        Private Structure OBJECT_ATTRIBUTES
            Dim Length As Integer
            Dim RootDirectory As IntPtr
            Dim ObjectName As IntPtr
            Dim Attributes As Integer
            Dim SecurityDescriptor As Integer
            Dim SecurityQualityOfService As Integer
        End Structure
    Monday, April 12, 2010 3:17 PM
  • Well it looks like my editing removed what I had replied. In any case CodePlex would be a good start but one first need to have a basis project that works before publishing it on CodePlex. You can contact me directly via mail @live.com

    I typically reference the USN as a (MagicUSN). There should be a design layout if it will be a Class and it needs to be thread safe so some considerations need to be made. Also I am VB.NET guy who doesn't use C# however I have don't have problems reading C++ and C# code.

    Monday, April 12, 2010 3:29 PM
  • Since the data is stored in the USN Journal and the user simply provides the state of the USN Journal when he last requested information from it, I don't think there is a thread safety issue.  This wouldn't be a singleton and the USN Journal is persistent on the volume.

    Before I supply a working project on CodePlex, I still need to solve the issue with getting a filename given a file reference number.  It annoys me to no end to know I've seen and used it before...

    The only question I have about CodePlex is whether we'll get the kind of review we want?  Will enough folks visit the project who have a serious enough interest that they'll contribute?


    StCroixSkipper
    Monday, April 12, 2010 6:23 PM
  • Hi StCroixSkipper,

    I have posted about your issue with obtaining the filename from the filereference number. Did you see the post or do you have an issue with converting it to c# code?

    What I meant by implenting the design in thread safe manner is that if you decide to implement any type of events or wait for the journal delete (progress) on another thread that the event should be signaled on the same thread the user initated the call. This all depends what you want to expose to the user of course for example if you wanted to allow the user to know when a journal was deleted.

    OnJournalDeleteCompleted()

    Lets just say for example I use a windows form to use the class. I call Journal.Delete() from the UI thread(10).. The class creates a thread to wait for the pending delete to finish.. When it's complete you must raise this event on the same thread it was called. (10). The easiest way to handle these features is with AsyncOperationManager() and AsyncOperation pattern flow.

    CodePlex is just a place to get people actively working on the same open source projects. Obviously right now it appears only you and I are interested in such a project if thats the case then it's a two man job but it's always available to recurit anyone who is interested in the project.

    -ErikLitze

     

     

    Monday, April 12, 2010 7:59 PM
  • Erik,

    No problem converting your code to C#.  I should have known it was an Nt-funtion.

    With that out of the way, what do you think about the interface so far?  I could implement this quickly and we'd at least have something to work with.

    Once we get a working copy, maybe then we can move to CodePlex.


    StCroixSkipper
    Tuesday, April 13, 2010 2:09 PM
  • Also, I have another interest.  I've written code to use the Volume Shadowcopy Service (VSS) to create shadowcopies of volumes to enable backup up open files and applications like Exchange.  I would like to contribute to a VSS project too.  Is any interest in Volume Shadowcopy Service Requestor code?
    StCroixSkipper
    Tuesday, April 13, 2010 2:21 PM
  • The Base class is completley fine. The only thing I would change is to remove things that can already be done with the .NET framework and keep the focus directly on the Change Journal. The class should be robust and never do to much work as to slow down the main thread for this purpose it's probrably best that we just use full blow asynchronous calls with DeviceIoControl(). However some methods could still be used Synchronously such as (Creating,Querying) the journal. Implement the Reading of the journal asynchronously by opening the volume handle with FILE_FLAG_OVERLAPPED. This could avoid the "polling" technique all together. The idea is that it should be "lightweight" and never use more than one thread and a couple of thread pool threads.

    One thing I have done in the past as a personal project was to allow the ability to create profiles. These profiles are just basic XML files that a user could customize to return some statistic information.

    I have had a couple things in mind that might be easier to deal with for both the developer and users.

    Instead of allowing the users to directly use the class to manipulate the journal we could take a different approach and create a service. Here is the basic idea behind this concept.

    1) Create one class that is private internally to handle the journal. Using that class inside the service to manage every change on volume(s) and storing this information in a local database.

    2) Create a class that the user can use to simply read from the database. This gives the ability for the user easily use (database) routines to read the changes. This is what should be exposed to the user. It would also make it easier to implement types of custom profiles of files / folders of interest by a simple query.

    The idea here is that if it's implemented as a service that a user can install then just use the basic class to read from the database might be the best approach. This also makes it easier for both the developer and user. The reason is because once the service is completed and fully functional the user class should be the only class that need to be updated to add more functionality.

    The fact that the journal needs administrative rights I think this might be the best approach to the design. Take some time to think about it and im sure you understand it might make sense in the long run.

    CodePlex: Description could read.

    JournalMagic is a service that runs on your computer to make it easier to manage the Change journal and MFT records on all NTFS filesystems.

    Also we don't need to use CodePlex.com there is plenty of Free services or small fee web service providers that we could publish the application.

    Let me know if you like this approach or if you want to keep it just a basic class.

    BTW: contact me directly via  (I don't bite) if you want faster responses it's easier than checking this forum. :)

    -ErikLitze

     

     

    Tuesday, April 13, 2010 3:19 PM
  • I don't have much background with the VSS services. I wouldn't be able to help with this feature. I do have background with basic backup routines such as BackupRead() BackupWrite() allowing reading and writing to any file even with security permissions enabled. The only problem is processes that have exclusively locked a file or a files byte range. This is where the VSS service is more appropriate.
    Tuesday, April 13, 2010 3:37 PM
  • This is a suggestion from a friend of mine, Rick Winter. 

    Check out this link:

     

    http://msdn.microsoft.com/en-us/library/aa364953(v=VS.85).aspx

     

    You need to open the file by ID (http://msdn.microsoft.com/en-us/library/aa365432(VS.85).aspx) and then get the name using FileNameInfo with GetFileInformationByHandleEx.

     


    StCroixSkipper
    Tuesday, April 13, 2010 5:10 PM
  • Your friend is absolutely on the right track and correct however they are basically doing the same thing as the above snippet I posted. The only difference is that the example I posted shouldn't be limited to Vista. The GetFileInformationByHandleEx and OpenByFileId looks to exported on Vista and above.

     

    Tuesday, April 13, 2010 5:25 PM
  • Here is a very small example of using the PathFromFrn() to enumerate the MFT with path information. What I did was use ParentFileReference and append the filename.

    BTW: It will be slow if you output to the debug window and it's currently synchrounous example.

    You can easily use it like this:

     

    Dim myMFT As New EnumMFT
    myMFT.FindAllFiles(
    "E:")

    Imports System.Runtime.InteropServices
    
    Public Class EnumMFT
    
        Private Const INVALID_HANDLE_VALUE = (-1)
        Private Const GENERIC_READ = &H80000000
        Private Const FILE_SHARE_READ = &H1
        Private Const FILE_SHARE_WRITE = &H2
        Private Const OPEN_EXISTING = 3
        Private Const FILE_READ_ATTRIBUTES = &H80
        Private Const FileNameInformationClass = 9
        Private Const FILE_FLAG_BACKUP_SEMANTICS = &H2000000
        Private Const FILE_OPEN_FOR_BACKUP_INTENT = &H4000
        Private Const FILE_OPEN_BY_FILE_ID = &H2000
        Private Const FILE_OPEN = &H1
        Private Const OBJ_CASE_INSENSITIVE = &H40
    
    
        Private Const FSCTL_ENUM_USN_DATA = &H900B3
    
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure MFT_ENUM_DATA
            Dim StartFileReferenceNumber As Long
            Dim LowUsn As Long
            Dim HighUsn As Long
        End Structure
    
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure USN_RECORD
            Dim RecordLength As Integer
            Dim MajorVersion As Short
            Dim MinorVersion As Short
            Dim FileReferenceNumber As Long
            Dim ParentFileReferenceNumber As Long
            Dim Usn As Long
            Dim TimeStamp As Long
            Dim Reason As Integer
            Dim SourceInfo As Integer
            Dim SecurityId As Integer
            Dim FileAttributes As Integer
            Dim FileNameLength As Short
            Dim FileNameOffset As Short
        End Structure
    
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure IO_STATUS_BLOCK
            Dim Status As Integer
            Dim Information As Integer
        End Structure
    
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure UNICODE_STRING
            Dim Length As Short
            Dim MaximumLength As Short
            Dim Buffer As IntPtr
        End Structure
    
        <StructLayout(LayoutKind.Sequential)> _
        Private Structure OBJECT_ATTRIBUTES
            Dim Length As Integer
            Dim RootDirectory As IntPtr
            Dim ObjectName As IntPtr
            Dim Attributes As Integer
            Dim SecurityDescriptor As Integer
            Dim SecurityQualityOfService As Integer
        End Structure
    
        '// MFT_ENUM_DATA
        <DllImport("kernel32.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function DeviceIoControl(ByVal hDevice As IntPtr, ByVal dwIoControlCode As Integer, ByRef lpInBuffer As MFT_ENUM_DATA, ByVal nInBufferSize As Integer, ByVal lpOutBuffer As IntPtr, ByVal nOutBufferSize As Integer, ByRef lpBytesReturned As Integer, ByVal lpOverlapped As IntPtr) As Boolean
        End Function
    
        <DllImport("kernel32.dll", SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function CreateFile(ByVal lpFileName As String, ByVal dwDesiredAccess As Integer, ByVal dwShareMode As Integer, ByVal lpSecurityAttributes As IntPtr, ByVal dwCreationDisposition As Integer, ByVal dwFlagsAndAttributes As Integer, ByVal hTemplateFile As IntPtr) As IntPtr
        End Function
    
        <DllImport("kernel32.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function CloseHandle(ByVal lpObject As IntPtr) As Int32
        End Function
    
        <DllImport("ntdll.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function NtCreateFile(ByRef FileHandle As IntPtr, ByVal DesiredAccess As Integer, ByRef ObjectAttributes As OBJECT_ATTRIBUTES, ByRef IoStatusBlock As IO_STATUS_BLOCK, ByVal AllocationSize As Integer, ByVal FileAttribs As Integer, ByVal SharedAccess As Integer, ByVal CreationDisposition As Integer, ByVal CreateOptions As Integer, ByVal EaBuffer As Integer, ByVal EaLength As Integer) As Integer
        End Function
    
        <DllImport("ntdll.dll", ExactSpelling:=True, SetLastError:=True, CharSet:=CharSet.Auto)> _
        Private Shared Function NtQueryInformationFile(ByVal FileHandle As IntPtr, ByRef IoStatusBlock As IO_STATUS_BLOCK, ByVal FileInformation As IntPtr, ByVal Length As Integer, ByVal FileInformationClass As Integer) As Integer
        End Function
    
        Private m_hCJ As IntPtr
        Private m_Buffer As IntPtr
        Private m_BufferSize As Integer
        Private m_DriveLetter As String
    
        Private Function OpenVolume(ByVal szDriveLetter As String) As IntPtr
    
            Dim hCJ As IntPtr '// volume handle
    
            m_DriveLetter = szDriveLetter
    
            hCJ = CreateFile("\\.\" & szDriveLetter, GENERIC_READ, _
                             FILE_SHARE_READ Or FILE_SHARE_WRITE, IntPtr.Zero, OPEN_EXISTING, 0, 0)
    
            Return hCJ
    
        End Function
    
        Private Sub Cleanup()
    
            If m_hCJ <> 0 Then
                ' Close the volume handle.
                CloseHandle(m_hCJ)
                m_hCJ = INVALID_HANDLE_VALUE
            End If
    
            If m_Buffer <> 0 Then
                ' Free the allocated memory
                Marshal.FreeHGlobal(m_Buffer)
                m_Buffer = IntPtr.Zero
            End If
    
        End Sub
    
        Public Sub FindAllFiles(ByVal szDriveLetter As String)
    
            Dim usnRecord As USN_RECORD
            Dim mft As MFT_ENUM_DATA
            Dim dwRetBytes As Integer
            Dim cb As Integer
    
            ' This shouldn't be called more than once.
            If m_Buffer.ToInt32 <> 0 Then
                Console.WriteLine("invalid buffer")
                Exit Sub
            End If
    
            ' Assign buffer size
            m_BufferSize = 65536 '64KB
    
            ' Allocate a buffer to use for reading records.
            m_Buffer = Marshal.AllocHGlobal(m_BufferSize)
    
            ' Open the volume handle 
            m_hCJ = OpenVolume(szDriveLetter)
    
            ' Check if the volume handle is valid.
            If m_hCJ = INVALID_HANDLE_VALUE Then
                Console.WriteLine("Couldn't open handle to the volume.")
                Cleanup()
                Exit Sub
            End If
    
            mft.StartFileReferenceNumber = 0
            mft.LowUsn = 0
            mft.HighUsn = Long.MaxValue
    
            Do
                If DeviceIoControl(m_hCJ, FSCTL_ENUM_USN_DATA, mft, Marshal.SizeOf(mft), m_Buffer, m_BufferSize, dwRetBytes, IntPtr.Zero) Then
                    cb = dwRetBytes
                    ' Pointer to the first record
                    Dim pUsnRecord As New IntPtr(m_Buffer.ToInt32() + 8)
    
                    While (dwRetBytes > 8)
                        ' Copy pointer to USN_RECORD structure.
                        usnRecord = Marshal.PtrToStructure(pUsnRecord, usnRecord.GetType)
                        ' The filename within the USN_RECORD.
                        Dim FileName As String = Marshal.PtrToStringUni(New IntPtr(pUsnRecord.ToInt32() + usnRecord.FileNameOffset), _
                                                                        usnRecord.FileNameLength / 2)
    
    
                        If usnRecord.FileAttributes And vbDirectory Then
                            Dim dirPath As String = PathFromFrn(usnRecord.ParentFileReferenceNumber)
                            ' directory
                            Console.WriteLine("Directory {0} ", szDriveLetter & dirPath)
                        Else
                            ' files
                            Dim filePath As String = PathFromFrn(usnRecord.ParentFileReferenceNumber) & "\" & FileName
                            Console.WriteLine("File {0} ", szDriveLetter & filePath)
                        End If
    
                        ' Pointer to the next record in the buffer.
                        pUsnRecord = New IntPtr(pUsnRecord.ToInt32() + usnRecord.RecordLength)
    
                        dwRetBytes -= usnRecord.RecordLength
    
                    End While
    
                    ' The first 8 bytes is always the start of the next USN.
                    mft.StartFileReferenceNumber = Marshal.ReadInt64(m_Buffer, 0)
    
                Else
    
                    Exit Do
    
                End If ' DeviceIoControl)
    
            Loop Until cb <= 8
    
            '// cleanup
            Cleanup()
            Console.WriteLine("Done")
    
        End Sub
    
    
    
        Private Function PathFromFrn(ByVal Id As Long) As String
    
            Dim fOk As Integer
            Dim FileName As String = String.Empty
            Dim UnicodeString As UNICODE_STRING
            Dim ObjAttributes As OBJECT_ATTRIBUTES
            Dim IoStatusBlock As IO_STATUS_BLOCK
            Dim hFile As IntPtr ' out handle 
            Dim Buffer As IntPtr = Marshal.AllocHGlobal(4096) ' Raw buffer
            Dim Refptr As IntPtr = Marshal.AllocHGlobal(8) ' 8 byte FileID
            Dim ObjAtt As IntPtr = Marshal.AllocHGlobal(Marshal.SizeOf(ObjAttributes)) 'pointer to the unicode string struct
    
            ' pointer>>fileid
            Marshal.WriteInt64(Refptr, 0, Id)
    
            ' 8 byte file id
            UnicodeString.Length = 8
            UnicodeString.MaximumLength = 8
            UnicodeString.Buffer = Refptr
    
            ' copy unicode structure to pointer
            Marshal.StructureToPtr(UnicodeString, ObjAtt, True)
    
            ' InitializeObjectAttributes Macro
            ObjAttributes.Length = Marshal.SizeOf(ObjAttributes)
            ObjAttributes.ObjectName = ObjAtt
            ObjAttributes.RootDirectory = m_hCJ
            ObjAttributes.Attributes = OBJ_CASE_INSENSITIVE
    
            fOk = NtCreateFile(hFile, 0, ObjAttributes, IoStatusBlock, 0, 0, _
                               FILE_SHARE_READ Or FILE_SHARE_WRITE, _
                               FILE_OPEN, FILE_OPEN_BY_FILE_ID Or FILE_OPEN_FOR_BACKUP_INTENT, 0, 0)
    
            If fOk <> INVALID_HANDLE_VALUE Then
    
                fOk = NtQueryInformationFile(hFile, IoStatusBlock, Buffer, 4096, FileNameInformationClass)
    
                If fOk = 0 Then
    
                    ' The first 4 bytes is the length
                    Dim FileLength As Integer = Marshal.ReadInt32(Buffer, 0)
                    ' The filename is just after the first 4 bytes.
                    FileName = Marshal.PtrToStringUni(New IntPtr(Buffer.ToInt32() + 4), FileLength / 2)
    
                End If
    
            End If
            ' free allocated memory and handles
            CloseHandle(hFile)
            Marshal.FreeHGlobal(Buffer)
            Marshal.FreeHGlobal(ObjAtt)
            Marshal.FreeHGlobal(Refptr)
    
    
            Return FileName
        End Function
    
    End Class
    
    Tuesday, April 13, 2010 6:09 PM
  • I found 'OpenFileById' and 'GetFileInformationByHandleEx' in Kernel32.dll and it says "WinBase.h (include Windows.h), FileExtd.h on Windows Server 2003 and Windows XP".

    So it appears we could get it to work on XP and Server 2003 by creating a dll for FileExtd.lib.  I'm using NtCreateFile, ... for now.


    StCroixSkipper
    Wednesday, April 14, 2010 3:14 PM
  • Also, I have another interest.  I've written code to use the Volume Shadowcopy Service (VSS) to create shadowcopies of volumes to enable backup up open files and applications like Exchange.  I would like to contribute to a VSS project too.  Is any interest in Volume Shadowcopy Service Requestor code?
    StCroixSkipper

    I've been monitoring the thread. Sure, it is very interesting. If you post the code I will archive it for a rainy day.

    Thanks.


    AlexB - Win_7 Pro64, SqlSer64 WinSer64
    Thursday, April 15, 2010 4:43 PM
  • Me too i'm following this ...

    highly interesting

     

    what i'm wondering is why not read the raw mft sectors of the volume and convert those bits and bytes... i guess that's even the fastest possible?

    i want to write C# code what does mft raw sector conversion


    signature
    Wednesday, April 21, 2010 4:58 PM
  • I'm not sure I understand your comment.  The DeviceIoControl() with FSCTL_READ_USN_JOURNAL reads the raw bytes of the journal.

    bool bRtn = Win32Api.DeviceIoControl(
                            _changeJournalRootHandle,
                            Win32Api.FSCTL_READ_USN_JOURNAL,
                            rujdBuffer,
                            sizeRujd,
                            pbData,
                            pbDataSize,
                            out outBytesReturned,
                            IntPtr.Zero);

    And the statement:  usn = new Win32Api.USN_RECORD(pUsnRecord); converts the bits from the raw read into a USN_RECORD in its constructor.  I know I can do it a little faster in C++ because I can just cast the memory to the structure. 

    Other than that, I'd love to find a faster way but I don't see one.  Even reading the raw sparse file.

    On another topic, I've been side tracked these last few days working on another project.  But I've been thinking about what I would like the ideal UsnJournal object to look like.  In resolving the issues between removing the complexity and still exposing the functionality of the actual USN Journal, I've decided I minimally want my UsnJournal object to expose the ability for the user to specify which kinds of entries he/she wants to see.

    This means giving the caller the ability to specify ReasonMask and RetrunOnlyOnClose and perhaps BytesToWaitFor in the typedef struct {
      USN       StartUsn;
      DWORD     ReasonMask;
      DWORD     ReturnOnlyOnClose;
      DWORDLONG Timeout;
      DWORDLONG BytesToWaitFor;
      DWORDLONG UsnJournalID;
    } READ_USN_JOURNAL_DATA, *PREAD_USN_JOURNAL_DATA;

    When I get a working copy of the code, I'll post it here.


    StCroixSkipper
    Thursday, April 22, 2010 2:43 PM
  • Here it is!  StCroixSkipper's USN Journal Explorer version 1.0.

    I would post the code here but it would take three or four posts.  You can get the code at http://www.dreamincode.net/forums/blog/1017/entry-2502-stcroixskippers-usn-journal-explorer-10/ where I've attached a zip file with all of the source.

    The UI is written in WPF and consists of a column of buttons that allows you to select a volume, Query, Create, Delete, Save State, View Changes, and List Folders. In the right hand pane, I display information in a listbox. Typically the listbox contains folder entries from the Master File Table (List Folders) or USN Journal entries and displays the file or folder name.  You can see the detail by clicking or selecting an entry.  I display a simple window with the detail about the entry formatted for readability.

    I haven't written the code to build the fully qualified path name given a File Reference Number yet and there is one hitch.  It is possible to be working on one entry whose Parent File Reference Number belongs to a folder that has already been deleted but the entry for the deleted folder is downstream in the USN Journal.  I'm working on that.

    Most of the buttons are self explanatory.  But in order to view the changes to a volume (i.e. the USN Journal entries) you need to have a 'previous' state, then some changes to the volume.  You'll need to 'Save State', then force some changes to the volume, then 'View Changes'.  I typically bring up another development environment, 'Save State' then clean or rebuild a project and then view changes.

    Also, you'll get an 'Access is denied' exception unless you have admin rights.  I haven't added code to elevate priviliges yet.

    There are a couple of bugs I haven't tracked down. I'm running Vista with indexing on. What I've discovered is that if I delete the USN Journal, it almost immediately is recreated by some other process running on my machine.  I think it is the indexer.

    The important class is NtfsUsnJournal. I'd like comments.  I've tried to expose most of the functionality of the USN Journal while masking the complexity of actually accessing the data.  I can add functions that make it even simpler to identify just changes to files and directories, adds, deletes, and changes.

    If you can improve the code, I'm open to suggestions.

    The application could certainly be improved but it is secondary to the NtfsUsnJournal class but feel free to comment.

    Also, this is a work in progress but there may be a better forum to continue this.  I'm open to suggestions.


    StCroixSkipper
    Wednesday, April 28, 2010 9:00 PM
  • I've actually written the USN Journal as a service and used it in a product.  But what I found is that there is very little value provided by the service.  The USN Journal in itself is much like a service.  It is always available and accessible. If each 'client' keeps track of the USN Journal states of interest, it can service multiple clients.  They need only to keep track of the previous state and process up to the current state. 

    From a commercial product perspective, users don't particularly like to see additional services running on their systems.  Given a choice with very little or no value add, they would choose the option without the service.

    I've also thought about the Overlapped IO.  That might be something of interest.  My background is backup products so it didn't really apply.  You typically run the backup app at designated intervals. We've talked a lot about 'Near Continuous Backup' with the Backup Service running all the time waiting for volume changes to record.

    Anyone else have an opinion on the Overlapped IO issue?

    Wednesday, April 28, 2010 9:08 PM
  • Hi, StCroixSkipper

    I went to review the project but the zip file appears to be corrupt. :( Sure usermode is fine the only problem is that at some point the journal will eventually purge records. What this means is that if you have a (magicUsn) value that you want to return records from you might not get all the records if alot of activity took place on the volume or the application wasn't run for awhile. The service however would be able to maintain all of these records so that if the journal does purge existing records they would still exist in the database. That is why there is only two reasons to use a service
    1) The administrative rights requirment.
    2) The journal can purge records.
    In this case MagicUsn wouldn't be to useful but you can re-work that into searching by a specific date or time rather than the MagicUsn value.

    I have written UAC elevation using the COM moniker this allows you to be manifested AsInvoker but yet execute code as Administrator. I could help you with adding elevation to your process without manifesting it with RequireAdministrator. The advantage to that approach is the user won't be prompted when they launch the application but only prompted when you need to run Administrative tasks such as with the journal.

     Overlapped/IO would be fine, I only see two things that would need to use the Async IO that would be
    1) Waiting for the journal to finish deleting or Wait for a pending delete from another application.
    2) Waiting for new records.. You can poll the journal or use the overlapped/IO here would remove the need to poll.

     

    Friday, April 30, 2010 1:48 PM
  • I've already coded in the rights elevation... 

    I've been trying to set up a project in CodeProject.  I've gotten one set up but I'm having trouble uploading the code.  Which client should I use, and how to get it to work...

    Erik,

    Have you tried to use the latest code?


    StCroixSkipper
    Friday, April 30, 2010 4:19 PM
  • I tried to download again but it's still corrupt. I am using the dreamincode link. You may want to try uploading to codeplex instead. You could recruit anyone else interested in the project to help with further production. It also allows you post recent updates and fixes it makes things easier to manage since you basically have full control of what is placed on your project site.

    Saturday, May 01, 2010 12:42 AM
  • Obviously right now it appears only you and I are interested in such a project if thats the case then it's a two man job but it's always available to recurit anyone who is interested in the project.

    -ErikLitze

    Hi Guys,

       Please don't assume that people aren't interested! I just haven't been checking hotmail for the last few years! Personally, I think this has the basis to be a fantastic .NET library that would really show off the speed of using a lower level method such as reading the MFT as opposed to .NET provided methods. Unfortunately, you have both far surpassed my level of knowledge in file systems so I don't feel qualified to contribute, but I will certainly keep a keen eye on the progress.

    Keep it up, you are doing a great job!

    Gareth

    Sunday, May 02, 2010 11:18 AM
  • I had some time this weekend so I added the code to call NtCreateFile() and NtQueryFileInformation() to get the path and display it in the detail window.  This works but I haven't addressed the edge case where the parent folder has been deleted by the time I process the USN Journal entry.  

    I've posted the code for the changes files in my DreamInCode blog at this link: http://www.dreamincode.net/forums/blog/1017-stcroixskippers-blog/


    StCroixSkipper
    Monday, May 03, 2010 12:26 AM
  • I've changed a couple of things.

    1. If you didn't save the USN Journal State before you tried to 'View Changes', it appeared that the app received an error accessing the USN Journal. What actually happened was that no saved usn state was available and I sent a USN Journal ID of zero to the DeviceIoControl() function which produced the error. I now test for a saved usn journal state and if one doesn't exist, I remind you that to 'View Changes' you need a saved usn journal state. Also if there are no changes since the saved usn journal state, I show a 'No entries found' message.

    2. It is interesting to view the files shown in the usn entries. So now if you double click on the name or the path, I call the Process.Start() function with the full path. If there is a file extension association for that file it will open that file in the associated application.  If no association exists it will allow you to select an application like NotePad to use to view the file.

    I've posted the code at http://www.dreamincode.net/forums/blog/1017/entry-2540-stcroixskippers-usn-journal-explorer-11/


    StCroixSkipper
    Thursday, May 06, 2010 5:27 PM
  • is MD5 value already in USNJournal so that we can just read it, without calculating by ourselves?
    VB Geek
    Tuesday, July 27, 2010 2:54 AM
  • is it possible to gather only a folder, rather than a drive then filter it?
    VB Geek
    Tuesday, July 27, 2010 3:41 AM
  • in your latest journal exploert 1.3, there are some missing code, for example, the code file of VolumeSelectDialog.xaml, and resultsLb_Selectionchanged in  MainWindow.xaml

    could you please provide a working copy containing in a zip file?

    • Edited by WilsonChan Tuesday, July 27, 2010 8:44 AM more detail
    Tuesday, July 27, 2010 8:39 AM
  • is it possible to get all file info: file size, last modified date etc?
    VB Geek
    Tuesday, July 27, 2010 9:28 AM
  • I'm not sure I understand your comment.  The DeviceIoControl() with FSCTL_READ_USN_JOURNAL reads the raw bytes of the journal.

    bool bRtn = Win32Api.DeviceIoControl(
                            _changeJournalRootHandle,
                            Win32Api.FSCTL_READ_USN_JOURNAL,
                            rujdBuffer,
                            sizeRujd,
                            pbData,
                            pbDataSize,
                            out outBytesReturned,
                            IntPtr.Zero);

    And the statement:  usn = new Win32Api.USN_RECORD(pUsnRecord); converts the bits from the raw read into a USN_RECORD in its constructor.  I know I can do it a little faster in C++ because I can just cast the memory to the structure. 

    Other than that, I'd love to find a faster way but I don't see one.  Even reading the raw sparse file.

    On another topic, I've been side tracked these last few days working on another project.  But I've been thinking about what I would like the ideal UsnJournal object to look like.  In resolving the issues between removing the complexity and still exposing the functionality of the actual USN Journal, I've decided I minimally want my UsnJournal object to expose the ability for the user to specify which kinds of entries he/she wants to see.

    This means giving the caller the ability to specify ReasonMask and RetrunOnlyOnClose and perhaps BytesToWaitFor in the typedef struct {
      USN       StartUsn;
      DWORD     ReasonMask;
      DWORD     ReturnOnlyOnClose;
      DWORDLONG Timeout;
      DWORDLONG BytesToWaitFor;
      DWORDLONG UsnJournalID;
    } READ_USN_JOURNAL_DATA, *PREAD_USN_JOURNAL_DATA;

    When I get a working copy of the code, I'll post it here.


    StCroixSkipper


    My first comment on this thread.  Excellent discussion, very valuable info. 

    I have not pulled down your code from dreamincode, so my comments may be OBE.

    Instead of using a class and constructor, I used a managed structure and a marshalled memory buffer, then copied between the two (like your constructor does, only not as a constructor).  The value it affords me is that the managed structure looks like the Win32 API.  I think I will go the constructor route, but extend the class/struct to more closely resemble the Win32 api.

    From your earlier code posted above, I took and eliminated the "safe" decorator and the need to compile in safe mode by using Marshal.SizeOf where you use "sizeof".  Maybe you've already changed this, I haven't reviewed in detail yet.  I have a personal preference against "safe" declaration, though I will use it if I have to (but here I didn't have to).  But I do wonder if Marshal.SizeOf adds any overhead over and above "sizeof".

     


    Les Potter, Xalnix Corporation, Yet Another C# Blog
    Tuesday, September 07, 2010 12:42 PM
  • Hi Skipper,

    I am currently experimenting with your code with some mods and am about to use it in an application. My question is simply, does the USNJournal that I am exploring contain EVERY file on the disk. I have done a traditional file scan and one thru the USNJournal and there are less files in the USN Journal.

    This, it seems is because of some weird file behaviors ... like the vgasyst.fon file. This file appears to be located in the C:\Windows\Fonts folder according to a recursive file scan, however the USNJournal says it is actually elsewhere some WinSxS folder. Perhaps a backwards compatibitily feature of sort. 

    Is searching for files in the USNJournal the same as searching through the MFT ?

    Anyhow, great work and if you'd like some help creating a CodePlex project or managing it all let me know. This code is golden and needs to managed correctly.

    Cheers.

     

     

    Monday, February 14, 2011 1:11 AM
  • Hi,

    I've setup http://usnjournal.codeplex.com

    Please do contact me if you wish to use it. I am happy to maintain it also if you wish.

    Cheers.

     

    Monday, February 14, 2011 1:32 AM
  • Also, I have another interest.  I've written code to use the Volume Shadowcopy Service (VSS) to create shadowcopies of volumes to enable backup up open files and applications like Exchange.  I would like to contribute to a VSS project too.  Is any interest in Volume Shadowcopy Service Requestor code?

    StCroixSkipper

    Hi,

    I have great interesting in this area. After trying this myself i found the excellent AlphaVSS Libraries that have given my Backup Software the ability to copy files that are in use.

    As there are Managed Wrappers, there are specific libraries for WinXp, 2008 x86 and x64 etc. 

    it would be great to have a nice easy to use Library that would use PInvokes directly and not have to worry too much about the deployment OS.

     

    Hope that gets you on the right track.

    Monday, February 14, 2011 4:58 AM
  • John,

    I’ve never done a thorough check but it would be an interesting test.  I think I’ve always found more files and folders with the Master File Table than with FindFirst/FindNext.  If you use the standard .Net Directory.GetFiles() function it tests each file/folder name and throws an exception if it finds a bad name.  FindFirst/FindNext lets you find all of the files.

     

    So here is the interesting test.  Collect up all the files and all of the folders on a volume both ways and write out the differences…  Those found one way and not the other and vice versa.

     

    It wouldn’t take long to write and test on a couple of volumes.

     

    I have always assumed that the Master File table has everything in it and FindFirst/FindNext filters out a couple.  When I gotten properties on a volume and it shows the number of files , the number of files you find in the MFT is always larger in my experience.

     

    You will find differences if you have Mounted volumes.  If you specify a mount point, Find First/Find Next may find all of the ‘mounted’ files as well as those in the MFT.  When I was working with mount points, it would only show one entry in the MFT for the mount point  The MFT is strictly volume based.

     

    I would like to use it on CodePlex. I see the link and will try it...


    StCroixSkipper
    Monday, February 14, 2011 9:59 PM
  • John,

    It is pretty simple to write Volume Shadowcopy Requestors once you find all the information you need to know. I've thought about putting some code and explanations up on the net.  There is a major problem with XP.  It only lets you create one shadowcopy.  Vista and Windows 7 lets you create 64 or 256.  So one of the problems I've tackles is product upgrade.  If you upgrade a working product and the upgrade fails, you want to be able to get back to your original point rather that to leave the system in an inoperable state. 

    I've used a combination of the USN Journal to understand what has changed and the Volume Shadowcopy Service to 'freeze' a copy of the volume before you start the upgrade.  Volume Shadowcopies are preserved across reboots so on Vista or Windows 7 you could create a new shadow of the volume every time you needed to reboot during the upgrade.  Not so with XP.  With XP you need to use the USN Journal to keep track of files that have changed, been deleted, been added, folders too.

     

    -Scott


    StCroixSkipper
    Monday, February 14, 2011 10:05 PM
  • I just had a shock.  I tried using my USN Journal code on Window 7 and it couldn't find the fully qualified paths of the files and folders it finds on the volume.  I started debugging the code and sure enough, the NtCreateFile() function fails when I try to open a file with its FRN.  When I do the

    "Win32Exception win32Error = new Win32Exception(Marshal.GetLastWin32Error());" the message in the exception is "The operation completed successfully." 

    I'm going back to my Vista machine tonight and try it there. 

    Does anyone know if Windows 7 changes the way NtCreateFile() opens files?  Its been a while since I played with this code.  I've forgotten how I stumbled on the flags to use with the NtCreateFile() function.  I'm using FILE_OPEN_BY_FILE_ID | FILE_OPEN_FOR_BACKUP_INTENT.

     


    StCroixSkipper
    Tuesday, March 22, 2011 12:19 AM
  • I checked the NTSTATUS return code.  It is 0xc000000d which is 'STATUS_INVALID_PARAMETER'.  I'll bet is the way I'm trying to open the file.  I'm using the FRN instead of the file name.  I have seen any documentation about changes to the NtCreateFile() and its calling sequence.  I'm running System 7 on and X64 machine.

    Has anyone else seen any changes to the doc on NtCreateFile()?


    StCroixSkipper
    Wednesday, March 23, 2011 5:13 PM
  • My version works on Windows 7 maybe you use READ access and the file is locked. My version uses FILE_READ_ATTRIBUTES

    Also check the IoStatusBlock [out] members to see the final status

    • FILE_CREATED
    • FILE_OPENED
    • FILE_OVERWRITTEN
    • FILE_SUPERSEDED
    • FILE_EXISTS
    • FILE_DOES_NOT_EXIST
    Wednesday, March 23, 2011 6:13 PM
  • I forgot to mention that you have to be careful especially for x64 or AnyCPU compiled. For example a pointer is 4 bytes when your process is running under x86 and pointer is 8 bytes when running process under x64. There could be an issue with your pointers especially if you have some code that converts to Integer as in...

    Consider the following when your running your code under AnyCPU or x64 builds.

    path = Marshal.PtrToStringUni(new IntPtr(buffer.ToInt32() + 4), nameLength / 2);

    should be

    path = Marshal.PtrToStringUni(new IntPtr(buffer.ToInt64() + 4), nameLength / 2);

    // OR

    If IntPtr.Size = 4 Then
    '// use ToInt32() version
    End If

    If IntPtr.Size = 8 Then
    '// Use ToIntt64() version
    End If

     

    Wednesday, March 23, 2011 6:55 PM
  • Thanks, eriklitze,

    I think you are right.  I haven't had time to try your suggestion but I'll bet it is right on.  Thanks for the response...


    StCroixSkipper
    Monday, April 11, 2011 8:45 PM
  • Hi,

     

    I am stopping in to share a how to calculate the progress when enumerating the MFT with FSCTL_ENUM_USN_DATA. In the DeviceIoControl call you specify a buffer that is to be filled with UsnRecords but in order to calculate the progress we would have to know the size of the Master File Table.. That is where you can use FSCTL_GET_NTFS_VOLUME_DATA. In addition it also gives you the NTFS Minor and Major version if you pass a large enough buffer. Take notice to the member in BOLD. (MftValidDataLength) well there you have it. You can use this to calculate progress based on the supplied buffer you use when calling FSCTL_ENUM_USN_DATA. In addition you can determine if the operation will take longer than expected. For example my C:\ drives MFT length is 332MB but my other partition E:\ has only 22MB. Your application can use this to determine how to process records but the most important part is providing a percentage which is useful when populating the database. Enjoy!

    typedef struct {
     LARGE_INTEGER VolumeSerialNumber;
     LARGE_INTEGER NumberSectors;
     LARGE_INTEGER TotalClusters;
     LARGE_INTEGER FreeClusters;
     LARGE_INTEGER TotalReserved;
     DWORD     BytesPerSector;
     DWORD     BytesPerCluster;
     DWORD     BytesPerFileRecordSegment;
     DWORD     ClustersPerFileRecordSegment;
     LARGE_INTEGER <strong>MftValidDataLength</strong>;
     LARGE_INTEGER MftStartLcn;
     LARGE_INTEGER Mft2StartLcn;
     LARGE_INTEGER MftZoneStart;
     LARGE_INTEGER MftZoneEnd;
    } NTFS_VOLUME_DATA_BUFFER, *PNTFS_VOLUME_DATA_BUFFER;
    
    

    Sunday, April 24, 2011 4:00 AM
  • skipper.  Impressive code work.  

      I was wondering if I could talk to you outside of the forums about a problem I am having and see if you can offer any insight.  It is off-topic to the MFT and this topic, but something you mentioned earlier about de-duplication piqued my interest and I think you may be able to answer the question.  If you are interested in hearing me out, just add @gmaildotcom to my username and shoot me an email.  I have been looking for someone who can assist me with a small (relative to ones knowledge) project and I have come across some issues and I need someone knowledgeable in filesystems. 

    Thanks for your time.

    Monday, May 16, 2011 4:20 PM
  • Skipper and Erik,

    I really like the work you've done here! I too am interested in reading from the MFT, from the Journal, and getting into VSS and Restore points. My biggest problem...I;m a noobie! :) It's been 16+ years since I studied C in college and I never used it outside of school. The good news is that I was able to pick up C# pretty quick. Using the Skipper's code from above I managed to get something working. My interest in the MFT is from a forensics point of view. I'd like to be able to re-cursively read the MFT and output File index, File name, size and the timestamps. I have been plowing through documentation on Kernel32, the MFT, and other forums but I still think I am missing something. I guess my question is how to get that info out of the MFT. Initially, I was thinking I would get the highest value for the MFT index and loop through to collect what I need. Am I close? Or am I in outer space?

     

    Any Help is appreciated.

     

    P.S. Please be gentle! :) I realize you time is just as valuable as mine! :)

     

    Regards,

    C

    Tuesday, May 24, 2011 1:51 AM
  • John,

    I’ve never done a thorough check but it would be an interesting test.  I think I’ve always found more files and folders with the Master File Table than with FindFirst/FindNext.  If you use the standard .Net Directory.GetFiles() function it tests each file/folder name and throws an exception if it finds a bad name.  FindFirst/FindNext lets you find all of the files.

     

    So here is the interesting test.  Collect up all the files and all of the folders on a volume both ways and write out the differences…  Those found one way and not the other and vice versa.

     

     

    StCroixSkipper

    I have implemented this test. The code is available for download on the Codeplex site.  http://usnjournal.codeplex.com/

    There is a huge speed hit trying to convert the FileReferenceNumber into a Path. Its so slow that it negates the whole point of using the MFT at the moment but it might just be something i've not looked into yet.

     

    See what you think.... BTW it takes forever to do the comparison so don't use it on a huge drive just yet. I need to add some hashing or something to speed up the lookups.

     

    Cheers 

     

    Tuesday, May 31, 2011 5:00 AM
  • I have since fixed this speed issue. it now takes about 1500 - 1600 milliseconds to enumerate 235,000 files on my C:\ Drive.

    That's fast ;)

    Tuesday, May 31, 2011 6:35 AM
  • Dear StCroixSkipper,

    The piece of code on USN journal that you given is very helpfull. Thanks for that.

    I have some doubt regarding USN journal records. Please help me out.

    I am working on VC++ and trying to get the data cluster chain of deleted files of greater than 4GB in size.

    If a file of less than 4GB in size is deleted (lets assumes that the MFT and its data is not overwritten after deleting). We can recover its data by reading the cluster chain of this file from MFT.

    But if the file is of greater than 4GB in size an dwe delete it, the cluster chain of the data from its MFT is also got deleted.

    So I am trying to get the data cluster chain of this deleted file. As we know that every time when any change happens in NTFS the log is created as a record. I am trying to get the information about this file from $LogFile. 

    If we follow the approach of creating a USN journal and read the USN record, can we get that cluster chain of a deleted file.

    My questions are-

    1- Does USN journal helpful in getting my objective?

    2- Can we get the listing of deleted files with the help of USN records?

    3- How can we get the cluster chain of a deleted file from USN journal. 

     

     

    Thanks

    Madan

    Wednesday, September 07, 2011 9:11 AM
  • I have since fixed this speed issue. it now takes about 1500 - 1600 milliseconds to enumerate 235,000 files on my C:\ Drive.

    That's fast ;)

    thats nice John!

    I also hit the 'no entry' gate at this speed issue.

    would you mind sharing how you got past it?

    thanks!

    Saturday, December 17, 2011 9:00 AM
  • i used the code in the mft scanner project to play around with some stuff and it is stupid fast to bring back a list of files, faster than anything else i have seen. However, it does little good to get a list of 200000 files and then loop thru them looking for things you are interested in (by name, extension, etc) as it negates the speed gain almost entirely.

    so my question is, does the API being used to pull the file listing from the MFT support passing in wild cards so i can get, for example, *.jpg files? this is much more useful than a list containing every file and i still have to loop thru it. might as well walk the directories by hand without the API supporting some kind of filtering

     

     

    Sunday, January 01, 2012 4:52 PM
  • EDIT: SOLUTION: The answer is that MftValidDataLength needs to be divided by MFT record length which is 1024 giving total number of MFT records making progress feedback trivial.

     

    Hi,

     

    Could someone help me to understand why sum of sizes from enumeration with FSCTL_ENUM_USN_DATA does not add up to MftValidDataLength?

    I got 682,885,120 from FSCTL_GET_NTFS_VOLUME_DATA (which I confirmed is correct with fsutil fsinfo ntfsinfo C:) but when adding up outBytesReturned in procedure EnumerateFiles() I am getting only 29,286,240? This is around 23 times less then I would expect. What do I do wrong?

     

     

            unsafe private void EnumerateFiles(IntPtr medBuffer, ref Dictionary<ulong, FileNameAndFrn> files, string[] fileExtensions) {
                IntPtr pData = Marshal.AllocHGlobal(sizeof(UInt64) + 0x10000);
                PInvokeWin32.ZeroMemory(pData, sizeof(UInt64) + 0x10000);
                uint outBytesReturned = 0;
    
                ulong progress = 0;
                while (false != PInvokeWin32.DeviceIoControl(_changeJournalRootHandle, PInvokeWin32.FSCTL_ENUM_USN_DATA, medBuffer,
                                        sizeof(PInvokeWin32.MFT_ENUM_DATA), pData, sizeof(UInt64) + 0x10000, out outBytesReturned,
                                        IntPtr.Zero)) {
                    IntPtr pUsnRecord = new IntPtr(pData.ToInt32() + sizeof(Int64));
                    progress += outBytesReturned;
                    ...
                    Log.InfoFormat("Progress: {0}", progress);
            ...
    

     

     

    By the way c# implementation of FSCTL_GET_NTFS_VOLUME_DATA:

     

    //PInvokeWin32.cs
    
            public const UInt32 FSCTL_GET_NTFS_VOLUME_DATA = 0x00090064;
    
            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public struct NTFS_VOLUME_DATA_BUFFER {
                public UInt64 VolumeSerialNumber;
                public UInt64 NumberSectors;
                public UInt64 TotalClusters;
                public UInt64 FreeClusters;
                public UInt64 TotalReserved;
                public UInt32 BytesPerSector;
                public UInt32 BytesPerCluster;
                public UInt32 BytesPerFileRecordSegment;
                public UInt32 ClustersPerFileRecordSegment;
                public UInt64 MftValidDataLength;
                public UInt64 MftStartLcn;
                public UInt64 Mft2StartLcn;
                public UInt64 MftZoneStart;
                public UInt64 MftZoneEnd;
            }
    
    
    //Mft.cs in Mft class
            unsafe private PInvokeWin32.NTFS_VOLUME_DATA_BUFFER GetNtfsInfo() {
                IntPtr pAlloc = IntPtr.Zero;
    
                PInvokeWin32.NTFS_VOLUME_DATA_BUFFER retValue = new PInvokeWin32.NTFS_VOLUME_DATA_BUFFER();
    
                try {
                    
                    int q = 96;
                    uint size = 0;
                    pAlloc = Marshal.AllocHGlobal((int)q);
                    IntPtr pDest = pAlloc;
    
                    bool fResult = PInvokeWin32.DeviceIoControl(
                        _changeJournalRootHandle,
                        PInvokeWin32.FSCTL_GET_NTFS_VOLUME_DATA,
                        IntPtr.Zero, 
                        0,  
                        pDest,
                        sizeof(PInvokeWin32.NTFS_VOLUME_DATA_BUFFER),  
                        out size,
                        IntPtr.Zero);
    
                    if (!fResult) {
                        throw new Exception(Marshal.GetLastWin32Error().ToString());
                    }
                    
    
                    retValue.VolumeSerialNumber = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.NumberSectors = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.TotalClusters = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.FreeClusters = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.TotalReserved = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.BytesPerSector = (UInt32)Marshal.PtrToStructure(pDest, typeof(UInt32));
    
                    pDest = (IntPtr)((Int64)pDest + 4);
                    retValue.BytesPerCluster = (UInt32)Marshal.PtrToStructure(pDest, typeof(UInt32));
    
                    pDest = (IntPtr)((Int64)pDest + 4);
                    retValue.BytesPerFileRecordSegment = (UInt32)Marshal.PtrToStructure(pDest, typeof(UInt32));
    
                    pDest = (IntPtr)((Int64)pDest + 4);
                    retValue.ClustersPerFileRecordSegment = (UInt32)Marshal.PtrToStructure(pDest, typeof(UInt32));
    
                    pDest = (IntPtr)((Int64)pDest + 4);
                    retValue.MftValidDataLength = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.MftStartLcn = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.Mft2StartLcn = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.MftZoneStart = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    pDest = (IntPtr)((Int64)pDest + 8);
                    retValue.MftZoneEnd = (UInt64)Marshal.PtrToStructure(pDest, typeof(UInt64));
    
                    return retValue;
                } finally {
                    Marshal.FreeHGlobal(pAlloc);
                    pAlloc = IntPtr.Zero;
                }
            }
    
    
    // call method used in EnumerateVolume()
    PInvokeWin32.NTFS_VOLUME_DATA_BUFFER volInfo;
    try {
        GetRootFrnEntry();
        GetRootHandle();
        volInfo = GetNtfsInfo();
        Log.InfoFormat("MftValidDataLength: {0}", volInfo.MftValidDataLength);
        ...
    } catch (Exception e) {
        Log.Info(e.Message, e);
        ...
    } finally {
        ...
    }
    


     



    • Edited by MaxOfLondon Monday, January 16, 2012 10:14 PM Answer to my question
    Monday, January 16, 2012 1:21 PM
  • i used the code in the mft scanner project to play around with some stuff and it is stupid fast to bring back a list of files, faster than anything else i have seen. However, it does little good to get a list of 200000 files and then loop thru them looking for things you are interested in (by name, extension, etc) as it negates the speed gain almost entirely.

    so my question is, does the API being used to pull the file listing from the MFT support passing in wild cards so i can get, for example, *.jpg files? this is much more useful than a list containing every file and i still have to loop thru it. might as well walk the directories by hand without the API supporting some kind of filtering

     

     

    Hi Eric,

    You can specify array of file extensions you want to filter in EnumerateVolume()

    Hope this helps

    cheers

    Max

            private void cmdRun_Click(object sender, EventArgs e) {
                Dictionary<UInt64, FileNameAndFrn> result;
                Mft mft = new Mft();
                mft.Drive = "C:";
    
                mft.EnumerateVolume(out result, new string[] { ".jpg" });
    
                System.IO.StreamWriter _file = new System.IO.StreamWriter("_files.txt");
                foreach (KeyValuePair<UInt64, FileNameAndFrn> entry in result) {
                    FileNameAndFrn file = (FileNameAndFrn)entry.Value;
                    _file.WriteLine(mft.GetFilePath(file.ParentFrn) +Path.DirectorySeparatorChar + file.Name);
                }
                _file.Close();
    
            }
    


    Monday, January 16, 2012 2:09 PM
  • Hi,

    I've setup http://usnjournal.codeplex.com

    Please do contact me if you wish to use it. I am happy to maintain it also if you wish.

    Cheers.

    That is a very good work.

    I have some related questions about MFT and USN Change Journal. Since each USN journal entry contains a file reference number, is this file reference number the same as the reference number (for that file) in the MFT? Is there any relationship between them (USN & MFT reference numbers of the same created file)?

    Please let me know.

    Thanks.

    -BoBo

    Thursday, February 09, 2012 9:15 PM
  • This is the API definition in vb.net to access the master file table Imports System.Runtime.InteropServices Module APIDefination #Region "Functions to declared" Public Declare Auto Function CreateFile Lib "Kernal32.dll" (ByVal IpFileName As String, ByVal DwDesiredAccess As Int32, ByVal DwShareMode As Int32, ByRef IpSecurityAttributes As SECURITY_ATTRIBUTES, ByVal DwCreationDisposion As Int32, ByVal DwFlagsAndAttributes As Int32, ByVal TempletFile As IntPtr) As Integer Public Declare Auto Function ReadFile Lib "kernal32.dll" (ByVal Ref As Integer, ByVal IpBuffer As Byte(), ByVal NumberOfBytesToRead As Integer, ByRef NumberOfBytea As Integer, ByRef OverLapped As System.Threading.NativeOverlapped) As Boolean <DllImport("kernal32.dll", SetLastError:=True)> Function WaitforSingleObject(ByVal handel As IntPtr, ByVal milliseconds As UInt32) As UInt32 End Function Public Declare Auto Function CloseHandel Lib "kernal32.dll" (ByVal hobject As IntPtr) As Boolean #End Region #Region "Constants" Public Const INFINATE As UInt32 = &HFFFFFFFFUI Public Const WAIT_ABANDONED As UInt32 = &H80UI Public Const WAIT_OBJECT_0 As UInt32 = &H0UI Public Const WAIT_TIMEOUT As UInt32 = &H102UI Public Const GENERIC_READ As Int32 = &H80000000 Public Const GENERIC_WRITE As Int32 = &H4000000 Public Const OPEN_EXESTING As Int32 = &H3 Public Const FILE_ATTRIBUTES_NORMAL As Int32 = &H80 Public Const FILE_FLAG_OVERLAPPED As Int32 = &H40000000 Public Const FILE_SHARE_READ = &H1 Public Const INVALID_HANDEL_VALUE = -1 Public Const INVALID_SET_FILE_POINTER = -1 Public Const FILE_BEGIN = 0 #End Region #Region "Structures" Public Structure SECURITY_ATTRIBUTES Dim nLength As Integer Dim lpSecurityDescription As Integer Dim InheritHandel As Boolean End Structure <StructLayout(LayoutKind.Sequential)> Public Structure MFTAttributes Dim Attributetype As Int32 Dim length As Int32 Dim NonResFlag As Byte Dim NameLength As Byte Dim offsetToTheName As Int16 Dim Flags As Int16 Dim AttributeID As Int16 Dim Lengthoftheattributes As Int32 Dim indexedFlag As Byte Dim padding As Byte End Structure #End Region End Module
    Tuesday, March 13, 2012 11:51 AM
  • AlexBB are you still on the MSDN?

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me


    Tuesday, March 20, 2012 8:55 AM
  • Note: The author failed to specify what versiom OS he was runnins under. This DOES NOT work om WIN7 run with admin.

    The other version DID work on XP.

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me

    Friday, March 30, 2012 10:51 PM
  • AlexBB zre you still on the MSDN?

    Unfortunately, haven't seen him posting for a very long time. :p

    Eyal (http://shilony.net), Regards.

    Saturday, March 31, 2012 3:48 AM
  • Has anyone accomplished the getdirectory described in this thread?

    If so....Please let me know by writing a note.

    VBGEEK, if you are still around, all of that is standard informaton in the VB header.

    Thanks.

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me



    Tuesday, April 17, 2012 3:39 PM
  • Hi,

    WOW, It has been a very long time but I have finally got back to the investigation of this MFT vs File System Scanner issue. Take a look at http://mftscanner.codeplex.com .  The new version will write all the files in lower case, sorted order so that a tool such as WinMerge can show you the differences.

    I think the code is quite optimized (mostly) and most MFT open/dumping can be achieved in less than a few seconds. This is very promising, but I'd love to know why there are differences in the numbers of files reported on System type folders.

    Cheers.

    Friday, August 10, 2012 1:32 PM
  • If you don't know what the mft is, you shouldn't read the question.

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me

    Wednesday, March 13, 2013 8:51 AM
  • Is MiedievalKnight is around? (Please say yes....)

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me

    Saturday, December 21, 2013 12:05 AM
  • Let me try it this way, does anyone reading this have working MFT code?

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me

    Saturday, December 21, 2013 4:11 AM
  • "USN_RECORDS return only FileNames this is to reserve as much space possible in the change log. You can map (fileReference) or (ParentFileReference) to complete paths using API. This approach is so much easier to use than keeping a database or enumerating the directories and walking the chain to get the full path."

    Really? I'm dying to know the routine.

    Renee


    "MODERN PROGRAMMING is deficient in elementary ways BECAUSE of problems INTRODUCED by MODERN PROGRAMMING." Me

    Wednesday, December 25, 2013 2:14 PM