locked
How to debug Visual Studio extensions? RRS feed

  • Question

  • Hi!

    I'm trying to write my own custom Visual Studio editor. I've implemented what I believe is to be the most basic version of that editor (in my case, that's editor factory, pane and simple user control), set that up according to instructions in [1], and then started up the experimental instance of VS.

    Debugger kicked in and that's what I got:

    • Editor factory was instantiated correctly;
    • Editor factory created  editor pane;
    • Pane created the user control and returned it through the overridden Window property (three times)
    • All my code finished executing (my, in terms: the code I could debug)
    • Visual Studio shown information: "The operation could not be completed".

    Output contains no useful data:

    Entering constructor for: Spooksoft.Spk.SpkPackage
    Entering Initialize() of: Spooksoft.Spk.SpkPackage

    Visual Studio log neither:

    Importing pkgdef file C:\USERS\WSURA\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\11.0EXP\EXTENSIONS\SPOOKSOFT\SPKPACKAGE\1.0\SpkPackage.pkgdef
    (...)
    Successfully loaded extension... C:\USERS\WSURA\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\11.0EXP\EXTENSIONS\SPOOKSOFT\SPKPACKAGE\1.0\
    Extension is enabled... C:\USERS\WSURA\APPDATA\LOCAL\MICROSOFT\VISUALSTUDIO\11.0EXP\EXTENSIONS\SPOOKSOFT\SPKPACKAGE\1.0\
    (...)
    Begin package load [SpkPackage]
    End package load [SpkPackage]

    Event log is empty. 

    I won't ask, where the error is, because without source you have no chance to figuring it out (which is here, by the way, if you want to check it out: [2]). I'd rather want to ask, how can I debug my extension to figure out what goes wrong?

    [1]: http://msdn.microsoft.com/en-us/library/vstudio/bb165071.aspx 
    [2]: https://skydrive.live.com/redir?resid=8B169EF451FFD9FE!611&authkey=!ADEyYwo60k_Qgyk

    Wednesday, October 2, 2013 2:28 PM

Answers

  • 1: Look for exception output in the VS Output Window while debugging. There will be a lot (unfortunately), but the ones closest to the failure message box are probably the interesting ones.

    2: Implement ICustomQueryInterface and see what COM interfaces VS is probing your object for, perhaps there is one you are missing.

    3: Break when the message box is up, get a full callstack (mixed mode, all symbols resolved against the MS public symbol servers) and post it here, I can take a look at where the error is emanating from.

    Wednesday, October 2, 2013 2:46 PM

All replies

  • 1: Look for exception output in the VS Output Window while debugging. There will be a lot (unfortunately), but the ones closest to the failure message box are probably the interesting ones.

    2: Implement ICustomQueryInterface and see what COM interfaces VS is probing your object for, perhaps there is one you are missing.

    3: Break when the message box is up, get a full callstack (mixed mode, all symbols resolved against the MS public symbol servers) and post it here, I can take a look at where the error is emanating from.

    Wednesday, October 2, 2013 2:46 PM
  • Well, that actually helped :) I suspected, that VS expects me to cast to some interface, but I couldn't find a way to determine, which ones are required. By the way - regarding to point 1 from your answer, I already mentioned, that Output window contains no useful data...

    I'm still in the process of determining, what went wrong, but now I can make another approach to the problem.

    I'll leave here some code in case someone encounters the same problem.

    Implementation of ICustomQueryInterface may look like the following:

    public CustomQueryInterfaceResult GetInterface(ref Guid iid, out IntPtr ppv) { Debug.WriteLine(String.Format("Attempt to cast to interface with iid {0}", iid)); ppv = IntPtr.Zero; return CustomQueryInterfaceResult.NotHandled; }

    You'll get the following result:

    Attempt to cast to interface with iid f22a0ad5-8f51-4f66-a644-ea64770cf8b7
    Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid 9d71890d-090c-4b67-80c3-4cb55c600b60
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid 00020400-0000-0000-c000-000000000046
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid d5d49c61-1c0b-4ea1-9adb-a79fb1dbc7b5
    Attempt to cast to interface with iid 8560cecd-dfac-4f7b-9d2a-e6d9810f3443
    Attempt to cast to interface with iid 8560cecd-dfac-4f7b-9d2a-e6d9810f3443

    This gives us the GUIDs of interfaces, to which VS tries to cast our class. Now we can use the following class to retrieve the actual types from the GUIDs.

    As would Seven say, crude, but effective.

        public static class Util {
    
            private static Dictionary<string, Type> guidMap;
    
            static Util()
            {
                guidMap = new Dictionary<string, Type>();
    
                AppDomain.CurrentDomain.AssemblyLoad += (s, e) =>
                {
                    try {
                        _ScanAssembly(e.LoadedAssembly);
                    } catch {
    
                    }
                };
    
                foreach (Assembly a in AppDomain.CurrentDomain.GetAssemblies())
                {
                    try {
                        _ScanAssembly(a);
                    } catch {
    
                    }
                }
            }
    
            private static void _ScanAssembly(Assembly a)
            {
                foreach (Type t in a.GetTypes())
                {
                    var attrs = t.GetCustomAttributes<GuidAttribute>(false);
                    foreach (var item in attrs) {
    
                        if (!guidMap.ContainsKey(item.Value.ToLower()))
                            guidMap.Add(item.Value.ToLower(), t);
                    }
                }
            }
    
            public static Dictionary<string, Type> GuidMap {
    
                get {
    
                    return guidMap;
                }
            }
        }
    }
    
    Now we can implement ICustomQueryInterface in a little bit different way:
            public CustomQueryInterfaceResult GetInterface(ref Guid iid, out IntPtr ppv) {
    
                if (Util.GuidMap.ContainsKey(iid.ToString().ToLower())) {
    
                    Type t = Util.GuidMap[iid.ToString()];
                    Debug.WriteLine(String.Format("Attempt to cast to interface {0}", t.ToString()));
                }
                else
                    Debug.WriteLine(String.Format("Attempt to cast to interface with iid {0}", iid));
    
                ppv = IntPtr.Zero;
                return CustomQueryInterfaceResult.NotHandled;
            }
    

    And the result is a lot more helpful:

    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsDeferredDocView
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData2
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface System.Runtime.InteropServices.NativeMethods+IDispatch
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.Shell.Interop.IVsPersistDocData
    Attempt to cast to interface Microsoft.VisualStudio.TextManager.Interop.SVsCodeWindow
    Attempt to cast to interface Microsoft.VisualStudio.TextManager.Interop.SVsCodeWindow
    Now I'll try to determine, which of these interfaces are required and which are optional.
    Thursday, October 3, 2013 6:31 AM