locked
Is there any way I can trigger my debugger add-in when my custom project type is created or loaded? RRS feed

  • Question

  • Hi --

    I have a custom language and project system that I am adding to Visual Studio and have created an add-in package that both loads a debugger extension (built off the DKM debug engine) and installs a custom project type (via the ProvideProjectFactory attribute on the package class, and also inheriting this class from ProjectPackage and not just Package).

    The debugger extension is configured by the vsdconfig mechanism to listen for when native Windows modules get loaded, and I can trigger breakpoints in the extension if I load an appropriate project type into the experimental debug instance of Visual Studio and run it under the debugger.   But I am interested in activating the debugger extension not when the user executes a loaded project, but when the project itself is actually loading.   The reason for this is that I would like to instantiate my core language runtime at this time and have it debugged by the original Visual Studio instance's debugger that is debugging the extension.   In essence I want to be able to debug my language runtime along with the VSIX package itself as part of the VSIX extension.  I also want my language's runtime not to be explicitly visible to the user of my custom project type, but loading and operating entirely in the background.

    Don't know how much of this made sense, but there are a couple of ways I could imagine accomplishing if the mechanism does indeed support it.   If it turns out that my debugger extension can only be activated when the experimental instance's own debugger is activated, then I could imagine possibly being able to manipulate the experimental instance's automation objects to make this happen by launching and executing a background process.  But if it turns out that the debugger extension will activate when any native windows module is launched, independent of whether it happens as part of executing the experimental instance's debugger, then I could imagine launching a module directly from my project factory whenever one of my projects is created or loaded and having it trigger a breakpint in my debugger extension.

    Once again, not sure if any of this is intelligble, but if some kind soul who has better understanding of debugger extensions than I could take a shot at helping me understand how debugger extensions are actually activated, I would be enormously grateful and seriously in your debt.

    Thanks very much.

    Regards,

    Mike

     


    Thursday, November 13, 2014 6:47 PM

All replies

  • One more thing:

    One thing that I didn't explain in the message above is that I need the debugger extension to be active when I debug my language runtime, because the language runtime itself is executed by writing intel assembly language bytes to memory and then executing these once they've sufficiently bootstrapped an execution engine.   Apparently the Visual Studio debugger is not equipped to work with breakpoints on in-memory code this way without some additional help, which is part of what I want to implement in the debugger extension.

    There's a whole meta- dimension to this that is somewhat head twisting.  it may turn out that in order to debug my language runtime in the context of the debugger extension I have been discussing that I will need a debugger extension for the original Visual Studio that launched my VSIX package and the experimental instance and that anything I do in the package extension I've been discussing so far will only be of help in debugging programs executed under the experimental instance's debugger. 

    It's hard to talk about this without sounding hopelessly confused.  I don't think I am, but I do need a better understanding of how debugger extensions get activated.

    Thursday, November 13, 2014 7:05 PM
  • Hi Mike,

    I don't have experience in extending the debugger, but there's a code sample here which I think could give you some ideas:

    Visual Studio Debug Engine Sample

    Also MSDN document here may help:

    Visual Studio Debugger Extensibility


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.


    • Edited by Caillen Friday, November 14, 2014 10:42 AM
    Friday, November 14, 2014 10:24 AM
  • So, I have in another place asked this question a different way, and figure I have nothing to lose by reposing it here as well.  I realize that my current needs tend to the esoteric, so I'm not expecting a wellspring of response.   But it certainly doesn't hurt to ask.  Please pardon the lengthy post, but it is hard to be succinct with this particular issue.  And it is important to me to get beyond this.  So here it goes...

    I'm working on integrating a custom language and execution runtime into Visual Studio.  I'm working with the various samples that are out there, including a DKM debugger extension sample.  The basic problem that I'm grappling with at the moment is trying to understand better how the dkm extension code gets triggered.   As best I understand it, the vsdconfig that is set up for my dkm extension indicates that it is to be invoked whenever a native windows module is loaded -- and indeed this does work:  I can load a native C++ project into the experimental instance of Visual Studio that is hosting the extension and have a breakpoint in my dkm extension code get hit when a native DLL is loaded.  

    But there is a meta- dimension to what I'm trying to do that is (for me, at least) rather head twisting.  My runtime engine initially bootstraps itself by having a small C++ program write enough intel bytes to memory to fashion a minimal vm execution engine and parser and then jumps to this and bootstraps the rest from there.  Apparently Microsoft's assembly language debug engine isn't equipped to handle code that is generated in this way -- at least not without help from a debugger extension.  You can set breakpoints via the DTE automation APIs and they even appear in the disassembly window, but unfortunately the debug engine ignores them.   So the first thing I'm trying to do is use a debugger extension to help me debug my runtime code by implementing whatever is missing from the assembly language debugger out of the box.   

    But the problem is that this runtime needs to execute when someone loads one of my custom projects, as it is at this point that I want to launch it and have it interact with tool windows and editor integration during what is for that Visual Studio instance still "design time".  I want this to happen without necessarily having to execute the code in the custom project itself.  In other words, it should be able to run as part of project loading, not just as part of project execution.  This means, as far as I can tell, that the code that launches the runtime needs also to be running as part of a package extension.  So I believe that what I'm talking about is a debugger extension loaded into one instance of Visual Studio that will launch an experimental Visual Studio instance in which the project that is being debugged is a second package extension, which will in turn launch a second experimental instance when it is debugged.   It is this second extension that will actually implement the custom project as well as my language source code editing and debugging, while the first extension will implement the enhancement to the assembly language debugging I discussed above.  At least that's the conceptual picture that I have of it.

    Only problem is, it doesn't exactly work.  I really do apologize for the lengthy setup, but it was sort of necessary to allow me to ask this question:  can anyone shed any light on what I might need to do to be able to have the first debugger extension get triggered when the second package loads a native dll module.   I'm no longer silly enough to try to write my extensions in my own language (that is really excessively meta-), so I'm writing the extensions in C#, which of course is managed code.  But when I run the second extension out of the first extension's experimental instance host, I can never even get the breakpoints I set in the first extension to activate, much less get triggered  (i.e, their little red breakpoint circles remain hollow).  

    I'm not exactly sure what the problem is, but I can think of a couple things that it might be.  It might possibly be that loading the second extension module via an external program instead of directly from the ide (as of course one must do with package extensions) may somehow be interfering with the first extension getting notified when a module loads.  Or it might be that since the first extension is configured in its vsdconfig only to be notified when native modules load, it never gets a notification because the extension that's being debugged is managed C#.  (While this second extension is loading my unmanaged DLL in its project factory CreateProject method, it's doing it through the DLLImport mechanism and possibly this is making it appear to the debugger as a managed DLL???).  Or of course it could be something else altogether.


    If I understood the vsdconfig mechanism better, I might be able to configure some other (more "tiggerable") condition that didn't rely on native module loading.  But I don't understand the sample well enough to modify it.   At any rate, I feel like I could really benefit from a better understanding of what it is that actually activates and triggers a debugger extension module like the one I've written.  Is it the host VIsual Studio's debug engine, which reads and registers event notifications specified in the vsdconfig?  This would make sense, since this is what the debugger extension is extending.  In this case, I might expect that the debug engine of the second visual studio experimental instance would activate the debugger extension it is hosting when one of my custom projects is loaded thereby notifying the debug engine of the first visual studio experimental instance of the breakpoint i have set there (which in fact does indeed happen; this breakpoint is triggered, allowing me to trace through the project factory's CreateProject method).  But I would then hope that executing past the unmanged DLL load statement in this CreateProject method  would in turn activate the debugger extension in the first visual studio experimental instance and notify it that an unmanaged DLL had been loaded.  This notification handler would in turn hit the breakpoint I set there and notify the debugger in the original (non-experimental) visual studio instance that kicked all this off.  It seems like this should work, but alas, this last step that isn't happening.   I am trying to gain a better understanding of why not.

    So there you have it.   Once again, there literally is no other forum where I can ask a question such as this, so I'm hoping you all will be tolerant of my excessively lengthy post.

    Thanks very much for any thoughts anybody might have.



    Saturday, November 15, 2014 12:26 AM
  • Hi --

    Even though I have to be the only person on the planet at the moment that is interested in this question, I figured I would circle back and report how I solved the issues that I raised in this thread.

    There were two issues I was dealing with:

    1) A deficiency in Visual Studio's out-of-the-box native Assembly Language debug engine, whereby the engine is unable (without the help of an extension) to debug dynamically generated assembly code (as much of ours is).  

    2) The second issue was that we needed to load our core language runtime immediately on the creation of one of our custom projects and not when the user launches and executes the project.  The reason for this is that we need the language runtime to support a number of features that are active as soon as a project is loaded during both design time and run time, including an interactive REPL tool window with full execution and debug support as well as design-time editor support and tool window presentation of our virtual machine (the stacks. registers and symbol tables,etc. ). 

    The issue I was dealing with was how to set this up so that the extension that addresses the first problem can be used to help debug our core runtime when it is loaded on the creation of a new project (or the loading of an existing one).

    I ended up with a solution that involves two distinct package extensions and three distinct Visual Studio instances (the originating one and two experimental instances).  The originating IDE loads the first package as its solution and debugs it launching the first experimental instance with the "Start external program" application (and the "/rootsuffix Exp" command line argument).  And then our core runtime is loaded as this experimental instance's project and it is debugged by launching the second experimental instance (also launched with the "Start external program" option and the "/rootsuffix Exp" command line argument).  

    So the first experimental instance loads our first package extension, which is a DKM debugger extension needed to debug the generated assembly language in our core runtime.  The second experimental instance loads our second package extension, but not as the debugging target of the first experimental instance, but simply as a deployed extension in the experimental hive.  This package includes a custom project type as well as editor and runtime debugging support for the custom language built on top of our core runtime.   These elements don't need to be debugged, because they all will just hand off to our core runtime, which will be being debugged in the first experimental instance IDE.

    The mistake I was making earlier was that I was trying to have the second package be the debugee of the first experimental IDE instance and load the runtime from there in the CreateProject method of the custom project code.  Placing this extra level of indirection between the debug engine and the runtime code was what was apparently preventing the necessary events from reaching the DKM debugger extension that was extending that debug engine.   The debugger extension is configured (via its vsdconfig file) to be loaded whenever the debugee loads a native module.   I thought that having the second package as the debugee would work because it is indeed what is loading our native module core runtime.  But it turns out that that was preventing the necessary "modeule loaded" events from reaching our DKM debugger extension.   So in the fix the runtime is still loaded by the package and still through the CreateProject method, but making the runtime itself the debugee (instead of the package that loads it) ended up being the key to making it all work.

    I'm discussing all this in so much detail only on the chance that it might help someone in the future.  Not likely, but you never know.  In any event, thank you all for your patience in letting me publicly work through all of this and for putting up with my confused ramblings.

    Regards,

    Mike





    Wednesday, November 19, 2014 5:24 PM