locked
Trying to access debugger's DTE object from package code running in separate Eperimental Visual Studio instance RRS feed

  • Question

  • Hi --

    I am debugging a vsx package that has been loaded into the Experimental instance of Visual Studio via the standard debugging process initiated from a separate debugger instance of Visual Studio.  (i.e my package code is loaded as a project into one instance of Visual Studio, which then launches a separate Visual Studio Experimental instance when I press F5).  The executing package initializes a runtime for the language I am integrating into Visual Studio, and as part of that runtime bootstrapping process I wish to set some breakpoints back in the debugger instance.    Previously I was not executing this code as part of a package, but as a Visual Studio project itself and thus there was only one instance of VisualStudio and I was able to interact with the debugger via the local DTE object.  I want to do the same thing here, but now the DTE I need to access is running in a separate process from the one where the code is executing that will access it.

    Don't know if this is clear, but the whole reason I am doing this is so that I don't have to manually set breakpoints every time I execute my language runtime.  These breakpoints are set on dynamically generated assembly code that gets created as part of my bootstrap process, so there's no way Visual Studio could know about them across sessions.   They are set at key places that allow me to stop and debug the interpreter while my language runtime bootstraps itself.  THe problem was that having set these once, they would still appear in the breakpoint list on subsequent sessions; they just wouldn't do anything.   So I would have to remove them from the list and then add them again each and every time I executed the code -- something that was just too much of a chore and begged to be automated.

    Now that the code is executing in a different process from where the debugger is running, if I am to be able to continue to do this, I must access the DTE of a separate Visual Studio instance.   I'm hoping this is indeed possible, but my original code unchanged doesn't seem to be doing the trick.  Here's the code I'm using to try to do this:

    while (((DTE2^)dte) == nullptr)
    {
    System::Runtime::InteropServices::ComTypes::IEnumMoniker^ theMonikerEnumerator;
    theROT->EnumRunning(theMonikerEnumerator);
    Int32 theReturnedCount;
    IntPtr theCountPtr(theReturnedCount);
    System::String^ theReturnedName = gcnew System::String("");
    array<System::Runtime::InteropServices::ComTypes::IMoniker^>^ theROTMonikers = gcnew array<System::Runtime::InteropServices::ComTypes::IMoniker^>(1);
    System::Runtime::InteropServices::ComTypes::IMoniker^ theDummyMoniker;
    int numTries = 0;
    while ((!theMonikerEnumerator->Next(1,theROTMonikers,theCountPtr)) &&
       (numTries < 100))
    {
    try
    {
    theROTMonikers[0]->GetDisplayName(theContext,theDummyMoniker,theReturnedName);
    if (theReturnedName->Contains("ExecutableModelingDLL"))
    {
    System::Object^ theDTEObject;
    theROT->GetObject(theROTMonikers[0],theDTEObject);
    DTE2^ theDTE = dynamic_cast<DTE2^>(theDTEObject);
    Debugger2^ theDebugger = dynamic_cast<Debugger2^>(theDTE->Debugger);
    Process2^ theROTProcess = dynamic_cast<Process2^>(theDebugger->DebuggedProcesses->Item(1));
    if (theROTProcess->ProcessID == theProcess->Id)
    {
    return theDTE;
    }
    numTries++;
    }
    }

    catch (...)
    {
    theMonikerEnumerator->Reset();
    numTries++;
    }
    }

    return nullptr;
    }

    The problem is that the object I get back from the Running Object Table via the call to GetObject in the code above doesn't seem to be well-formed.   I can't actually look at the values, since Visual Studio's inspector mechanism doesn't seem to function during my package initialization.  I can set breakpoints, I just can't inspect values.  I suspect that since calls will now be cross-process, I can't simply cast the return from GetObject directly to a DTE2 ptr, but need to do something more involved.  But what that is isn't entirely clear to me.

    If anybody sees anything that I am doing wrong in the code above or has any ideas for me on how to accomplish this, I would be most appreciative.  In the code above, thedynamic_cast instruction that sets my DTE2 object executes correctly, but the instruction right after it that attempts to use that object to return a pointer to the debugger always throws an exception.

    Any help would be greatly appreciated.

    Thanks.

    Mike

     

    Saturday, August 23, 2014 3:44 AM

Answers

  • Hi --

    Probably a waste for me to update this message, as I have to be the only one in the world at this point who is trying to do what I'm trying to do, but I thought I would report back, just in case this is useful to anyone.

    Turns out the code above largely works, except for one mistake I made.  In hopes of distinguishing the instance of Visual Studio that I was looking for, I had changed the following line in the code above from what it originally was:

    if (theReturnedName->Contains("ExecutableModelingDLL"))

    Originally this line had been:

     

    if (theReturnedName->StartsWith("!{"))

    I was lulled into a false sense of security when I observed that execution was actually entering the branch guarded by this condition, and believed that I had correctly hit upon the way to identify the desired Visual Studio instnace, thinking that by GetDisplayName (the line immediately before this one in the code above) this meant get the window text of the Visual Studio instance I was looking for.

    Wrong!

    I'm not sure which registered object actually met this criteria, (although it probably is the object that actually contains the code I'm discussing), but it certainly wasn't the Visual Studio instance.  I returned the code to its original state (which is in fact an accurate way of identifying at least some Visual Studio instance) and inside the block that follows it, I replaced the test for a match with code that checks that the solution name of the DTE object's current solution contains "ExecutableModelingDLL" This did the trick and it turned out that the DTE object returned by the (thus modified) code above is perfectly okay to use across processes.

    Now I just have one more serious problem, which I will enter in a separate entry.


    • Marked as answer by Caillen Tuesday, September 2, 2014 2:06 AM
    Saturday, August 23, 2014 6:56 AM

All replies

  • Hi --

    Probably a waste for me to update this message, as I have to be the only one in the world at this point who is trying to do what I'm trying to do, but I thought I would report back, just in case this is useful to anyone.

    Turns out the code above largely works, except for one mistake I made.  In hopes of distinguishing the instance of Visual Studio that I was looking for, I had changed the following line in the code above from what it originally was:

    if (theReturnedName->Contains("ExecutableModelingDLL"))

    Originally this line had been:

     

    if (theReturnedName->StartsWith("!{"))

    I was lulled into a false sense of security when I observed that execution was actually entering the branch guarded by this condition, and believed that I had correctly hit upon the way to identify the desired Visual Studio instnace, thinking that by GetDisplayName (the line immediately before this one in the code above) this meant get the window text of the Visual Studio instance I was looking for.

    Wrong!

    I'm not sure which registered object actually met this criteria, (although it probably is the object that actually contains the code I'm discussing), but it certainly wasn't the Visual Studio instance.  I returned the code to its original state (which is in fact an accurate way of identifying at least some Visual Studio instance) and inside the block that follows it, I replaced the test for a match with code that checks that the solution name of the DTE object's current solution contains "ExecutableModelingDLL" This did the trick and it turned out that the DTE object returned by the (thus modified) code above is perfectly okay to use across processes.

    Now I just have one more serious problem, which I will enter in a separate entry.


    • Marked as answer by Caillen Tuesday, September 2, 2014 2:06 AM
    Saturday, August 23, 2014 6:56 AM
  • Probably a waste for me to update this message, as I have to be the only one in the world at this point who is trying to do what I'm trying to do, but I thought I would report back, just in case this is useful to anyone.

    Hi mclagett,

    Thanks a lot for sharing your answer here. I believe it  will  help someone who has similar problems.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, August 25, 2014 11:46 AM