locked
Out Of Memory exception when closing dirty editor in VS2010 RRS feed

  • Question

  • I have a set of custom editors in a VS2010 extension package. When I try and close the editor when it's marked as dirty (i.e. it has unsaved changes and gets an * next to the name) I get a system exception messagebox showing "Not enough storage is available to complete this operation. (Exception from HRESULT: 0x8007000E (E_OUTOFMEMORY))" with header "Microsoft Visual Studio". It doesn't break anywhere in my code and stepping through with the debugger is of no use. I've narrowed down that it only occurs if the editor is marked as dirty. Can you provide any advice or assistance here?
    Monday, February 20, 2012 4:03 PM

Answers

  • Ok it's fixed now, don't know why though. About 3 months ago we removed the following line of code to fix a different issue: http://social.msdn.microsoft.com/Forums/en-US/vsx/thread/398c01df-9425-4068-889d-942cd875602d

    if (result == null)
                {
                    return VSConstants.DISP_E_MEMBERNOTFOUND;
                }

    This we removed from the end of method "public int GetProperty(uint itemid, int propid, out object result)" from our implementation of an item associated with an editor and implementing these interfaces: IVsUIHierarchy, IVsPersistHierarchyItem2, IVsHierarchyDeleteHandler, IVsHierarchyDropDataTarget, IVsHierarchyDropDataSource2.

    It used to cause a dead hang in VS2010 when we opened an editor, and removing that code made opening the editor work fine (we couldn't tell why). Cluthcing at straws we tried putting it back in, surprisingly editors now open and close fine. I suspect a hotfix in VS2010 or the 2010 SDK fixed the original issue.

    Tuesday, February 21, 2012 2:59 PM

All replies

  • What line of code is it happening on?  It has to be happening on a COM call which is non-preserve sig (which is why the failing HRESULT is being translated into an exception).  There is likely some code on the native side that is encountering NULL unexpectedly and deciding it is an allocation failure (and returning E_OUTOFMEMORY).

    Ryan

    Monday, February 20, 2012 6:17 PM
  • Unfortunately it's not breaking on any line of my code. I can't find any hooks to intercept the Close() call on an editor either.
    Monday, February 20, 2012 6:29 PM
  • Turn off Enable Just My Code, set it to debug both native and managed, and point at the Microsoft Symbol Servers so all the frames can be resolved.

    Ryan

    Monday, February 20, 2012 7:51 PM
  • Cool thanks for the tip! Do you mean native and managed for JIT debugging? I set it break on all thrown exceptions and now I get a "no source available" window with a callstack of:

    Microsoft.VisualStudio.Platform.WindowManagement.dll!Microsoft.VisualStudio.Platform.WindowManagement.RdtLock.Unlock(

    Microsoft.VisualStudio.Platform.WindowManagement.RdtFlags flags) + 0xc9 bytes.

    That doesn't mean a whole lot to me, neither does the disassembly below (I've indicated where the golden arrow is pointed):

    00000000  push        ebp 
    00000001  mov         ebp,esp 
    00000003  push        edi 
    00000004  push        esi 
    00000005  mov         esi,ecx 
    00000007  mov         edi,edx 
    00000009  mov         ecx,esi 
    0000000b  call        dword ptr ds:[54BF295Ch] 
    00000011  mov         edx,edi 
    00000013  mov         ecx,esi 
    00000015  call        FFFCC9B8 
    0000001a  mov         edi,eax 
    0000001c  cmp         edi,80004004h 
    00000022  je          0018D63C 
    00000028  cmp         edi,8004000Ch 
    0000002e  je          0018D63C 
    00000034  cmp         edi,2 
    00000037  jae         0018D642 
    0000003d  jmp         dword ptr [edi*4+54C5DC90h] 
    00000044  cmp         byte ptr [esi+8],0 
    00000048  jne         00000080 
    0000004a  mov         edi,dword ptr [esi+4] 
    0000004d  mov         edx,6Dh 
    00000052  lea         ecx,[edx-6Ch] 
    00000055  call        FFFCACD8 
    0000005a  push        dword ptr [eax+40h] 
    0000005d  mov         edx,esi 
    0000005f  mov         ecx,edi 
    00000061  call        dword ptr ds:[54BF296Ch] 
    00000067  xor         edx,edx 
    00000069  mov         dword ptr [esi+4],edx 
    0000006c  mov         ecx,esi 
    0000006e  mov         eax,dword ptr [ecx] 
    00000070  mov         eax,dword ptr [eax+28h] 
    00000073  call        dword ptr [eax+14h] 
    00000076  mov         ecx,esi 
    00000078  mov         eax,dword ptr [ecx] 
    0000007a  mov         eax,dword ptr [eax+28h] 
    0000007d  call        dword ptr [eax+18h] 
    00000080  mov         byte ptr [esi+8],1 
    00000084  mov         ecx,esi 
    00000086  call        FFFC9A60 
    0000008b  mov         eax,1 
    00000090  pop         esi 
    00000091  pop         edi 
    00000092  pop         ebp 
    00000093  ret 
    00000094  xor         eax,eax 
    00000096  pop         esi 
    00000097  pop         edi 
    00000098  pop         ebp 
    00000099  ret 
    0000009a  mov         edx,645h 
    0000009f  mov         ecx,1 
    000000a4  call        FFE3D770 
    000000a9  test        edi,edi 
    000000ab  jge         000000C9 
    000000ad  mov         edx,55Fh 
    000000b2  mov         ecx,1 
    000000b7  call        FFE3D770 
    000000bc  mov         edx,dword ptr [eax+00000C74h] 
    000000c2  mov         ecx,edi 
    000000c4  call        FFE3C480 
    000000c9  xor         eax,eax                        <=== The golden arrow points here
    000000cb  pop         esi 
    000000cc  pop         edi 
    000000cd  pop         ebp 
    000000ce  ret  
    Monday, February 20, 2012 10:08 PM
  • Do you have a repro?  I looked through the running document table code and it appears it is likely trying to save the document and if that fails (it is a rather complex process) I see some paths that may return E_OUTOFMEMORY, which is probably not accurate, I doubt you actually have no memory :)

    I can't really debug more without a repro though as just doing code inspection is already too difficult with the complexity/flow of the code involved.

    Ryan

    Tuesday, February 21, 2012 12:05 AM
  • Unfortunately I can't distribute the project outside of my employer's offices. Do you perhaps do consultations?

    Below is the code we use to create the editor, I can't think of anything more relevant to include. Our editors are quite different but all are opened this way. They operate fine when hosted in a Windows application, so perhaps the problem is the way we create them as VS editors? It works fine in VS2008.

    (I know there's a few code smells below, I can fix those as soon as we get this working.)

    public void OpenEditor(string filename, uint ItemId)
            {
                IVsUIShellOpenDocument openDoc = (IVsUIShellOpenDocument)Package.GetGlobalService(typeof(SVsUIShellOpenDocument));

                IVsWindowFrame winFrame = null;
                Guid ed = GuidList.guidEditorFactory;

                try
                {
                    Guid logicalView = VSConstants.LOGVIEWID_Primary;

                    if (myPackage.OpenFrames.ContainsKey(ItemId))
                    {
                        winFrame = myPackage.OpenFrames[ItemId];
                        winFrame.Show();
                    }
                    else
                    {
                        try
                        {
                            myPackage.tempWinframe = filename;
                            string test = "{B1A1CDB7-C255-4644-ACFE-38E64B8955A6}";
                            int result = openDoc.OpenSpecificEditor(
                                (uint)__VSOSPEFLAGS.OSPE_OpenAsNewFile,
                                filename + "  (" + ItemId.ToString() + ")",  
                                ref ed,
                                test,
                                ref logicalView,
                                filename,
                                (HierNode)this,
                                ItemId,
                                IntPtr.Zero,
                                PackageSingleton.Package,
                                out winFrame);

                            myPackage.tempWinframe = "";

                            ErrorHandler.ThrowOnFailure(result);
                            if (winFrame != null)
                            {
                                myPackage.OpenFrames[ItemId] = winFrame;
                                winFrame.Show();
                            }
                        }
                        catch (FormatException fex)
                        {
                        }
                    }
                }
                catch (Exception ex)
                {
                    MessageBox.Show(ex.Message.ToString(), IDE.ProductFullName);
                }

            }

    Tuesday, February 21, 2012 10:12 AM
  • Ok it's fixed now, don't know why though. About 3 months ago we removed the following line of code to fix a different issue: http://social.msdn.microsoft.com/Forums/en-US/vsx/thread/398c01df-9425-4068-889d-942cd875602d

    if (result == null)
                {
                    return VSConstants.DISP_E_MEMBERNOTFOUND;
                }

    This we removed from the end of method "public int GetProperty(uint itemid, int propid, out object result)" from our implementation of an item associated with an editor and implementing these interfaces: IVsUIHierarchy, IVsPersistHierarchyItem2, IVsHierarchyDeleteHandler, IVsHierarchyDropDataTarget, IVsHierarchyDropDataSource2.

    It used to cause a dead hang in VS2010 when we opened an editor, and removing that code made opening the editor work fine (we couldn't tell why). Cluthcing at straws we tried putting it back in, surprisingly editors now open and close fine. I suspect a hotfix in VS2010 or the 2010 SDK fixed the original issue.

    Tuesday, February 21, 2012 2:59 PM
  • What were you returning here when you removed this line?  S_OK?  You can't return S_OK + no result (i.e. a null out param for instance), it isn't proper COM convention.

    Ryan

    Tuesday, February 21, 2012 3:57 PM