locked
SDK Package: my command filter causes an AccessViolationException RRS feed

  • Question

  • 

    Hi,

    I am trying to create a Visual Studio 2012 extension that can record key presses in a text editor and replay them. I am developing on Windows 8.

    I have a IVsTextViewCreationListener that calls AddCommandFilter() to add a command filter to any new text editor being created:

    public class VsTextViewCreationListener : IVsTextViewCreationListener
    {
        public void VsTextViewCreated(IVsTextView textViewAdapter)
        {
            var filter = new MyCommandFilter();
    
            IOleCommandTarget next;
            if (ErrorHandler.Succeeded(textViewAdapter.AddCommandFilter(filter, out next)))
                filter.Next = next;
        }
    }

    The command filter looks like this:

    public class MyCommandFilter : IOleCommandTarget
    {
        public IOleCommandTarget Next { get; set; }
    
        public int Exec(ref Guid pguidCmdGroup, uint nCmdID, uint nCmdexecopt, IntPtr pvaIn, IntPtr pvaOut)
        {
            if (nCmdID == (uint)VSConstants.VSStd2KCmdID.TYPECHAR)
            {
                // Save values of pguidCmdGroup, nCmdID, nCmdexecopt and GetTypedChar(pvaIn)
                // ...
            }
    
            return Next.Exec(ref pguidCmdGroup, nCmdID, nCmdexecopt, pvaIn, pvaOut);
        }
    
        public int QueryStatus(ref Guid pguidCmdGroup, uint cCmds, OLECMD[] prgCmds, IntPtr pCmdText)
        {
            return Next.QueryStatus(ref pguidCmdGroup, cCmds, prgCmds, pCmdText);
        }
    
        public void Playback()
        {
            // Resend the values
            var pvaIn = Marshal.AllocCoTaskMem(4);
            Marshal.GetNativeVariantForObject((ushort)savedChar, pvaIn);
            Next.Exec(ref pguidCmdGroup, nCmdID, nCmdexecopt, pvaIn, IntPtr.Zero);
        }
    
        private static char GetTypedChar(IntPtr pvaIn)
        {
            return (char)(ushort)Marshal.GetObjectForNativeVariant(pvaIn);
        }
    }

    (I have clipped out the portion of the code that holds the values in a list)

    It works in the sense that it captures and replays the key presses, however after replaying them, it often (not always) crashes with the following exception:

    AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

    This occurs on the on return Next.QueryStatus(...).

    I have never written any VS extension before and surely what I am doing is sketchy at best...

    I tried to release the memory allocated with AllocCoTaskMem() - it seems to sometimes behave better and sometimes it will cause the same exception on the call to FreeCoTaskMem(), after the call to Exec().

    Would appreciate any ideas.

    (cross posted from Stack Overflow)

    Friday, October 26, 2012 12:19 AM

Answers

  • Your sizing is wrong, a VARIANT is 16 bytes not 4 bytes. You should be doing something more like:

    var pvaIn = Marshal.AllocCoTaskMem(16);
    
    try
    {
        //Do something with the VARIANT here
    }
    finally
    {
        if(pvaIn != IntPtr.Zero)
        {
            Marshal.FreeCoTaskMem(pvaIn);
        }
    }


    Also your Exec is incorrect in only checking nCmdId. The value of nCmdId in the Std2k 'command namespace' is NOT unique across all 'namespaces', you can very well see an Exec with the same nCmdId that is NOT TYPECHAR. You need to do this:

    if ((pguidCmdGroup == VSConstants.VSStd2K) && (nCmdID == (uint)VSConstants.VSStd2KCmdID.TYPECHAR))
    {
    }

    Ryan


    Friday, October 26, 2012 1:15 AM

All replies

  • Your sizing is wrong, a VARIANT is 16 bytes not 4 bytes. You should be doing something more like:

    var pvaIn = Marshal.AllocCoTaskMem(16);
    
    try
    {
        //Do something with the VARIANT here
    }
    finally
    {
        if(pvaIn != IntPtr.Zero)
        {
            Marshal.FreeCoTaskMem(pvaIn);
        }
    }


    Also your Exec is incorrect in only checking nCmdId. The value of nCmdId in the Std2k 'command namespace' is NOT unique across all 'namespaces', you can very well see an Exec with the same nCmdId that is NOT TYPECHAR. You need to do this:

    if ((pguidCmdGroup == VSConstants.VSStd2K) && (nCmdID == (uint)VSConstants.VSStd2KCmdID.TYPECHAR))
    {
    }

    Ryan


    Friday, October 26, 2012 1:15 AM
  • Thanks, it's working perfectly now!
    Friday, October 26, 2012 1:56 AM