Access memory error after unmanaged interop in Vista.

Question

I have got a Access memeroy error when get a memory be malloced from unmage code to manage code in Vista, but it works well in XP SP2.

 

My manage code are

public partial class Form1 : Form

    {

        private void button1_Click(object sender, EventArgs e)

        {

               string name = RunInterop.GetStringFromUnmanage();

           }

     }

Interop code

  public static class RunInterop

    {

        const string RunDllFileName = "Unmanage.dll";

        [DllImport(RunDllFileName)]

        public static extern string GetStringFromUnmanage();

}

 

Unmage code

extern "C"

{

__declspec(dllexport) char* __stdcall GetStringFromUnmanage()

{

      char *lpchar = (char *)malloc(strlen("test) + 1);

      memcpy(lpchar," test ",strlen("test "));

      return lpchar;

}

}

There have a crash in the ntdll!RtlpLowFragHeapFree function in Vista, but it works well in Vista. I am sure lots of developers have meet this error, any suggestion? I know it is ok if using StringBuilder to get the string from the unmanage code, but I am

 

0:000> kb

ChildEBP RetAddr  Args to Child             

001ce4d4 770a18c3 001ce814 001ce814 00db25f3 ntdll!RtlpLowFragHeapFree+0x31

001ce4e8 76677a7e 00200000 00000000 04183398 ntdll!RtlFreeHeap+0x101

001ce4fc 7653d5d6 00200000 00000000 041833a0 KERNEL32!HeapFree+0x14

001ce510 7653de31 7660e6fc 041833a0 001ce5f4 ole32!CRetailMalloc_Free+0x1c

001ce520 7a0b425b 041833a0 001ce814 7a0b7ab9 ole32!CoTaskMemFree+0x13

001ce52c 7a0b7ab9 001ce7bc 001ce804 f9131f59 mscorwks!DefaultMarshalOverrides<CSTRMarshalerBase>::ReturnCLRFromNative+0x33

001ce768 79ea35c6 00db25f0 001ce804 001ce7c0 mscorwks!RunML+0x77a

001ce7e8 79ea34ba 00245360 001ce894 00757030 mscorwks!NDirectGenericStubPostCall+0x194

001ce850 00db2647 00245360 001ce894 fca439b0 mscorwks!NDirectGenericStubReturnFromCall+0x1f

001ce87c 00d20303 014a0fb8 0148e8c4 0000004b CLRStub[StubLinkStub]@db2647

001ce8d4 7b0693eb 00000000 00000000 00000000 Manage2Un!Manage2Un.Form1.button1_Click(System.Object, System.EventArgs)+0x33 [D:\Fuhaitao\Manage2Un\Manage2Un\Form1.cs @ 23]

001ce96c 7b1065e9 00000000 00000000 00000000

Answer 1

Hi,

 

When you are allocating a string buffer in unmanaged code and return it to managed code, .NET will try to free your memory allocated in GetStringFromUnmanage() by calling CoTaskMemFree, which leads to a crash because it was allocated by malloc, and is in a different heap than what CoTaskMemFree is using. So you'll have to use CoTaskMemAlloc instead of malloc/new in this case.

The reason why only Vista is failing is most probably because vista has become more sensitive (or robust) to such kind of allocation differences. In fact, you should always rely on CoTaskMemAlloc/CoTaskMemFree, since that's exactly what .NET/CLR is using.

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

 

 

 

 

Answer 2

further question, why .NET/CLR was trying to free the memory allocated in GetStringFromUnmamaged() ? Is it always trying to do so?

 

What I thought is .NET/CLR just needs to make a copy at managed heap, rather than worry about how to free memory in GetStringFromUnmamaged().

 

If I just change the return type from string to intptr for public static extern string GetStringFromUnmanage(),   is there any potential issues?  It worked, at least.

Answer 3

In this case, since unmanaged function GetStringFromUnmanaged is returning a pointer to a block of memory, the piece of memory by convention is now owned by the CLR. After CLR makes a copy of that memory, it decides that it no longer needs the memory any more (since it already has a copy) and frees it by CoTaskMemoryFree (or SysFreeString in the case of BSTR. SysFreeString basically does the same thing as CoTaskMemoryFree, except that it needs to do some pointer adjustment for the BSTR pointer).

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 4

This makes no sense.  The CLR cannot possibly free the pointer, it doesn't have to reside in any heap at all or might point into the middle of an allocated block.  I'm pretty sure it won't try.

It is clear we are not looking at the actual code, it is missing a double-quote.  Nevertheless, it allocates insufficient bytes to store the string and anything can happen after it executes, including an AccessViolation.  The way this code is written would be questionable even if only a native client would use it.  The function signature should be

  int GetStringFromUnmanage(char* buffer, int buflen)

so that the client can pass the function a buffer that it then fills with the string.  You would pass an initialized StringBuilder at the managed side:

  StringBuilder sb = new StringBuilder(1024);
  GetStringFromUnmanage(sb, sb.Capacity);
  string retval = sb.ToString();
...
  [DllImport(...)]
  private static extern int GetStringFromUnmanage(StringBuilder buffer, int buflen);

Answer 5

Huh?

 

I don't think so. The CLR always attempts to free that memory allocated at unmanaged world, except a block of memory referenced by a IntPtr type. MSDN did say  that.

 

Anyway, I just took a quick demo and that proved.

 

Code Snippet

[DllImport("UnmanagedStrAlloc.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi)]

public extern static IntPtr GetStringFromCoAlloc();

 

IntPtr str2 = GetStringFromCoAlloc();

str = Marshal.PtrToStringAnsi(str2);

Marshal.FreeCoTaskMem(str2);

 

 

It worked at Vista, and if I just passed a string back, evil there!!!

Answer 6

I think the behavior of CLR is right. StringBuild can not hold unmanaged buffer, so it must alloc a new managed buffer by itself. Image CLR do NOT free the unmanaged buffer. There is a memory leak, I think.

Answer 7

To nobugz:

In some case the CLR will free the memory, when the function signature and marshalling attribute tells the CLR to do so. When the function is written this way: char *GetStringFromUnmanaged(), by convention it is expecting the caller to free the memory. The CLR here is following the COM convention: if you are returning a pointer to something new in a function as out/return value, then the memory that the pointer points to is owned by the caller, and should be released by the caller. If the function doesn't follow this convention, then the function should be re-written or you should do the marshalling manually.

Of course, in pure unmanaged world, what you said is correct. The caller should not expect the returned buffer to be always allocated in heap. To avoid any confusion about the ownership of the memory, the function should be rewritten in the way you suggested: int GetStringFromUnmange(char *buffer, int buflen);

To summarize, by letting CLR marshalling the return value, the marshalling rule must be followed, otherwise you should marshal it by your own.

 

To Buddhist:

I believe inside your GetStringFromCoAlloc function, it allocates a string by using CoTaskMemAlloc, right? Otherwise it won't work in Vista.

 

To deepnight,

In this case, if the CLR doesn't free the memory, yes, there will be a leak. Using StringBuilder in the argument as [out] would be another different story though.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

 

 

Answer 8

Right, I did a CotTaskMemAlloc :-)

 

Using StringBuilder in the argument as [out] would be another different story though.

Could you please talk it a little bit more?

I tried this way, and it failed at Vista.

 

 

Answer 9

Hi buddhist,

Could you please elaborate on how you were using StringBuilder in your case? Some code would be nice

Answer 10

__declspec(dllexport) void __stdcall GetStringFromStrBuilder(char** buffer)
{

 buffer = (char**)malloc(1000);

 *buffer = malloc(1000);

 memset(*buffer, 'a', 1000);
 *buffer[999] = '\0'; 
}

 

 

[DllImport("UnmanagedStrAlloc.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi)]
        public extern static void GetStringFromStrBuilder(out StringBuilder str);

 

......

StringBuilder sb = null;
GetStringFromStrBuilder(out sb);
Console.WriteLine(sb.ToString());

 

 

Looks like it's same problem and same reason compared with using String. Right?

How do you think about safely using StringBuilder?

Answer 11

We seem to agree how this should be fixed.  I disagree with your assessment of the original function signature.  There is no COM convention for dealing with a char* return value, there is only a convention for BSTR.  Returning a char* from an unmanaged function is quite legal in general, the function might be returning static strings or strings that are malloced for the lifetime of the DLL instance.  There is no attribute that would make the marshaler automatically free the memory and it won't automatically call CoTaskMemFree().

The marshaler has built-in knowledge about the StringBuilder type.  There is no need to declare the argument with ref or out or use OutAttribute.

Answer 12

To buddhist,

 

There are 3 places in your code that need to be fixed:

1. Similar problem: The memory allocation should be done using CoTaskMemAlloc

2. buffer = (char **)malloc(1000) is incorrect. You should allocate 1000 bytes of memory and assign it to *buffer

3. memset is actually assigning 1000 integer 97, which is a buffer overflow

 

So, You native function should be written like the following:

 

Code Snippet

 *buffer = (char *)CoTaskMemAlloc(100);
    strcpy(*buffer, "ABC");

 

In your case it doesn't make much difference whether you are using a string or StringBuilder.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

 

 

Answer 13

Hi nobugz,

 

Such marshalling behavior is documented in MSDN. Please refer to http://msdn2.microsoft.com/en-us/library/z6cfh6e6(VS.80).aspx : Default marshalling for arrays.

It mentioned the following:

 

The interop marshaler uses the CoTaskMemAlloc and CoTaskMemFree methods to allocate and retrieve memory. Memory allocation performed by unmanaged code must also use these methods.

 

Also, as for the char * marshalling rule, although there aren't any specific rule about char * in COM, usually COM follows the convention that when you are returning a block of memory to the caller, the caller owns the memory. This should apply to any kind of piece of memory, whether it is pointing to a string buffer or not.

 

If we are talking about pure unmanaged function in general, I agree the function should be able to return anything.

 

However, when you are dealing with the CLR and let the CLR marshal the arguments, you'll have to follow the CLR's marshalling rule, by specify correct marshalling attributes, otherwise please marrshal on your own.

 

About StringBuilder, I agree by default it is passed by ref. However, you can modify its behavior by using out (not the attribute, but the keyword), as shown in the example Buddhist is showing above.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 14

To nobugz,

 

In Adam Nathan's ".NET and COM - The complete Interoperability Guild", he point out that CLR will do a CoTaskMemFree. And from my debugging experience, at least, CLR on Vista does it.

 

Deepnight

Answer 15

The library topic that describes marshaling of strings would probably be more appropriate.  It is however completely silent about marshaling of return values.  Note that it does give a specific example of how to use StringBuilder without the ref keyword.

Just to verify your claim, I tried this:

Unmanaged:
const char* string = "Nobugz waz here";
extern "C" __declspec(dllexport) const char* GetString() {
  return string;
}

Managed:
    [DllImport(@"c:\projects\cppcsdll\debug\cppcsdll.dll")]
    private static extern string GetString();
...
      string txt = GetString();
      Console.WriteLine(txt);

It worked as expected, no exceptions.  That's not exactly proof though, it might silently call CoTaskMemFree() anyway.  It doesn't have a HRESULT return value, it just says it might "unexpectedly terminate the application" when it gets an invalid pointer.  I'm not sure how to otherwise verify whether or not CoTaskMemFree() gets called, short from stepping through the marshaling code.  That's a bit much.

Answer 16

Hi nobugz,

 

It is quite interesting that in your case it doesn't crash. I tried your program here in Vista and it doesn't crash either. However, CoTaskMemFree does get called during the marshalling process:

 

ntdll!RtlFreeHeap
  KERNEL32!HeapFree+0x14
  ole32!CRetailMalloc_Free+0x1c
  ole32!CoTaskMemFree+0x13
  mscorwks!CSTRBufferMarshalerBase::ClearNative+0x1d
  mscorwks!DefaultMarshalOverrides<CSTRMarshalerBase>::ReturnCLRFromNative+0x479
  mscorwks!RunML+0x2a8e
  mscorwks!NDirectGenericStubPostCall+0x5e1
  mscorwks!NDirectGenericStubReturnFromCall+0x1f
  CLRStub[StubLinkStub]@6ea6f9
  main2!Program.Main(System.String[])+0x1c (calls the unmanaged function)

 

To verify that in your side, you can fire up WinDbg, put a breakpoint in ole32!CoTaskMemFree, and verify that whether the pointer being passed in is the pointer you have returned.

 

 What is actually happening is that, CoTaskMemFree calls HeapFree, which will check the pointer to see if it is valid. Since the memory you have returned is not allocated by HeapAlloc/HeapReAlloc (malloc/new indirectly calls HeapAlloc), it will fail, set last error to STATUS_INVALID_PARAMETER, and return FALSE. CLR simply ignores that. I'm not sure why it doesn't throw a exception when CoTaskMemFree failed. Probably it should.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 17

Okay, your stack trace convinced me you're right.  This is a potentially nasty problem.  The MSDN library documentation on string marshaling certainly doesn't mention that we should use CoTaskMemAlloc().  This could lead to heap corruption if the CRT heap sub-allocator would just happen to allocate a new block in the heap for the string.  You've got better tools than me, would you mind checking what happens if you malloc() the string buffer?  Ask for a big buffer so you wouldn't get a sub-allocated block.

Answer 18

You are right. I think we need to update the article or write another one to explain this rule.

In Vista, if you allocate a large block of memory by malloc, it will call HeapAlloc in a heap (in my machine it is 0x009f0000). Later, CLR will call HeapFree (by calling CoTaskMemFree) with a different heap handle (in my machine it is 0x000d0000), which is being used in ole32.dll. HeapFree checks the pointer and found that it is a heap allocated memory, but in a different heap, therefore it will fail, trigger a int 0x3 and then access a invalid pointer which will crash itself.

As for XP, HeapFree might be tolerant with such kind of errors, and just ignores the problem. I don't have a XP right now so I cannot verify.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 19

Thanks for checking, I can live with a crash instead of silent heap corruption.  No crash in Windows XP though, I tried.  I don't have the symbols and debugging tools to check if heap corruption is the outcome.  Ultimately, this explains the OP's problem: crashes on Vista, doesn't crash on XP.  This really should be documented in the MSDN article, could you set the wheels in motion?

Answer 20

 I'm not sure if the heap would be corrupted in XP. I'll check out that later.

 

As for MSDN documentation, I just came across an interop article about memory management in MSDN:

http://msdn2.microsoft.com/en-us/library/f1cf4kkz(VS.80).aspx

 

It states that :

 

The interop marshaler always attempts to free memory allocated by unmanaged code. This behavior complies with COM memory management rules, but differs from the rules that govern native C++.

...

The runtime always uses the CoTaskMemFree method to free memory. If the memory you are working with was not allocated with the CoTaskMemAlloc method, you must use an IntPtr and free the memory manually using the appropriate method. Similarly, you can avoid automatic memory freeing in situations where memory should never be freed, such as when using the GetCommandLine function from Kernel32.dll, which returns a pointer to kernel memory. For details on manually freeing memory, see the Buffers Sample.

 

However, this article still doesn't contain enough information (it doesn't mention anything about how unmanaged code should free memory passed from CLR) and is short of examples. Actually I'm planning to write something for MSDN, and this topic might be a appropriate start. Let me see what I can do.

Answer 21

Hi nobugz,

 

It is quite interesting that in your case it doesn't crash. I tried your program here in Vista and it doesn't crash either. However, CoTaskMemFree does get called during the marshalling process:

 

ntdll!RtlFreeHeap
  KERNEL32!HeapFree+0x14
  ole32!CRetailMalloc_Free+0x1c
  ole32!CoTaskMemFree+0x13
  mscorwks!CSTRBufferMarshalerBase::ClearNative+0x1d
  mscorwks!DefaultMarshalOverrides<CSTRMarshalerBase>::ReturnCLRFromNative+0x479
  mscorwks!RunML+0x2a8e
  mscorwks!NDirectGenericStubPostCall+0x5e1
  mscorwks!NDirectGenericStubReturnFromCall+0x1f
  CLRStub[StubLinkStub]@6ea6f9
  main2!Program.Main(System.String[])+0x1c (calls the unmanaged function)

 

To verify that in your side, you can fire up WinDbg, put a breakpoint in ole32!CoTaskMemFree, and verify that whether the pointer being passed in is the pointer you have returned.

 

 What is actually happening is that, CoTaskMemFree calls HeapFree, which will check the pointer to see if it is valid. Since the memory you have returned is not allocated by HeapAlloc/HeapReAlloc (malloc/new indirectly calls HeapAlloc), it will fail, set last error to STATUS_INVALID_PARAMETER, and return FALSE. CLR simply ignores that. I'm not sure why it doesn't throw a exception when CoTaskMemFree failed. Probably it should.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.

Answer 22

Hi nobugz,

 

It is quite interesting that in your case it doesn't crash. I tried your program here in Vista and it doesn't crash either. However, CoTaskMemFree does get called during the marshalling process:

 

ntdll!RtlFreeHeap
  KERNEL32!HeapFree+0x14
  ole32!CRetailMalloc_Free+0x1c
  ole32!CoTaskMemFree+0x13
  mscorwks!CSTRBufferMarshalerBase::ClearNative+0x1d
  mscorwks!DefaultMarshalOverrides<CSTRMarshalerBase>::ReturnCLRFromNative+0x479
  mscorwks!RunML+0x2a8e
  mscorwks!NDirectGenericStubPostCall+0x5e1
  mscorwks!NDirectGenericStubReturnFromCall+0x1f
  CLRStub[StubLinkStub]@6ea6f9
  main2!Program.Main(System.String[])+0x1c (calls the unmanaged function)

 

To verify that in your side, you can fire up WinDbg, put a breakpoint in ole32!CoTaskMemFree, and verify that whether the pointer being passed in is the pointer you have returned.

 

 What is actually happening is that, CoTaskMemFree calls HeapFree, which will check the pointer to see if it is valid. Since the memory you have returned is not allocated by HeapAlloc/HeapReAlloc (malloc/new indirectly calls HeapAlloc), it will fail, set last error to STATUS_INVALID_PARAMETER, and return FALSE. CLR simply ignores that. I'm not sure why it doesn't throw a exception when CoTaskMemFree failed. Probably it should.

 

--

Yi Zhang, Microsoft

This posting is provided "AS IS" with no warranties, and confers no rights.