Memory leak as a result of Async Plugable Protocol not releasing instances

Question

Hi,

I developed an application using a Web Browser Control (WBC) and used the APP sdk of Igor Tandetnik. I am using the WBC in order to run a BP multiple times. My memory is growing. I used WinDbg in order to locate the leak, and found that the main reason (among other tiny leaks) is the following stack trace:

7724dd82 ntdll!RtlAllocateHeap+0x00000274
        ffa7513 MSVCR100D!_heap_alloc_base+0x00000053
        ffb5b5c MSVCR100D!_heap_alloc_dbg_impl+0x000001fc
        ffb58ff MSVCR100D!_nh_malloc_dbg_impl+0x0000001f
        ffb58ac MSVCR100D!_nh_malloc_dbg+0x0000002c
        ffb8e3b MSVCR100D!malloc+0x0000001b
        ffa6f81 MSVCR100D!operator new+0x00000011
        6e515aa LrWebIEInternetProtocols!ATL::CComCreator<ATL::CComAggObject<CWebIEInetAsyncPlugableProtocol> >::CreateInstance+0x000000aa
        6e5034d LrWebIEInternetProtocols!ATL::CComCreator2<ATL::CComCreator<ATL::CComObject<CWebIEInetAsyncPlugableProtocol> >,ATL::CComCreator<ATL::CComAggObject<CWebIEInetAsyncPlugableProtocol> > >::CreateInstance+0x0000007d
        6e1ac72 LrWebIEInternetProtocols!ATL::CComClassFactory::CreateInstance+0x000000d2
        6e504e1 LrWebIEInternetProtocols!PassthroughAPP::CComClassFactoryProtocol::CreateInstance+0x00000121
        7499923f urlmon!COInetSession::CreateFirstProtocol+0x000000dd
        74998ad8 urlmon!CTransaction::StartEx+0x00000292
        60782ab2 mshtml!CTridentFilterHost::StartFilter+0x00000083
        60846331 mshtml!CDwnBindData::Bind+0x000004d5
        60841781 mshtml!NewDwnBindData+0x0000019d
        608415bd mshtml!CDwnLoad::Init+0x0000025c
        6088a1ed mshtml!CImgLoad::Init+0x00000043
        609e2337 mshtml!CDwnInfo::SetLoad+0x0000011e
        609e223d mshtml!CDwnCtx::SetLoad+0x00000086
        609ec49c mshtml!CImgCtx::SetLoad+0x0000004d
        60871397 mshtml!CDoc::NewDwnCtx2+0x0000030a
        6093a075 mshtml!CDoc::CreateDwnCtxForUrlImgCtxByIndex+0x000000c6
        60939f6f mshtml!CDoc::OnUrlImgCtxDeferredDownload+0x0000001e
        609ca029 mshtml!GlobalWndOnMethodCall+0x00000115
        609e98a0 mshtml!GlobalWndProc+0x00000302
        74d36238 USER32!InternalCallWinProc+0x00000023
        74d368ea USER32!UserCallWinProcCheckWow+0x00000109
        74d37d31 USER32!DispatchMessageWorker+0x000003bc
        74d37dfa USER32!DispatchMessageW+0x0000000f
        77b1e802 mfc100ud!AfxInternalPumpMessage+0x00000102

This stack is being created for every url, and its size is always 1760 bytes. Since I am deleting the cache between every BP, these objects are being re-created every time and my memory increasing.

Why these objects are not being released when the resource download is over? Should I release them by myself?

Thanks, Yehuda

Answer 1

yehuda s wrote:

Why these objects are not being released when the resource download is  over?

You have a reference counting bug somewhere in your code. A pointer gets  AddRef'ed more times then it gets Release'd.

---

Igor Tandetnik

Answer 2

Thanks for your replay Igor,

But how can it be? Since urlmon is the only one who creates those objects, and works with them, I don't even have access to that objects.

The one thing I do is to initialize:

typedef PassthroughAPP::CMetaFactory<PassthroughAPP::CComClassFactoryProtocol, CWebIEInetAsyncPlugableProtocol> MetaFactory;

....

CComPtr<IInternetSession> spSession;
	CoInternetGetSession(0, &spSession, 0);

// Create and register our custom HTTP protocol factory
HRESULT hr = MetaFactory::CreateInstance(CLSID_HttpProtocol, &m_spCFHTTP);

if(FAILED(hr)) return S_FALSE;
hr = spSession->RegisterNameSpace(m_spCFHTTP, CLSID_NULL, CComBSTR("http"), 0, 0, 0);
if(FAILED(hr)) return S_FALSE;


// Create and register our custom HTTPS protocol factory	
hr = MetaFactory::CreateInstance(CLSID_HttpSProtocol, &m_spCFHTTPS);
if(FAILED(hr)) return S_FALSE;

hr = spSession->RegisterNameSpace(m_spCFHTTPS, CLSID_NULL, CComBSTR("https"), 0, 0, 0);
if(FAILED(hr)) return S_FALSE;

CWebIEInetAsyncPlugableProtocol::Init();

	

Thanks, Yehuda

Answer 3

yehuda s wrote:

But how can it be? Since urlmon is the only one who creates those  objects, and works with them, I don't even have access to that
objects.

What do you mean, you don't have access? CWebIEInetAsyncPlugableProtocol  is your class, isn't it?

The one thing I do is to initialize:

Do you do this only once, or once per "BP", whatever that is? What is a  BP, by the way? How exactly do you "delete the cache"?

---

Igor Tandetnik

Answer 4

Hi Igor,

It is my class but it is not an instance I created, so unless I will save the 'this' in the ctor, I can't access the instance.

BP = Business Process = flow of a user on some site. I am initializing only once.

I am deleting the cache using wininet functions: FindFirstUrlCacheEntry, DeleteUrlCacheEntry, FindNextUrlCacheEntry.

Thanks

Answer 5

On 3/29/2012 10:03 AM, yehuda s wrote:

It is my class but it is not an instance I created, so unless I will save the 'this' in the ctor, I can't access the instance.

The instance can certainly access itself, in all its member functions. Somehow, the object ends up AddRef'ing itself, or passing its interface pointer somewhere which AddRefs it.

Let's put it this way: do you observe similar leaks in my sample application? If not, your code is doing something wrong.

---

Igor Tandetnik

Answer 6

Hi Igor, sorry for the long time to reply (weekend...).

I checked your demo application and found that it is indeed leaking, and the objects that are leaking are in the same stack trace that my application is leaking.

I checked 2 scenarios:

1. I changed the OnGo function to always perform a navigate to same site (www.qfly.com in my test), and for about ~10 min I (to be precise, a tool that simulates mouse clicks - AutoMouse) clicked the Go button, clicked some links in the site and clicked Go again....

This is the perfmon (private bytes):



2. I added a function in order to delete the cache, and a button to invoke it. I again changed the OnGo function as above. For ~1 hour, I clicked the OnGo, some links and the Delete button .... 

The perfmon (only ~15 min of it):

The stack trace of the objects leaked (same as the one in my first flow):

        76f8df42 ntdll!RtlAllocateHeap+0x00000274

        b9ea43 PassthruAPP!_heap_alloc_base+0x00000053

        b8e74c PassthruAPP!_heap_alloc_dbg_impl+0x000001fc

        b8e4bf PassthruAPP!_nh_malloc_dbg_impl+0x0000001f

        b8e45c PassthruAPP!_nh_malloc_dbg+0x0000002c

        b9393b PassthruAPP!malloc+0x0000001b

        b94421 PassthruAPP!operator new+0x00000011

        b53fe0 PassthruAPP!ATL::CComCreator<ATL::CComAggObject<CTestAPP> >::CreateInstance+0x000000a0

        b52113 PassthruAPP!ATL::CComCreator2<ATL::CComCreator<ATL::CComObject<CTestAPP> >,ATL::CComCreator<ATL::CComAggObject<CTestAPP> > >::CreateInstance+0x00000073

        b52dc8 PassthruAPP!ATL::CComClassFactory::CreateInstance+0x000000c8

        b522cd PassthruAPP!PassthroughAPP::CComClassFactoryProtocol::CreateInstance+0x0000010d

        747f923f urlmon!COInetSession::CreateFirstProtocol+0x000000dd

        747f8ad8 urlmon!CTransaction::StartEx+0x00000292

        6d462ab2 MSHTML!CTridentFilterHost::StartFilter+0x00000083

        6d526331 MSHTML!CDwnBindData::Bind+0x000004d5

        6d521781 MSHTML!NewDwnBindData+0x0000019d

        6d5215bd MSHTML!CDwnLoad::Init+0x0000025c

        6d56a592 MSHTML!CScriptLoad::Init+0x00000075

        6d6c2337 MSHTML!CDwnInfo::SetLoad+0x0000011e

        6d6c223d MSHTML!CDwnCtx::SetLoad+0x00000086

        6d551397 MSHTML!CDoc::NewDwnCtx2+0x0000030a

        6d52c83d MSHTML!CDoc::NewDwnCtx+0x0000005b

        6d5611f3 MSHTML!CScriptData::DownLoadScript+0x00000238

        6d3deb34 MSHTML!CScriptData::OnSrcChange+0x0000016b

        6d3deac7 MSHTML!CScriptElement::OnPropertyChange+0x00000033

        6d59b533 MSHTML!BASICPROPPARAMS::SetStringProperty+0x0000036a

        6d631932 MSHTML!BASICPROPPARAMS::SetUrlProperty+0x00000042

        6d34ec96 MSHTML!CBase::put_Url+0x00000036

        6d49ec86 MSHTML!GS_BSTR+0x0000016a

        6d5fac21 MSHTML!CBase::ContextInvokeEx+0x0000084c

        6d483bfc MSHTML!CElement::VersionedInvokeEx+0x00000068

        6d5083fc MSHTML!CBase::PrivateInvokeEx+0x00000082

I will be happy to send you your demo app with our changes, if it will be helpful. 

Thank you for your help.

Answer 7

yehuda s wrote:

I checked your demo application and found that it is indeed leaking,  and the objects that are leaking are in the same stack trace
that my application is leaking.

1. I changed the OnGo function to always perform a navigate to same  site (www.qfly.com in my test), and for about ~10 min I (to
be precise, a tool that simulates mouse clicks - AutoMouse) clicked  the Go button, clicked some links in the site and clicked Go
again.... 

I instrumented my code to log the calls to the APP's constructor and  destructor, and confirmed that, in each individual navigation to  www.qfly.com, every constructor call is eventually matched with a  destructor. So there doesn't seem to be a systemic problem with the  framework, leading to leaks. I haven't tried the stress test of the sort  you describe (too much work, I'm lazy).

Try adding similar instrumentation to your code. If you have a leak  (constructor/destructor mismatch) on every navigation, the problem is  likely in your code. If you only have this mismatch every once in a  while, randomly, then it's possible there's a bug in UrlMon where it  sometimes fails to release an APP.

I seem to recall people reporting memory consumption growth, similar to  what you describe, when simply navigating WebBrowser control around for  a long time, with no custom APP involved. So I wouldn't be surprised if  there's a leak somewhere in Microsoft's code. One possible way to work  around this issue: periodically, shut down and restart the process that  does the automatic navigation.

---

Igor Tandetnik

Answer 8

Hi Igor,

I used your demo app and also logged all ctor/dtor calls and as you said there is a match. There is no match when I am deleting the cache between the navigations (the BP). I believe that this is urlmon bug / misuse by me. Deleting the cache somehow causes it not calling the Release on the objects.

In order to overcome the problem, I pushed all the IUnkown* into a set in creation ( CComObjectSharedRef<Base>::CComObjectSharedRef(IUnknown* punkOuter)), and removed them in the Release (STDMETHODIMP_(ULONG) CComObjectSharedRef<Base>::Release()). In the end of each flow (BP) I run on the pointers and releasing them (calling Release). This caused the ctor/dtor calls to match. Ugly, but doing the job.

Do you think it may cause a problem? In the meantime from my testing it seems OK.

Thanks for your help, again.

Yehuda

Answer 9

On 4/4/2012 11:23 AM, yehuda s wrote:

I used your demo app and also logged all ctor/dtor calls and as you
said there is a match. There is no match when I am deleting the cache
between the navigations (the BP). I believe that this is urlmon bug /
misuse by me. Deleting the cache somehow causes it not calling the
Release on the objects.

Off the top of my head, I don't quite see how one can lead to the other, but I have never played games with the cache myself.

One thing to try: in your APP, intercept IInternetBindInfo::GetBindInfo call and add BINDF_GETNEWESTVERSION flag. This should effectively disable the cache, without the need to physically delete it (that is, the response is still written to the cache, but the subsequent request should ignore the cache entry and go to the origin server).

Another thing to try: between BPs, call InternetSetOption with INTERNET_OPTION_END_BROWSER_SESSION and/or INTERNET_OPTION_RESET_URLCACHE_SESSION. See if this changes anything. I'm not exactly sure what these do, except that they reset some kind of internal in-memory information that WinInet maintains.

---

Igor Tandetnik

Answer 10

Another thing to try: between BPs, call InternetSetOption with INTERNET_OPTION_END_BROWSER_SESSION and/or INTERNET_OPTION_RESET_URLCACHE_SESSION. See if this changes anything. I'm not exactly sure what these do, except that they reset some kind of internal in-memory information that WinInet maintains.

Hi Igor, I already tried these two options. The END_BROWSER_SESSION is causing any authentication in the session to end, the RESET_URLCACHE_SESSION I also don'e know, couldn't find any useful documentation on it.

I am imlementing the IInternetBindInfo, and I will try what you sugested. But if I navigate to url X two times inside one BP, will it disable the cache also in the                second navigation?     Thanks

Answer 11

On 4/4/2012 2:47 PM, yehuda s wrote:

I am imlementing the IInternetBindInfo, and I will try what you
sugested. But if I navigate to url X two times inside one BP, will it
disable the cache also in the               second navigation?

If you just throw the flag in unconditionally, it will, yes. I suppose you could keep a list of URLs observed since the beginning of the current BP, and only set the flag for URLs you see for the first time.

---

Igor Tandetnik

Answer 12

I am afraid that I will find my self implementing a mechanizm of cache... I think I am starting to like the memory leak :) 

Thank you for your help.