none
memory leaks when using FwpsCloneStreamData0 [KB 979223]

    Question

  • Hello,

    I have a WFP driver which filters TCP connections through a user mode application. In my stream handler I call FwpsCloneStreamData0 to keep a reference to the data stream, I return FWP_ACTION_BLOCK, send the data to the user-mode application, call FwpsStreamInjectAsync0 to reinject the data back into the stream and in the completion routine I call FwpsFreeCloneNetBufferList0 as specified.

    The data reaches my user-mode application and the client application fine, but this process generates a huge number of non-paged memory leaks. Poolused shows that the leaks are NET_BUFFERs and NET_BUFFER_LISTs objects allocated in netio.sys. I have added debug messages through out my code and all the cloned data is freed properly. The number of leaks is huge and the issue reproduces consistently so I doubt that there is a race condition somewhere.

    Driver verifier is enabled for my driver. The OS is Windows 7 build 7100 but I have received reports that the problem also occurs on WIndows 7 build 7600.

    Can anyone give me a clue about what I might be doing wrong because I have run out of ideas (aside from manually copying data)?
    Thank you!
    Thursday, October 01, 2009 7:27 AM

Answers

  • I can confirm the leak is due to a bug in WFP -- if a multi-NBL chain is blocked the last NBL is leaked.

     

    The leak affects Vista as well as Win7.

    Please contact your Microsoft Rep to request a hotfix for Vista and/or Win7.

    Thanks,
    Biao.W.

    Friday, October 09, 2009 7:22 AM
    Owner
  • The fix is located here: http://support.microsoft.com/kb/979223.

    Hope this helps
    Dusty Harper [MSFT]
    Microsoft Corporation
    ------------------------------------------------------------
    This posting is provided "AS IS", with NO warranties and confers NO rights
    ------------------------------------------------------------
    Tuesday, February 09, 2010 2:34 AM
    Moderator
  • It looks like they have made a fix for Vista but are still testing it.  If/When it gets released, the fix for Vista will be at that same URL.
    Wednesday, March 24, 2010 9:17 PM

All replies

  • Hi,

     I am just analyzing the same problem in our WFP driver. I can see exactly the same symptoms as Cristian. Additionaly, it seems that the leak occurs only if the cloned net buffer list chain consists of more than one net buffer list.

     I have reproduced this on both Vista SP1 and Windows 7 RTM.

    Ron
    Friday, October 02, 2009 7:54 AM
  • Hi Ron,

    Well, I'm sorry that you also have this problem, but at least I stopped thinking that I'm going crazy.

    More info:

    It seems that the culprit is the combo FwpsCloneStreamData0/FWP_ACTION_BLOCK/FwpsStreamInjectAsync0/FwpsFreeCloneNetBufferList0.
    If I do FwpsCloneStreamData0/FwpsStreamInjectAsync0/FWP_ACTION_BLOCK/FwpsFreeCloneNetBufferList0 there are no memory leaks.
    The leaked memory is about 20% of the amount of transfered data. Poolused shows that leaks are NET_BUFFER_LISTs and NET_BUFFERs from netio.sys, which would indicate that the NBL clones' parents are not freed properly. I also think that Ron's theory might be true.

    I guess that I'm going back to manually copying data (as I did in TDI filters). Too bad, I liked this mechanism more.

    Friday, October 02, 2009 10:18 PM
  • Hi, me again.

    Can anyone confirm if this is my bug or Microsoft's?

    I started implementing manually copying of data, but now I have another problem: I can't pend send operations. For incoming data I can return FWPS_STREAM_ACTION_DEFER, but the documentation specifies that this method will not work for outgoing data.  Is there a workaround for this?

    Thanks.
    Tuesday, October 06, 2009 4:30 PM
  • Hi Cristian,

     it turned out (at least in my case), that the problem is not related to FwpsCloneStreamData0 at all. It seems it is much more general: it occurs when a stream classify callout blocks a stream fragment which contains more than one net buffer list in the net buffer list chain. I have a minimal example demonstrating this problem. It simply registers a FWPM_LAYER_STREAM_V4 callout, which blocks only incoming http reply data (see below). Then I use netcat to send a http request somewhere, and monitor KdPrint output in WinDbg, and pool allocations in poolmon. For every "multi-buffer block" KdPrint I see in WinDbg, there is exactly one new unfreed pool allocation with tag 'Nbuf' (yes, net buffer from netio.sys).

     So unless Your problem is different from mine (but the symptoms are exactly the same), manual data copying will probably not help You.

     Hopefully someone from MS can comment on this, and give us some advice.

     Implementation of my callout demonstrating the leak (the full source including driver initialization and callout registration is cca 170 LOC, I can post it if needed):

    void StreamClassify(
    	IN const FWPS_INCOMING_VALUES0* inFixedValues,
    	IN const FWPS_INCOMING_METADATA_VALUES0* inMetaValues,
    	IN OUT void* layerData,
    	IN const FWPS_FILTER0* filter,
    	IN UINT64 flowContext,
    	OUT FWPS_CLASSIFY_OUT0* classifyOut)
    {
    	FWPS_STREAM_CALLOUT_IO_PACKET0 *pIoPacket = (FWPS_STREAM_CALLOUT_IO_PACKET0*)layerData;
    	FWPS_STREAM_DATA0 *pStreamData = pIoPacket->streamData;
    
    	int rport = inFixedValues->incomingValue[FWPS_FIELD_STREAM_V4_IP_REMOTE_PORT].value.uint16;
    	bool isHttpReply = (rport == 80) && (pStreamData->flags & FWPS_STREAM_FLAG_RECEIVE);
    
    	if (isHttpReply)
    	{
    		if (pStreamData->netBufferListChain->NetBufferListHeader.NetBufferListData.Next)
    			KdPrint(("multi-buffer block"));
    
    		pIoPacket->streamAction = FWPS_STREAM_ACTION_NONE;
    		classifyOut->actionType = FWP_ACTION_BLOCK;
    	}
    	else
    	{
    		pIoPacket->streamAction = FWPS_STREAM_ACTION_NONE;
    		classifyOut->actionType = FWP_ACTION_PERMIT;
    	}
    }
    


    Ron
    Wednesday, October 07, 2009 9:07 AM
  • One more thing: to bring up the memory leak, I have to do a http download from the internet. When I try it on local network, there is always just one net buffer list in the chain, and the leak does not occur. I presume it is related to timing and/or fragmenting of incoming tcp packets.

    Ron
    Wednesday, October 07, 2009 9:54 AM
  • Hi Ron,

    Unfortunately, you are right. Even if I don't call FwpsCloneStreamData0, the problem still occurs. The leak is about 25%-30% of the transferred data, so your theory about fragmented packets and/or multiple net buffer lists holds. I remembered that I tried injecting data in the context of the stream callout and no leaks occurred so I assumed that if I block data without cloning it the issue will be solved. I was wrong.

    I have officially run out of options.
    Wednesday, October 07, 2009 1:35 PM
  • We'll look into this. We'll see whether we can reproduce the leaks in our lab.

    Thanks,
    Biao.W.

    Thursday, October 08, 2009 1:33 AM
    Owner
  • I can confirm the leak is due to a bug in WFP -- if a multi-NBL chain is blocked the last NBL is leaked.

     

    The leak affects Vista as well as Win7.

    Please contact your Microsoft Rep to request a hotfix for Vista and/or Win7.

    Thanks,
    Biao.W.

    Friday, October 09, 2009 7:22 AM
    Owner
  • Hi Biao,

    Thank you for taking the time to verify and confirm this issue.

    Can you give us an estimate of how long will be until this fix reaches Windows Update? We have a lot of clients using our driver and I doubt that we can redistribute the fix. Or we can?
    Friday, October 09, 2009 1:54 PM
  • Cristian,

    Thanks for reporting the issue. Unfortunately I myself am not familiar with the Win7 servicing process so please contact your MS PSS Rep for an answer. you can reference this thread.

    Thanks,
    Biao.W.

    Saturday, October 10, 2009 12:17 AM
    Owner
  • Can you provide the KB/hotfix number so we know what to ask for and so we're not on a wild goose chase?
    Monday, November 09, 2009 5:32 PM
  • Is this hotfix available for ordinary users? If not yet, then when it will be? More than one month has passed from Mr. Wang's answer and still no sign of the patch...
    Friday, November 13, 2009 5:24 PM
  • Till now I can see the only request for Win7 QFE and no Vista hotfix for this issue. And the patch release for Win7 hotfix is under progress.
    I doubt if the QFEs are available to ordinary users, you need to contact Microsoft PSS team for the same.


    Thanks,
    Satyendra
    Saturday, November 14, 2009 5:05 PM
  • Thanks for reporting the problem! We're still working on the issue and will update this thread for further progress.

    Thanks,
    Charlie [MSFT]
    Charlie
    Tuesday, November 17, 2009 6:18 PM
  • Any Update on this? Because this Memory Leak is very annoying especially if you have NOD32 Anti-Virus running and you are downloading big files which this thread shows:
    http://social.technet.microsoft.com/Forums/en/w7itpronetworking/thread/38194b3f-e713-484b-8a16-c8154b9acff0

    Thursday, January 07, 2010 3:25 AM
  • I can confirm the memory leak affecting nod32 4.0, after 2 weeks of running non stop ekrn.exe, the AV kernel, climbs from 50MB RAM slowly to 130, and continues its steady climb each day without end.
    Tuesday, January 19, 2010 5:20 PM
  • Hello guys, any update on this? It's really annoying. I had NOD32 before, but I have removed it just in order to workaround this issue. I installed Micorsoft Security Essential, but the problem persist!!! When I download large files the memory leaks like crazy.
    Wednesday, January 27, 2010 8:29 AM
  • Current plans has this as a February release.  Once the package has been released, I'll post a link to the fix's location.

    Hope this helps

    Dusty Harper [MSFT]
    Microsoft Corporation
    ------------------------------------------------------------
    This posting is provided "AS IS", with NO warranties and confers NO rights
    ------------------------------------------------------------
    Wednesday, January 27, 2010 6:32 PM
    Moderator
  • The fix is located here: http://support.microsoft.com/kb/979223.

    Hope this helps
    Dusty Harper [MSFT]
    Microsoft Corporation
    ------------------------------------------------------------
    This posting is provided "AS IS", with NO warranties and confers NO rights
    ------------------------------------------------------------
    Tuesday, February 09, 2010 2:34 AM
    Moderator
  • Dusty,
    I have the same problem with Eset and Vista Ultimate x64. Is there a fix coming out for Vista?
    Thanks
    Wednesday, March 24, 2010 11:19 AM
  • It looks like they have made a fix for Vista but are still testing it.  If/When it gets released, the fix for Vista will be at that same URL.
    Wednesday, March 24, 2010 9:17 PM
  • It looks like they have made a fix for Vista but are still testing it.  If/When it gets released, the fix for Vista will be at that same URL.

    Thanks for the info.
    EDIT :-   The Vista fix is now available. Remains to be seen whether it works!
    Thursday, March 25, 2010 9:25 AM
  • Just for the benefit of anyone using Eset Smart security or Nod antivirus with Vista, I can confirm that the fix for Vista now available from http://support.microsoft.com/kb/979223 does fix the memory leak and subsequent crash that occurs when downloading a number of large files with programs such as Internet Download Manager etc.
    Hopefully, Google will lead them to this information.
    • Edited by meditek Sunday, May 16, 2010 8:55 AM spelling
    Sunday, May 16, 2010 8:54 AM
  • Hmm strange, when I access the link for the Hotfix I only get windows Vista as an OS option even though I am running Windows 7 Professional x64. I've tried to install the Vista variant but it says "The update is not applicable to your computer". So now I am assuming I require the Windows 7 Hotfix but I cannot access it?
    Wednesday, August 11, 2010 9:37 PM