locked
Calculating bytes sent in C++ Native Module? RRS feed

  • Question

  • User500639321 posted

    Hello,

    I am attempting to write a module that counts the bytes sent to the client.  I do not wish to calculate the bytes sent from the log files output.  I need to do this from inside a module.

    I need help working out where I can find this information, or calculate it, through the API.  Is there one definitive "number of bytes sent in this request" variable?

    Thanks,

    Paul.

    Thursday, March 20, 2008 2:37 AM

Answers

  • User511787461 posted

    The X-Powered-By header is added by the ProtocolSupportModule, which is probably running after you - you should register with PRIORITY_ALIAS_FIRST or PRIORITY_ALIAS_HIGH in RQ_SEND_RESPONSE if you want to run after that module (note that priorities are reversed in RQ_SEND_RESPONSE).  The other headers, Date/Connection/Content-Length are added by http.sys, so you will not see them in your module.  The server header is sent in that fashion to prevent http.sys from adding its own server version to the IIS generated server header.

    • Marked as answer by Anonymous Tuesday, September 28, 2021 12:00 AM
    Monday, March 31, 2008 3:58 PM

All replies

  • User113421904 posted

    Hi Paul,

    The native module call WriteClient to send data to the clients. You can check this function for the bytes to be sent.

     

    Tuesday, March 25, 2008 10:13 AM
  • User500639321 posted

    Hi Zhao Ji Ma,

    As far as I can tell, that's an API 1.0 function?  Is there a way to get to this information using the 2.0 API?  Or am I missing something?

    Thanks,

    Paul.
     

    Wednesday, March 26, 2008 1:28 AM
  • User113421904 posted

    Hi Paul,

    WriteClient is available since Windows NT 4.0,  actually the version (HSE_VERSION_INFO) should be 4.0 or later. So you can use it in IIS 7.

     

    Wednesday, March 26, 2008 10:45 PM
  • User500639321 posted

    I am writing an IIS7 module and have registered for the RQ_SEND_RESPONSE, and RQ_LOG_REQUEST notifications.

    I can't find where in an IIS7 modue "WriteClient" is called, nor can I find a simple "count of bytes sent" from the IHttpContext, ISendResponseProvider, or IHttpEventProvider interfaces supplied to OnSendResponse and OnLogRequest.

    I know this information must be available somewhere, because I have seen compression modules written, and presumably those modules need to get access to the raw buffers being sent to the client.  I think I should be able to do the same thing and add up the count of bytes being sent tho the client.  I also need to count the bytes being sent in the header, but can't see a "count of bytes in the entire header" number, either.

    I feel as though I am missing something here.  Can anyone help me with this one?

    Cheers,
    Paul.

    Wednesday, March 26, 2008 11:41 PM
  • User511787461 posted

    You can see the current response entity using pHttpContext->GetResponse()->GetRawHttpResponse()->pEntityChunks (and EntityChunkCount) - there is no way to just get the number of bytes already sent out.

    Thursday, March 27, 2008 11:32 AM
  • User500639321 posted

    I thought this might be the case.  Does the entity chunk data also count the bytes in the headers?  Or do I need to rip through the Headers structure to get that also?  [I would love to be able to debug the module running under iis7 but currently have not got that set up!]

    Cheers,
    Paul.
     

    Thursday, March 27, 2008 11:37 AM
  • User500639321 posted

    I thought this might be the case.  Does the entity chunk data also count the bytes in the headers?  Or do I need to rip through the Headers structure to get that also?  [I would love to be able to debug the module running under iis7 but currently have not got that set up!]

    Cheers,
    Paul.
     

    Thursday, March 27, 2008 11:37 AM
  • User511787461 posted

    No, that is just the response entity, headers are separate.

    Thursday, March 27, 2008 11:40 AM
  • User500639321 posted

    I have code to interpret the data in the headers structure now, to add up all the headers going out.  I have found some oddities, however, that I cannot explain.  For example, if I telnet to the server and paste a request in manually, I get the following response:

    HTTP/1.1 500 Internal Server Error
    Cache-Control: private
    Content-Type: text/html; charset=utf-8
    Server: Microsoft-IIS/7.0
    X-AspNet-Version: 2.0.50727
    X-Powered-By: ASP.NET
    Date: Mon, 31 Mar 2008 00:37:59 GMT
    Connection: close
    Content-Length: 7497

    Inside the HTTP Module, the code does the following:

    1. Adds up the module status, HTTP version, and reason lengths,

    2. Enumerates all the known headers and checks for Headers.KnownHeaders[i].RawValueLength > 0, if it is, then the length of the header + ': ' + RawValueLength is added to the tally.

    3. The _unknown_ headers are enumerated and added up (Headers.pUnknownHeaders[i].NameLength + 2 + Headers.pUnknownHeaders[i].RawValueLength + 2).

    The problem I am seeing is that I only get 1/2 of the headers in the LogResponse notification coming through.  For example, this is a log file produced by the module:

    Logged 36 bytes: HTTP/1.1 500 Internal Server Error
    Known header 0's length is 7, and the pointer is not null
    Logged 24 bytes: Cache-Control: private
    Known header 1's length is 0, and the pointer is null
    Known header 2's length is 0, and the pointer is null
    Known header 3's length is 0, and the pointer is null
    Known header 4's length is 0, and the pointer is null
    Known header 5's length is 0, and the pointer is null
    Known header 6's length is 0, and the pointer is null
    Known header 7's length is 0, and the pointer is null
    Known header 8's length is 0, and the pointer is null
    Known header 9's length is 0, and the pointer is null
    Known header 10's length is 0, and the pointer is null
    Known header 11's length is 0, and the pointer is null
    Known header 12's length is 24, and the pointer is not null
    Logged 40 bytes: Content-Type: text/html; charset=utf-8
    Known header 13's length is 0, and the pointer is null
    Known header 14's length is 0, and the pointer is null
    Known header 15's length is 0, and the pointer is null
    Known header 16's length is 0, and the pointer is null
    Known header 17's length is 0, and the pointer is null
    Known header 18's length is 0, and the pointer is null
    Known header 19's length is 0, and the pointer is null
    Known header 20's length is 0, and the pointer is null
    Known header 21's length is 0, and the pointer is null
    Known header 22's length is 0, and the pointer is null
    Known header 23's length is 0, and the pointer is null
    Known header 24's length is 0, and the pointer is null
    Known header 25's length is 0, and the pointer is null
    Known header 26's length is 0, and the pointer is not null
    Known header 27's length is 0, and the pointer is null
    Known header 28's length is 0, and the pointer is null
    Known header 29's length is 0, and the pointer is null
    Logged 27 bytes: Server: Microsoft-IIS/7.0
    Logged 29 bytes: X-AspNet-Version: 2.0.50727
    Logged 2 bytes:

    Checking by hand, all the byte values are correct (The byte values also count the \r\n at the end of each header, so they're out by two visually.

    My questions are:

    1. Why am I only seeing 1/2 the headers?  ie: Known Headers 'Date', 'Connection', and 'Content-Length' are missing.  Unknown Header 'X-Powered-By' is missing.

    2. Why is the mysterious known header 26 (HttpHeaderServer) non null but zero length?

    3. Why is the "server" header appearing in my log as an unknown header?

    For those who are curious, the code that produced the above is as follows:

        PHTTP_RESPONSE pHttpResponse = pResponse->GetRawHttpResponse();

        DWORD dwHeaders = 0;

        HANDLE hF = ::CreateFile("C:\\Temp\\Response.txt", GENERIC_WRITE, FILE_SHARE_READ, NULL, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);
        char sz [256];
        DWORD dw = 0;

        // Tally headers
        dwHeaders += 4; // HTTP response code (eg: "200 " )
        dwHeaders += pHttpResponse->ReasonLength + 1; // HTTP Response reason, ie: "Not Found "
        if ( pHttpResponse->Version.MajorVersion > 0 )
        {
            dwHeaders += 10; // Version, eg: HTTP/1.1\r\n
        }

        dw = sprintf(sz,
            "Logged %d bytes: HTTP/%d.%d %d %s\r\n",
            dwHeaders,
            pHttpResponse->Version.MajorVersion,
            pHttpResponse->Version.MinorVersion,
            pHttpResponse->StatusCode,
            pHttpResponse->pReason);

        ::WriteFile(hF, sz, dw, &dw, NULL);

        for ( int i = 0; i < HttpHeaderResponseMaximum; i++ )
        {
            dw = sprintf(sz,
                "Known header %d's length is %d, and the pointer is %s\r\n",
                i,
                pHttpResponse->Headers.KnownHeaders[i].RawValueLength,
                ((pHttpResponse->Headers.KnownHeaders[i].pRawValue == NULL) ? "null" : "not null"));

            ::WriteFile(hF, sz, dw, &dw, NULL);

            dwHeaders += pHttpResponse->Headers.KnownHeaders[i].RawValueLength;
            if ( pHttpResponse->Headers.KnownHeaders[i].RawValueLength > 0 )
            {
                dwHeaders += KNOWN_HEADER_NAME_LENGTH[i];
                dwHeaders += 2; // the \r \n

                dw = sprintf(sz,
                    "Logged %d bytes: %s%s\r\n",
                    pHttpResponse->Headers.KnownHeaders[i].RawValueLength + KNOWN_HEADER_NAME_LENGTH[i] + 2,
                    KNOWN_HEADERS[i],
                    pHttpResponse->Headers.KnownHeaders[i].pRawValue);

                ::WriteFile(hF, sz, dw, &dw, NULL);
            }
        }
        for ( int i = 0; i < pHttpResponse->Headers.UnknownHeaderCount; i++ )
        {
            dwHeaders +=
                pHttpResponse->Headers.pUnknownHeaders[i].NameLength +
                2 + // the ': '
                pHttpResponse->Headers.pUnknownHeaders[i].RawValueLength +
                2; // the \r \n

            dw = sprintf(sz,
                "Logged %d bytes: %s: %s\r\n",
                pHttpResponse->Headers.pUnknownHeaders[i].NameLength + 2 + pHttpResponse->Headers.pUnknownHeaders[i].RawValueLength + 2,
                pHttpResponse->Headers.pUnknownHeaders[i].pName,
                pHttpResponse->Headers.pUnknownHeaders[i].pRawValue);

            ::WriteFile(hF, sz, dw, &dw, NULL);
        }
        dwHeaders += 2; // The trailing /r/n at the end of the header block.

        dw = sprintf(sz,
            "Logged 2 bytes: \r\n");

        ::WriteFile(hF, sz, dw, &dw, NULL);

        ::CloseHandle(hF);

    Cheers,

    Paul. 

    Monday, March 31, 2008 12:52 AM
  • User511787461 posted

    The X-Powered-By header is added by the ProtocolSupportModule, which is probably running after you - you should register with PRIORITY_ALIAS_FIRST or PRIORITY_ALIAS_HIGH in RQ_SEND_RESPONSE if you want to run after that module (note that priorities are reversed in RQ_SEND_RESPONSE).  The other headers, Date/Connection/Content-Length are added by http.sys, so you will not see them in your module.  The server header is sent in that fashion to prevent http.sys from adding its own server version to the IIS generated server header.

    • Marked as answer by Anonymous Tuesday, September 28, 2021 12:00 AM
    Monday, March 31, 2008 3:58 PM
  • User500639321 posted

    Thank you for the PRIORITY_ALIAS_HIGH tip, will do that.

    Just to triply confirm - there is no way to get an accurate count of bytes sent to the client from within an IIS7 module, because HTTP.SYS is adding headers without any notification to / interaction with the IIS7 module?

    Thanks,
    Paul.

    Monday, March 31, 2008 7:20 PM
  • User511787461 posted

    Yes - although you can calculate the Content-Length and Connection header based on the other data you already have and guess Date within a few seconds, also http.sys can send a 100 continue response for POST requests which IIS has no visibility/control over.  Also, there are error responses for certain errors sent directly from http.sys.  Also, in cases where response is send in multiple flushes to http/1.1 clients, IIS will auto-chunk-transfer-encode response which is also completely transparent to modules.

    Monday, March 31, 2008 7:32 PM
  • User-1258788955 posted

    Having followed the discussion above, I am curious about how to find the size of the entity chunks mentioned. The chunks appear to come in different chunk types  ( HttpDataChunkFromMemory, HttpDataChunkFromFileHandle, HttpDataChunkFromFragmentCache, HttpDataChunkFromFragmentCacheEx) so I would expect to iterate arond the EntityChunks obtained from pHttpContext->GetResponse()->GetRawHttpResponse()->pEntityChunks and treat each one differently according to its type (as provided in pEntityChunks[N].DataChunkType) but only  the HttpDataChunkFromMemory type has a readily apparent size. Looking at the available properties I see:

    pEntityChunks[N].FromMemory.BufferLength // I'm guessing this is the size of a chunk coming from memory

    pEntityChunks[N].FromFileHandle.ByteRange.Length  // I'm hoping this is the size of a chunk coming from a file

    pEntityChunks[N].FromFragmentCache // doesn't seem to have a ready size field

    pEntityChunks[N].FromFragmentCacheEx.ByteRange.Length // hoping this is chunk size again

    Can anyone help here?
     

     
     

    Thursday, May 29, 2008 2:05 PM
  • User511787461 posted

    You cannot calculate the length of FromFragmentCache HTTP_DATA_CHUNK.

    Friday, May 30, 2008 8:17 PM
  • User-1258788955 posted

    Thanks anilr, 

    So is there a way to calculate the bytes sent at all? In IIS6 I used to do this with an ISAPI filter where it was trivial to trap the SF_NOTIFY_LOG event and cast the input as HTTP_FILTER_LOG and pick off the bytes sent (and bytes received). I thought of using this filter in IIS7 but isn't ISAPI filter support provided by a native module (isapiModule) now anyway? So how could such a filter obtain this information in IIS7? There appears to be no way to calculate bytes sent in a .net module either.

     Any suggestions would be gratefully received?

    Monday, June 2, 2008 4:22 AM
  • User-777241625 posted

    I stuck into the same issue about get size of HTTP headers' block.

    After playing with all thing around I've found out that call:

    DWORD bytesSent;
    BOOL completionExpected;
    hr = pHttpContext->GetResponse()->Flush(FALSE, FALSE, &bytesSent, &completionExpected);
    

    returns actual whole response size value in bytesSent.

    However, this call of Flush is somehow prevent normal IIS logging. Even if it's marked by PRIORITY_ALIAS_HIGH to be processed as last handler. 
    Anyway, I would like to know if there is exists a way to get actual response size without blocking any other functionality. After all, AdvancedLoggingModule can get it somehow. (I know that I can get necessary info from AdvancedLoggingModule, but in my case is very important to avoid any external dependencies).

    Thursday, May 30, 2013 4:40 AM