locked
How can i view entire HTML payload? RRS feed

  • Question

  • I'm using NetMon 3.4 (on Windows 7 64 bit) to view HTTP traffic, but NetMon doesn't seem to capture the entire HTTP payload. Even after I save the capture and reassemble, when I look at the response from a site, it shows me payload:HttpContentType:text/html;charset=UTF-8 but then in the next line, I see:

    HTMLElement:<some garbled text here>.

    Is there a way to capture the entire HTML response via Netmon (like Fiddler)?

     

    Thanks,

    Priya

    Thursday, May 12, 2011 12:12 PM

Answers

  • The chunked encoding would require reassembling the trace.  In the example above, the frame you are looking at has not been reassembled.  You need to find the related frame using the filter I showed above after you reassembled the trace. Another problem with Chunked data is that we cannot always reassemble this.  But you could manually copy the data from each full chunk, after reassembly, from the hex details.  This could allow you to stitch the text data back together manually.

    Paul

    • Marked as answer by Priya_M Thursday, May 12, 2011 3:32 PM
    Thursday, May 12, 2011 3:12 PM

All replies

  • When you perform reassembly, new frames are inserted but the original frames remain.  To make things easier to see, you can filter on only complete frames using "Property.HTTPCompleteFrame==1".  This requires the latest parsers from http://nmparsers.codeplex.com.  This Blog also has more informaiton about this feature.  There are also videos on the blog and one in particular about reassembly.

    Now there might be another question as to where your frame is HTML or some binary data.  To help us you could right click the frame details and copy to clipboard.  Then paste the results here and that might give us more information.

    Thanks,

    Paul

     

    Thursday, May 12, 2011 1:57 PM
  • Thank you for your response. I've pasted the contents of the frame below. As you can see, the payload contains partial encoded text rather than the actual HTML text.

     

      Frame: Number = 42, Captured Frame Length = 1556, MediaType = WiFi

    - WiFi: [Unencrypted Data] F.....P, (I) RSSI = -46 dBm, Rate = 54.0 Mbps

      - MetaData: RSSI = -46 dBm, Rate = 54.0 Mbps

         Version: 2 (0x2)

         Length: 32 (0x20)

       - OpMode: Extensible Station Mode

          StationMode:           (...............................0) Not Station Mode

          APMode:                (..............................0.) Not AP Mode

          ExtensibleStationMode: (.............................1..) Extensible Station Mode

          Unused:                (.0000000000000000000000000000...)

          MonitorMode:           (0...............................) Not Monitor Mode

         Flags: 0 (0x0)

         PhyType: Undefined Value (0)

         Channel: Undefined PhyType 0, Center Frequency: 2462 MHz

         lRSSI: -46 dBm

         Rate: 54.0 Mbps

         TimeStamp: 05/12/2011, 14:38:35.721503 UTC

      - FrameControl: Version 0,Data, Data, F.....P(0x4208)

         Version:        (..............00) 0

         Type:           (............10..) Data

         SubType:        (........0000....) Data

         DS:             (......10........) DS to STA via AP

         MoreFrag:       (.....0..........) No

         Retry:          (....0...........) No

         PowerMgt:       (...0............) Active Mode

         MoreData:       (..0.............) No

         ProtectedFrame: (.1..............) Yes

         Order:          (0...............) Unordered

        Duration: 44 (0x2C)

        DA: 002314 B8A2AC

        BSSID: Netgear Inc. FA26AA

        SA: Netgear Inc. FA26AA

      - SequenceControl: Sequence Number = 3702

         FragmentNumber: (............0000) 0

         SequenceNumber: (111001110110....) 3702

    - LLC: Unnumbered(U) Frame, Command Frame, SSAP = SNAP(Sub-Network Access Protocol), DSAP = SNAP(Sub-Network Access Protocol)

      - DSAP: SNAP(Sub-Network Access Protocol), Individual DSAP

         Address: (1010101.) SNAP(Sub-Network Access Protocol)

         IG:      (.......0) Individual Address

      - SSAP: SNAP(Sub-Network Access Protocol), Command

         Address: (1010101.) SNAP(Sub-Network Access Protocol)

         CR:      (.......0) Command Frame

      - Unnumbered: UI - Unnumbered Information

         MMM:  (000.....) 0

         PF:   (...0....) Poll Bit - No Response Solicited

         MM:   (....00..)

         Type: (......11) Unnumbered(U) Frame

    - Snap: EtherType = Internet IP (IPv4), OrgCode = XEROX CORPORATION

        OrganizationCode: XEROX CORPORATION, 0(0x0000)

        EtherType: Internet IP (IPv4), 2048(0x0800)

    - Ipv4: Src = 98.139.50.166, Dest = 192.168.0.5, Next Protocol = TCP, Packet ID = 10162, Total IP Length = 1492

      - Versions: IPv4, Internet Protocol; Header Length = 20

         Version:      (0100....) IPv4, Internet Protocol

         HeaderLength: (....0101) 20 bytes (0x5)

      - DifferentiatedServicesField: DSCP: 0, ECN: 0

         DSCP: (000000..) Differentiated services codepoint 0

         ECT:  (......0.) ECN-Capable Transport not set

         CE:   (.......0) ECN-CE not set

        TotalLength: 1492 (0x5D4)

        Identification: 10162 (0x27B2)

      - FragmentFlags: 16384 (0x4000)

         Reserved: (0...............)

         DF:       (.1..............) Do not fragment

         MF:       (..0.............) This is the last fragment

         Offset:   (...0000000000000) 0

        TimeToLive: 49 (0x31)

        NextProtocol: TCP, 6(0x6)

        Checksum: 50835 (0xC693)

        SourceAddress: 98.139.50.166

        DestinationAddress: 192.168.0.5

    - Tcp: Flags=...A...., SrcPort=HTTP(80), DstPort=6332, PayloadLen=1452, Seq=3411860115 - 3411861567, Ack=2158906032, Win=31 (scale factor 0x8) = 7936

        SrcPort: HTTP(80)

        DstPort: 6332

        SequenceNumber: 3411860115 (0xCB5CDA93)

        AcknowledgementNumber: 2158906032 (0x80AE4AB0)

      - DataOffset: 80 (0x50)

         DataOffset: (0101....) 20 bytes

         Reserved:   (....000.)

         NS:         (.......0) Nonce Sum not significant

      - Flags: ...A....

         CWR:    (0.......) CWR not significant

         ECE:    (.0......) ECN-Echo not significant

         Urgent: (..0.....) Not Urgent Data

         Ack:    (...1....) Acknowledgement field significant

         Push:   (....0...) No Push Function

         Reset:  (.....0..) No Reset

         Syn:    (......0.) Not Synchronize sequence numbers

         Fin:    (.......0) Not End of data

        Window: 31 (scale factor 0x8) = 7936

        Checksum: 0xA2EB, Good

        UrgentPointer: 0 (0x0)

        TCPPayload: SourcePort = 80, DestinationPort = 6332

    - Http: Response, HTTP/1.1, Status: Ok, URL: / 

        ProtocolVersion: HTTP/1.1

        StatusCode: 200, Ok

        Reason: OK

        Date:  Thu, 12 May 2011 14:38:33 GMT

        P3P:  policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"

        Set-Cookie:  searchTray=deleted; expires=Wed, 12-May-2010 14:38:33 GMT; path=/; domain=.delicious.com

        Pragma:  no-cache

        Cache-Control:  no-store, must-revalidate, no-cache, private, max-age=0, post-check=0, pre-check=0

        X-Xss-Protection:  0

        Expires:  Sun, 1 Jan 2006 01:00:00 GMT

        X-Ua-Compatible:  IE=7

        Set-Cookie:  delicious_us_production=v20O3Czr3Cy2.11hbnNTQPmOcsV1eDk6gW3EcBakO2cQ9Xxdcmjj0_G08jdFN9CqSWO4K4amNBhs2Jcsec5TGT3bHP0lQ2PcroWnIroe4ZtEel6U4Ie8IMRv1G9I.GZhep5c0yxMH4IH2L2mMNMaFXUHhAQC_7QnlH9t7ozQ2bJqEn0v2Vgt9hE8fzxeW3GsyoWjM8qVu694vkYKHxD0lz56oGuin1

        Vary:  Accept-Encoding

      - ContentType:  text/html; charset=UTF-8

       - MediaType:  text/html; charset=UTF-8

          MainType:  text/html

          charset: UTF-8

     

        ContentEncoding:  gzip

        Age:  2

        TransferEncoding:  chunked

        Connection:  keep-alive

        Server:  YTS/1.19.4

        HeaderEnd: CRLF

      - chunkSize: 7216

         Size: 7216

      - ChunkPayload: HttpContentType =  text/html; charset=UTF-8

         HtmlElement: ?

         HtmlElement: D5YQ>

         HtmlElement: ??ò¦ûF<hÔUMQ?Îj¨ÖOÓh[Qnnnê7F=?{J÷£ÒO}¯¡$iì?´n§vmoc?¥!½Ý

     ¡>

         HtmlElement: Å6»ÿv|?bÄèÉôßw¸[; ??©ÜE´??¸Û­¥ô6å,^#ÒÇqBÓÝOÝ·r?q?¢qã?K¥?P

    Thursday, May 12, 2011 2:41 PM
  • Hi Priya,

    If you look a couple of fields up you'll see "ContentEncoding: gzip" meaning that the stream is compressed.  Network Monitor doesn't decompress these types of streams.

    You'd have to use our API to save off the binary payload data from the reassembled frame and use some other function or program to uncompress the data.


    Michael Hawker | Program Manager | Network Monitor
    Thursday, May 12, 2011 2:55 PM
  • Ah, ok. Thank you for your response. I'm seeing other frames where transfer encoding is "chunked" and I have plain-text but incomplete data there. I'm guessing it's for a similar reason.

     

    Thanks for your time,

    Priya

    Thursday, May 12, 2011 3:00 PM
  • The chunked encoding would require reassembling the trace.  In the example above, the frame you are looking at has not been reassembled.  You need to find the related frame using the filter I showed above after you reassembled the trace. Another problem with Chunked data is that we cannot always reassemble this.  But you could manually copy the data from each full chunk, after reassembly, from the hex details.  This could allow you to stitch the text data back together manually.

    Paul

    • Marked as answer by Priya_M Thursday, May 12, 2011 3:32 PM
    Thursday, May 12, 2011 3:12 PM
  • Ok, that makes sense. Thank you!
    Thursday, May 12, 2011 3:32 PM