none
Large data file is attached twice in the http packet RRS feed

  • Question

  • I'm trying to send a large binary file via ONVIF. The Service Reference is auto-generated from the wsdl files using Visual Studio 2013 Update 5.

    I need to send the file via MTOM. So, I created a custom binding for that:

    private CustomBinding GetBindingWithMtomEncoding_() { return new CustomBinding( new MtomMessageEncodingBindingElement{ MessageVersion = MessageVersion.CreateVersion( EnvelopeVersion.Soap12, AddressingVersion.None)}, new HttpTransportBindingElement { AuthenticationScheme = AuthenticationSchemes.Anonymous, KeepAliveEnabled = true, TransferMode = TransferMode.StreamedRequest, MaxBufferSize = m_mtomBufferSize, MaxReceivedMessageSize = m_mtomBufferSize, MaxBufferPoolSize = m_mtomBufferPoolSize })

    }

    However, from looking at the packet, the file is being sent twice: in MIME field AND in the http field as chunked. 

    Frame 25171: 154 bytes on wire (1232 bits), 154 bytes captured (1232 bits) on interface 0
    Ethernet II, Src: IntelCor_2b:34:b7 (b4:96:91:2b:34:b7), Dst: Avigilon_1e:43:84 (00:18:85:1e:43:84)
    Internet Protocol Version 4, Src: 169.254.1.1, Dst: 169.254.180.241
    Transmission Control Protocol, Src Port: 62455, Dst Port: 80, Seq: 30312540, Ack: 1, Len: 100
    [20768 Reassembled TCP Segments (30312639 bytes): #804(415), #806(1460), #807(98), #808(190), #812(1460), #813(1460), #814(1460), #815(1460), #816(1460), #817(1460), #818(1460), #819(1460), #820(1460), #821(1460), #822(1460), #824(1460), #8]
    Hypertext Transfer Protocol
        POST /onvif/device_service HTTP/1.1\r\n
            [Expert Info (Chat/Sequence): POST /onvif/device_service HTTP/1.1\r\n]
                [POST /onvif/device_service HTTP/1.1\r\n]
                [Severity level: Chat]
                [Group: Sequence]
            Request Method: POST
            Request URI: /onvif/device_service
            Request Version: HTTP/1.1
        MIME-Version: 1.0\r\n
         [truncated]Content-Type: multipart/related; type="application/xop+xml";start="<http://tempuri.org/0>";boundary="uuid:91378036-99ed-4f90-b36b-d2a9fd91ad04+id=1";start-info="application/soap+xml"; action="http://www.onvif.org/ver10/device/
        Host: 169.254.180.241\r\n
        Transfer-Encoding: chunked\r\n
        Accept-Encoding: gzip, deflate\r\n
        Connection: Close\r\n
        \r\n
        [Full request URI: http://169.254.180.241/onvif/device_service]
        [HTTP request 1/1]
        HTTP chunked response
            Data chunk (1523 octets)
                Chunk size: 1523 octets
                Data (1523 bytes)
                    Data: 0d0a2d2d757569643a39313337383033362d393965642d34...
                    [Length: 1523]
                Chunk boundary: 0d0a
            Data chunk (22 octets)
                Chunk size: 22 octets
                Data (22 bytes)
                    Data: 3c2f733a426f64793e3c2f733a456e76656c6f70653e
                    [Length: 22]
                Chunk boundary: 0d0a
            Data chunk (184 octets)
                Chunk size: 184 octets
                Data (184 bytes)
                    Data: 0d0a2d2d757569643a39313337383033362d393965642d34...
                    [Length: 184]
                Chunk boundary: 0d0a
            Data chunk (30310400 octets)
                Chunk size: 30310400 octets
                Data (30310400 bytes)
                    Data: 4669656c645061636b6167652e696e666f2e637263000000...
                    [Length: 30310400]
                Chunk boundary: 0d0a
            Data chunk (54 octets)
                Chunk size: 54 octets
                Data (54 bytes)
                    Data: 0d0a2d2d757569643a39313337383033362d393965642d34...
                    [Length: 54]
                Chunk boundary: 0d0a
            End of chunked encoding
                Chunk size: 0 octets
            \r\n
        File Data: 30312183 bytes
    MIME Multipart Media Encapsulation, Type: multipart/related, Boundary: "uuid:91378036-99ed-4f90-b36b-d2a9fd91ad04+id=1"
        [Type: multipart/related]
        Preamble: 0d0a
        First boundary: --uuid:91378036-99ed-4f90-b36b-d2a9fd91ad04+id=1\r\n
        Encapsulated multipart part:  (application/xop+xml)
            Content-ID: <http://tempuri.org/0>\r\n
            Content-Transfer-Encoding: 8bit\r\n
            Content-Type: application/xop+xml;charset=utf-8;type="application/soap+xml"\r\n\r\n
            Media Type
                Media type: application/xop+xml; charset=utf-8;type="application/soap+xml" (1345 bytes)
        Boundary: \r\n--uuid:91378036-99ed-4f90-b36b-d2a9fd91ad04+id=1\r\n
        Encapsulated multipart part:  (application/octet-stream)
            Content-ID: <http://tempuri.org/1/636721955740614443>\r\n
            Content-Transfer-Encoding: binary\r\n
            Content-Type: application/octet-stream\r\n\r\n
            Data (30310400 bytes)
                Data: 4669656c645061636b6167652e696e666f2e637263000000...
                [Length: 30310400]
        Last boundary: \r\n--uuid:91378036-99ed-4f90-b36b-d2a9fd91ad04+id=1--\r\n

    I understand that because it's streamed, it doesn't know the size of the file and the transfer is chunked. But since it's included in the MTOM, why is it included in the http part as well??

    Large files are being sent to multiple devices at the same time so the transfer mode needs to be 'Streamed' to avoid running out of memory. Am I doing something wrong when creating the custom binding? How can I send the file via MTOM with streamed as transfer mode without it being included in http in chunks?? I'm not a xml/http expert so any help would be great. Thanks!




    • Edited by Alexb_23 Monday, October 15, 2018 8:23 PM Improve tittle
    Monday, October 15, 2018 7:42 PM

Answers

  • Hi Abraham, 

    Thanks for the insight. My question was leaning towards the http chunks being present. When the http binding's Transfer Mode is set to 'Buffered', the MIME was still present but the http chunks weren't.

    My worry was that somehow the file was being sent twice causing my devices to crash. And that I should somehow specify the Content-Length to avoid the http chunks. But you're right it is not sent twice. The problem was somewhere else.

    Issue:

    1. Transfer Mode set to Buffered. The application would run out of memory when connecting to several devices. 
    2. Transfer Mode set to StreamedRequest (sending a byte array). The device would crash running out of memory

    Solution:

    • Transfer Mode set to 'StreamedRequest' AND send the file as a Stream object.

    Thank you for your help!

    Alex 

    • Marked as answer by Alexb_23 Monday, October 22, 2018 8:31 PM
    Monday, October 22, 2018 8:31 PM

All replies

  • Hi Alexb_23,
    MTOM is a Full-Name Message transmission optimization mechanism, the model aimed to the interaction of large amounts of data, a solution is proposed for the cost of base64 encoding, which has additional 33% overhead. SOAP still use xml for message delivery when the amount of data is small.
    MTOM is an improved method based on SOAP message transmission, for the transfer of large amounts of data. Base64 encoding is not done, but directly wrapped in the MIME part of the SOAP message in the form of the binary raw data of the attachment for transmission. A soap message refers to binary content by pointing to the MIME part it sends.

    https://i.stack.imgur.com/AoFRk.png

    In short, it's not sent twice. It's just an optimization of the transmission.
    https://developers.de/blogs/damir_dobric/archive/2008/02/02/wcf-mtom-binary-data-transmission.aspx

    Feel free to let me know if you have any questions.

    Best Regards

    Abraham

    Thursday, October 18, 2018 9:18 AM
    Moderator
  • Hi Abraham, 

    Thanks for the insight. My question was leaning towards the http chunks being present. When the http binding's Transfer Mode is set to 'Buffered', the MIME was still present but the http chunks weren't.

    My worry was that somehow the file was being sent twice causing my devices to crash. And that I should somehow specify the Content-Length to avoid the http chunks. But you're right it is not sent twice. The problem was somewhere else.

    Issue:

    1. Transfer Mode set to Buffered. The application would run out of memory when connecting to several devices. 
    2. Transfer Mode set to StreamedRequest (sending a byte array). The device would crash running out of memory

    Solution:

    • Transfer Mode set to 'StreamedRequest' AND send the file as a Stream object.

    Thank you for your help!

    Alex 

    • Marked as answer by Alexb_23 Monday, October 22, 2018 8:31 PM
    Monday, October 22, 2018 8:31 PM