locked
WCF REST (WebHttpBinding compatible ) client gzip compression - decompress web responses from server RRS feed

  • Question

  • Hi,

    I've been searching for a little while for a way to add a new behaviour to my client WebHttpBinding based WCF REST web service interface.

    I have a Java EE server with gzip (normal compression strength) capability for responses. Normally this is "content negotiated" but for testing I can turn it on permanently for all responses (which happent to be XML documents).

    I've tried several suggested methods of adding decompression to the client side WCF and I hit one of several problems. A few of the solutions I've tried seem to be compatible with SOAP WS.* servces, so when mixed with my web REST service definition cause a run time error when the service channel is being created.

    The last solution I have based on adding the WcfExtensions.ServiceModel.Channels.CompressionBindingElement

    When I try with this, if I adjust my Java EE server not to compress responses the service calls work from the client. When I enable compression of responses I get an error of the form: "EndpointNotFoundException"

    I'm not massively familiar with WCF and the documentation on the REST support seems a little thin on the ground when it comes to customisations.

    If someone can even confirm this can be done and point me at a sample of code that should work, I can probably do the rest (no pun intended) - at the moment I'm not sure what are impossible limitiations in the WCF framework and what are mistakes or incorrect set up in my test code. Ideally I'd like a solution that's compatible with standard gzip web server compression, as is normally support by web browsers too etc.

    thanks for any help in advance! :)
    Tuesday, April 21, 2009 11:38 AM

All replies

  • I don't see any reason why a custom message encoder shouldn't work in this situation, have you tried the sample GZIP encoder? http://msdn.microsoft.com/en-us/library/ms751458.aspx

    It also matters how you construct the custom binding (you need to replace the TextMessageEncodingBindingElement in WebHttpBinding with the new encoder, see http://social.msdn.microsoft.com/forums/en-US/wcf/thread/beb10532-33f8-4cff-ba2d-99641775add1/ for an example of this, although you won't be wrapping the encoder in this case)

    • Marked as answer by edhickey Monday, April 27, 2009 9:13 PM
    • Unmarked as answer by LouisJB Tuesday, May 5, 2009 11:30 AM
    Thursday, April 23, 2009 12:56 AM
  • hi,

    thanks for the help!

    I may be being a bit slow, but I'm struggling to get this to work.

    I have a WebHttpBinding, I have the example (custom message encoder) referenced in my project.

    What I can't work out is the right way to create a custom binding that incoporates the compression binding. Programatically I have some test code a bit like this (which might not be quite right, but I've tried a few variations of this theme too):

    WebHttpBinding webHttpBinding = new WebHttpBinding();

    GZipMessageEncodingBindingElement compBindingElement = new GZipMessageEncodingBindingElement();
    BindingElementCollection bec = new BindingElementCollection();
    bec.Add(compBindingElement);
    bec.AddRange(webHttpBinding.CreateBindingElements().ToArray());
                       
    CustomBinding cb = new CustomBinding(bec);

    // create a channel factory
    ChannelFactory<IService> cf = new ChannelFactory<IService>(cb, hostPath);

    // add webhttpbehahvior to channelfactory endpoint
    cf.Endpoint.Behaviors.Add(webHttpBehavior);

    channel = cf.CreateChannel();

    and it throws an exception on the last line of the form:

    ex = {"The endpoint at 'http://host:8080/NETbuilder/' does not have a Binding with the None MessageVersion.  'System.ServiceModel.Description.WebHttpBehavior' is only intended for use with WebHttpBinding or similar bindings."}

    Which is the problem I'm generally experiencing, every example I've tried seems to end with a clash between web (rest) bindings and soap ones, which can not be mixed or result in a runtime error when trying to instantiate the channel.

    Maybe I'm just doing something silly here, is there a solution to make this work?

    Any help greatly appreciated, I'm really struggling with this and can't seem to find the resources I need to help me solve it.
    • Edited by LouisJB Thursday, April 30, 2009 7:46 PM
    Thursday, April 30, 2009 7:39 PM
  • Anyone able to shed any light on this problem in general?

    Tuesday, May 5, 2009 11:31 AM
  • I'm having the same problem.  Does anyone have a solution to this?
    Wednesday, September 9, 2009 8:34 PM
  • The problem here is that the GZupMessageEncodingBindingElement must be initialized with the existing encoder from WebHttpBinding. The code below shows it. Notice that we also need a WebContentTypeMapper, to map the gzip content-type to the actual type that is used by the inner encoder.

        public class Post_8c3eafae_b6a1_441f_85ef_90721d941a1a
        {
            [DataContract]
            public class MyDC
            {
                [DataMember]
                public string str;
                [DataMember]
                public int[] intArray;
    
                public static MyDC CreateLargeInstance()
                {
                    Random rndGen = new Random(1);
                    StringBuilder sb = new StringBuilder();
                    int size = rndGen.Next(100, 300);
                    MyDC result = new MyDC();
                    for (int i = 0; i < size; i++)
                    {
                        sb.Append((char)rndGen.Next('a', 'z'));
                    }
                    result.str = sb.ToString();
                    size = rndGen.Next(300, 500);
                    result.intArray = new int[size];
                    for (int i = 0; i < size; i++)
                    {
                        result.intArray[i] = rndGen.Next();
                    }
                    return result;
                }
            }
            [ServiceContract]
            public interface ITest
            {
                [OperationContract]
                MyDC Echo(MyDC input);
            }
            public class Service : ITest
            {
                public MyDC Echo(MyDC input)
                {
                    return input;
                }
            }
    
            #region Gzip Encoder Sample
            //This class is used to create the custom encoder (GZipMessageEncoder)
            internal class GZipMessageEncoderFactory : MessageEncoderFactory
            {
                MessageEncoder encoder;
    
                //The GZip encoder wraps an inner encoder
                //We require a factory to be passed in that will create this inner encoder
                public GZipMessageEncoderFactory(MessageEncoderFactory messageEncoderFactory)
                {
                    if (messageEncoderFactory == null)
                        throw new ArgumentNullException("messageEncoderFactory", "A valid message encoder factory must be passed to the GZipEncoder");
                    encoder = new GZipMessageEncoder(messageEncoderFactory.Encoder);
    
                }
    
                //The service framework uses this property to obtain an encoder from this encoder factory
                public override MessageEncoder Encoder
                {
                    get { return encoder; }
                }
    
                public override MessageVersion MessageVersion
                {
                    get { return encoder.MessageVersion; }
                }
    
                //This is the actual GZip encoder
                class GZipMessageEncoder : MessageEncoder
                {
                    static string GZipContentType = "application/x-gzip";
    
                    //This implementation wraps an inner encoder that actually converts a WCF Message
                    //into textual XML, binary XML or some other format. This implementation then compresses the results.
                    //The opposite happens when reading messages.
                    //This member stores this inner encoder.
                    MessageEncoder innerEncoder;
    
                    //We require an inner encoder to be supplied (see comment above)
                    internal GZipMessageEncoder(MessageEncoder messageEncoder)
                        : base()
                    {
                        if (messageEncoder == null)
                            throw new ArgumentNullException("messageEncoder", "A valid message encoder must be passed to the GZipEncoder");
                        innerEncoder = messageEncoder;
                    }
    
                    //public override string CharSet
                    //{
                    //    get { return ""; }
                    //}
    
                    public override string ContentType
                    {
                        get { return GZipContentType; }
                    }
    
                    public override string MediaType
                    {
                        get { return GZipContentType; }
                    }
    
                    //SOAP version to use - we delegate to the inner encoder for this
                    public override MessageVersion MessageVersion
                    {
                        get { return innerEncoder.MessageVersion; }
                    }
    
                    //Helper method to compress an array of bytes
                    static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager, int messageOffset)
                    {
                        MemoryStream memoryStream = new MemoryStream();
                        memoryStream.Write(buffer.Array, 0, messageOffset);
    
                        using (GZipStream gzStream = new GZipStream(memoryStream, CompressionMode.Compress, true))
                        {
                            gzStream.Write(buffer.Array, messageOffset, buffer.Count);
                        }
    
    
                        byte[] compressedBytes = memoryStream.ToArray();
                        byte[] bufferedBytes = bufferManager.TakeBuffer(compressedBytes.Length);
    
                        Array.Copy(compressedBytes, 0, bufferedBytes, 0, compressedBytes.Length);
    
                        bufferManager.ReturnBuffer(buffer.Array);
                        ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset, bufferedBytes.Length - messageOffset);
    
                        return byteArray;
                    }
    
                    //Helper method to decompress an array of bytes
                    static ArraySegment<byte> DecompressBuffer(ArraySegment<byte> buffer, BufferManager bufferManager)
                    {
                        MemoryStream memoryStream = new MemoryStream(buffer.Array, buffer.Offset, buffer.Count - buffer.Offset);
                        MemoryStream decompressedStream = new MemoryStream();
                        int totalRead = 0;
                        int blockSize = 1024;
                        byte[] tempBuffer = bufferManager.TakeBuffer(blockSize);
                        using (GZipStream gzStream = new GZipStream(memoryStream, CompressionMode.Decompress))
                        {
                            while (true)
                            {
                                int bytesRead = gzStream.Read(tempBuffer, 0, blockSize);
                                if (bytesRead == 0)
                                    break;
                                decompressedStream.Write(tempBuffer, 0, bytesRead);
                                totalRead += bytesRead;
                            }
                        }
                        bufferManager.ReturnBuffer(tempBuffer);
    
                        byte[] decompressedBytes = decompressedStream.ToArray();
                        byte[] bufferManagerBuffer = bufferManager.TakeBuffer(decompressedBytes.Length + buffer.Offset);
                        Array.Copy(buffer.Array, 0, bufferManagerBuffer, 0, buffer.Offset);
                        Array.Copy(decompressedBytes, 0, bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);
    
                        ArraySegment<byte> byteArray = new ArraySegment<byte>(bufferManagerBuffer, buffer.Offset, decompressedBytes.Length);
                        bufferManager.ReturnBuffer(buffer.Array);
    
                        return byteArray;
                    }
    
    
                    //One of the two main entry points into the encoder. Called by WCF to decode a buffered byte array into a Message.
                    public override Message ReadMessage(ArraySegment<byte> buffer, BufferManager bufferManager, string contentType)
                    {
                        //Decompress the buffer
                        ArraySegment<byte> decompressedBuffer = DecompressBuffer(buffer, bufferManager);
                        //Use the inner encoder to decode the decompressed buffer
                        Message returnMessage = innerEncoder.ReadMessage(decompressedBuffer, bufferManager);
                        returnMessage.Properties.Encoder = this;
                        return returnMessage;
                    }
    
                    //One of the two main entry points into the encoder. Called by WCF to encode a Message into a buffered byte array.
                    public override ArraySegment<byte> WriteMessage(Message message, int maxMessageSize, BufferManager bufferManager, int messageOffset)
                    {
                        //Use the inner encoder to encode a Message into a buffered byte array
                        ArraySegment<byte> buffer = innerEncoder.WriteMessage(message, maxMessageSize, bufferManager, messageOffset);
                        //Compress the resulting byte array
                        return CompressBuffer(buffer, bufferManager, messageOffset);
                    }
    
                    public override Message ReadMessage(System.IO.Stream stream, int maxSizeOfHeaders, string contentType)
                    {
                        GZipStream gzStream = new GZipStream(stream, CompressionMode.Decompress, true);
                        return innerEncoder.ReadMessage(gzStream, maxSizeOfHeaders);
                    }
    
                    public override void WriteMessage(Message message, System.IO.Stream stream)
                    {
                        using (GZipStream gzStream = new GZipStream(stream, CompressionMode.Compress, true))
                        {
                            innerEncoder.WriteMessage(message, gzStream);
                        }
    
                        // innerEncoder.WriteMessage(message, gzStream) depends on that it can flush data by flushing 
                        // the stream passed in, but the implementation of GZipStream.Flush will not flush underlying
                        // stream, so we need to flush here.
                        stream.Flush();
                    }
                }
            }
            // This is constants for GZip message encoding policy.
            static class GZipMessageEncodingPolicyConstants
            {
                public const string GZipEncodingName = "GZipEncoding";
                public const string GZipEncodingNamespace = "http://schemas.microsoft.com/ws/06/2004/mspolicy/netgzip1";
                public const string GZipEncodingPrefix = "gzip";
            }
    
            //This is the binding element that, when plugged into a custom binding, will enable the GZip encoder
            public sealed class GZipMessageEncodingBindingElement
                                : MessageEncodingBindingElement //BindingElement
                                , IPolicyExportExtension
            {
    
                //We will use an inner binding element to store information required for the inner encoder
                MessageEncodingBindingElement innerBindingElement;
    
                //By default, use the default text encoder as the inner encoder
                public GZipMessageEncodingBindingElement()
                    : this(new TextMessageEncodingBindingElement()) { }
    
                public GZipMessageEncodingBindingElement(MessageEncodingBindingElement messageEncoderBindingElement)
                {
                    this.innerBindingElement = messageEncoderBindingElement;
                }
    
                public MessageEncodingBindingElement InnerMessageEncodingBindingElement
                {
                    get { return innerBindingElement; }
                    set { innerBindingElement = value; }
                }
    
                //Main entry point into the encoder binding element. Called by WCF to get the factory that will create the
                //message encoder
                public override MessageEncoderFactory CreateMessageEncoderFactory()
                {
                    return new GZipMessageEncoderFactory(innerBindingElement.CreateMessageEncoderFactory());
                }
    
                public override MessageVersion MessageVersion
                {
                    get { return innerBindingElement.MessageVersion; }
                    set { innerBindingElement.MessageVersion = value; }
                }
    
                public override BindingElement Clone()
                {
                    return new GZipMessageEncodingBindingElement(this.innerBindingElement);
                }
    
                public override T GetProperty<T>(BindingContext context)
                {
                    if (typeof(T) == typeof(XmlDictionaryReaderQuotas))
                    {
                        return innerBindingElement.GetProperty<T>(context);
                    }
                    else
                    {
                        return base.GetProperty<T>(context);
                    }
                }
    
                public override IChannelFactory<TChannel> BuildChannelFactory<TChannel>(BindingContext context)
                {
                    if (context == null)
                        throw new ArgumentNullException("context");
    
                    context.BindingParameters.Add(this);
                    return context.BuildInnerChannelFactory<TChannel>();
                }
    
                public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
                {
                    if (context == null)
                        throw new ArgumentNullException("context");
    
                    context.BindingParameters.Add(this);
                    return context.BuildInnerChannelListener<TChannel>();
                }
    
                public override bool CanBuildChannelListener<TChannel>(BindingContext context)
                {
                    if (context == null)
                        throw new ArgumentNullException("context");
    
                    context.BindingParameters.Add(this);
                    return context.CanBuildInnerChannelListener<TChannel>();
                }
    
                void IPolicyExportExtension.ExportPolicy(MetadataExporter exporter, PolicyConversionContext policyContext)
                {
                    if (policyContext == null)
                    {
                        throw new ArgumentNullException("policyContext");
                    }
                    XmlDocument document = new XmlDocument();
                    policyContext.GetBindingAssertions().Add(document.CreateElement(
                        GZipMessageEncodingPolicyConstants.GZipEncodingPrefix,
                        GZipMessageEncodingPolicyConstants.GZipEncodingName,
                        GZipMessageEncodingPolicyConstants.GZipEncodingNamespace));
                }
            }
            #endregion
    
            class MyMapper : WebContentTypeMapper
            {
                public override WebContentFormat GetMessageFormatForContentType(string contentType)
                {
                    return WebContentFormat.Xml;
                }
            }
            static Binding GetBinding()
            {
                CustomBinding custom = new CustomBinding(new WebHttpBinding());
                for (int i = 0; i < custom.Elements.Count; i++)
                {
                    if (custom.Elements[i] is WebMessageEncodingBindingElement)
                    {
                        WebMessageEncodingBindingElement webBE = (WebMessageEncodingBindingElement)custom.Elements[i];
                        webBE.ContentTypeMapper = new MyMapper();
                        custom.Elements[i] = new GZipMessageEncodingBindingElement(webBE);
                    }
                    else if (custom.Elements[i] is TransportBindingElement)
                    {
                        ((TransportBindingElement)custom.Elements[i]).MaxReceivedMessageSize = int.MaxValue;
                    }
                }
                return custom;
            }
            public static void Test()
            {
                string baseAddress = "http://" + Environment.MachineName + ":8000/Service";
                ServiceHost host = new ServiceHost(typeof(Service), new Uri(baseAddress));
                host.AddServiceEndpoint(typeof(ITest), GetBinding(), "").Behaviors.Add(new WebHttpBehavior());
                host.Description.Behaviors.Add(new ServiceMetadataBehavior { HttpGetEnabled = true });
                host.Open();
                Console.WriteLine("Host opened");
    
                ChannelFactory<ITest> factory = new ChannelFactory<ITest>(GetBinding(), new EndpointAddress(baseAddress));
                factory.Endpoint.Behaviors.Add(new WebHttpBehavior());
                ITest proxy = factory.CreateChannel();
                Console.WriteLine(proxy.Echo(MyDC.CreateLargeInstance()));
    
                ((IClientChannel)proxy).Close();
                factory.Close();
    
                Console.Write("Press ENTER to close the host");
                Console.ReadLine();
                host.Close();
            }
        }
    
    • Proposed as answer by Carlos Figueira Wednesday, September 9, 2009 11:37 PM
    Wednesday, September 9, 2009 10:47 PM
  • Awesome, thank you.

    How would we set this up via configuration in web.config?

    I guess like this:  http://msdn.microsoft.com/en-us/library/bb943479.aspx

    <customBinding>
    <binding name="JsonMapper">
    <webMessageEncoding webContentTypeMapperType=
    "MyMapper , MyMapper , Version=3.5.0.0, Culture=neutral, PublicKeyToken=null" />
    <httpTransport manualAddressing="true" />
    </binding>
    </customBinding>
    • Edited by pjepsen Thursday, September 10, 2009 12:00 AM
    Wednesday, September 9, 2009 11:52 PM
  • You can't really, since you need to first create an encoder (WebMessageEncodingBindingElement), then use it to create the outer encoder (GZipMEBE). If you're using a webhosted service, you can create a factory to be able to change those settings in config, as described in http://blogs.msdn.com/carlosfigueira/archive/2007/12/26/modifying-code-only-settings-on-webhosted-services.aspx.
    Wednesday, September 9, 2009 11:57 PM
  • Hey Carlos thanks to your help,  I've finally managed to get this partially working. 

    I modified my ServiceHostFactory as outlined in your blog, and calls to my service produced a downloadable .gz file containing an xml file.

    After further code modifications:

    Changing static string GZipContentType = "application/x-gzip";  to static string GZipContentType = "text/xml";

    And adding to service:

    HttpResponse Response = HttpContext.Current.Response;
    Response.AppendHeader("Content-Encoding", "gzip");

    Output is now xml Content-encoded gzip which displays correctly in Firefox, but not in other browsers.


    Making a request to the service in Fiddler yields the following error:

    Fiddler:  UnGZip failed

    The content could not be decompressed.

    The calculated checksum differs from the stored checksum.

    Any ideas?




    Thursday, September 10, 2009 6:26 PM
  • The only browser that will correctly parse the XML content is Firefox, other browsers seem to choke on the content. 

    From what I can tell the full XML content is not extracted by IE browsers.  Help!!!

    The XML page cannot be displayed

    Cannot view XML input using XSL style sheet. Please correct the error and then click the Refresh button, or try again later.


    The following tags were not closed:  ....

     

     

    Friday, September 11, 2009 5:18 PM
  • I believe the problem is that the compressed stream is "deflate" and not "gzip".    Upon inspection of the compressed data the second byte of the header is "0x08" which according to the spec is "deflate".    Because the Content-encoding response header says "gzip" and not "deflate" IE7 seems unable to correctly process the data whereas Firefox processes the response correctly.

    The fix I believe is:

    HttpResponse Response = HttpContext.Current.Response;
    Response.AppendHeader("Content-Encoding", "deflate");
     
    Friday, September 11, 2009 6:05 PM
  • I'm able to get it working both on IE and on Firefox by using Content-Encoding: gzip. The only difference I did was instead of using HttpResponse to add the header, I used the WebOperationContext to do the same:

    WebOperationContext.Current.OutgoingResponse.Headers[HttpResponseHeader.ContentEncoding] = "gzip";

    Friday, September 11, 2009 9:00 PM
  • Did you also change "text/xml"?

    In fiddler if I save the selected bytes to a .gz file I can correctly view the xml file inside using winrar.  However winrar does not report the uncompressed filesize, it states 0.

    Doing the same operation on a gzip stream from another site (works all browsers) yields an archive that does contain the uncompressed filesize.

    I'm wondering if the System.IO.Compression gzip implementation does not include this information in the gzip header?


    Also of note my operation contracts are:

    BodyStyle = WebMessageBodyStyle.Bare


    Friday, September 11, 2009 11:16 PM
  • Ok, after days of pulling my hair out I've discovered the problem.  As I thought gzip data had an incorrect size. 

    The CompressBuffer methods sets the incorrect compressedBytes.Length, here is a version which works:


     static ArraySegment<byte> CompressBuffer(ArraySegment<byte> buffer,  BufferManager bufferManager, int messageOffset)
           {
              ArraySegment<byte> byteArray;

              using (var memoryStream = new MemoryStream())
              {
                 var zipperStream = new GZipStream(memoryStream, CompressionMode.Compress, CompressionLevel.BestSpeed, true);

                 using (zipperStream)
                    zipperStream.Write(buffer.Array,
                       buffer.Offset, buffer.Count);

                 byte[] compressedBytes  = memoryStream.ToArray();
                 byte[] bufferedBytes = bufferManager.TakeBuffer(
                    compressedBytes.Length + messageOffset);
                    
                 Array.Copy(compressedBytes, 0, bufferedBytes, messageOffset,
                    compressedBytes.Length);

                 bufferManager.ReturnBuffer(buffer.Array);
                 byteArray = new ArraySegment<byte>(bufferedBytes, messageOffset,
                    compressedBytes.Length);
              }
              return byteArray;
           }

    • Proposed as answer by pjepsen Sunday, September 13, 2009 12:29 AM
    Sunday, September 13, 2009 12:29 AM
  • Pjepsen, would you please consider uploading your code or part of project somewhere? I have exactly the same problem.
    Tuesday, September 22, 2009 2:26 PM
  • FWIW: I was about to post about the same thing. CompressBuffer() will pad the output with zeros, thus wasting a lot of space.

    In one test I did, fixing this issue reduced the payload from 780KB to 606KB.

    PS: Would it not be better to call bufferManager.ReturnBuffer(buffer.Array); sooner to reduce the likelyhood of allocating a new buffer?

    PPS: I believe it is a good idea to deduct the messageOffset: return new ArraySegment<byte>(bufferedBytes, messageOffset, (int)memoryStream.Length - messageOffset); messageOffset is sometimes set, e.g. when using the net.tcp binding (hardcoded to 6 for some reason).

    PPPS: I also dropped the compressedBytes intermediary... I see no reason for that extra variable.
    Monday, October 5, 2009 2:37 PM
  • I tried your fixed CompressBuffer() but am getting an error about application/json not being recognized.  Is this technique only working with XML-serialized responses perhaps?

    Wednesday, July 14, 2010 8:31 PM