locked
How to use HTTP2 to download compressed content RRS feed

  • Question

  • User-760472622 posted

    Hello,

    As I know .NET Core 3 has support for HTTP2, so I want to do some test. The following is my code to download a simple web page. For quite a number of web sites, which support HTTP 2 protocol, the performance is better than HTTP 1.1.

    static async Task Page2_Http_Get(string url1)
    {
        using var client = new HttpClient(new WinHttpHandler(), true);
        var request = new HttpRequestMessage(HttpMethod.Get, url1)
        {
        Version = new Version(2, 0)
        };
        var response = await client.SendAsync(request);
        string html_content = await response.Content.ReadAsStringAsync();
    }

    However, using the old http client, I can handle the compressed content using the following code.

    static async Task Page1_Http_Get(string url1)
    {
        ServicePointManager.Expect100Continue = false;
        ServicePointManager.SecurityProtocol =
    SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
        ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
        HttpClientHandler clientHandler = new HttpClientHandler()
        {
        AllowAutoRedirect = true,
        AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip,
        };
        using HttpClient client1 = new HttpClient(clientHandler);
        HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url1);
        var response = await client1.SendAsync(request);
        string html_content = await response.Content.ReadAsStringAsync();
    }

    But for HTTP 2, I can’t find any code sample to add handling of compressed contents. Besides, the old http client can also handle redirect, how I can do this with HTTP 2 client?

    Please advice.

    PS: I am using .NET Core 3.0 on VS 2019 Version 16.3.1 on Windows 10 (1903).

    Thanks,

    Wednesday, October 2, 2019 6:58 AM

All replies

  • User-474980206 posted

    http2 has compression built in. it does not use deflate or gzip.  

    Wednesday, October 2, 2019 4:56 PM
  • User-760472622 posted

    Hello:

    I don't think so, as from my test.  For one specific web site, if I use Http1 client to download a HTML page without using compression option, I see only the garbage code (binary). But if I turn on compression options, I can see HTML contents in ASCII code.

    But for HTTP 2 client, I can only see the garbage code (binary), but the contents seem correct, just as the HTTP1 client returned contents without compression option.

    Therefore, I believe, it should be something I have to do with the compression, maybe not via compression option, but through IO compression method.

    Wednesday, October 2, 2019 6:01 PM
  • User753101303 posted

    Hi,

    The last time I checked my understanding was that "it just works" unless you want to benefit from advanced features such as pushing data to the client even before it is required (but I never tried, I remember in 4.x they added a method that apparently allows that).

    Edit: I believe I saw that at https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis which gives some details all along with another article I don't find right now...

    Wednesday, October 2, 2019 6:08 PM
  • User-474980206 posted

    http2 compresses and can sends blocks out of order, its part of the protocol. unless you used a network sniffer, you will not see the compression.

    Wednesday, October 2, 2019 6:17 PM
  • User-760472622 posted

    Hello:

    I give you a real world example, see my code:

            static async Task Http1_Get_No_Compression(string url1)
            {
                ServicePointManager.Expect100Continue = false;
                ServicePointManager.SecurityProtocol =
                    SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
                ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
                HttpClientHandler clientHandler = new HttpClientHandler()
                {
                    AllowAutoRedirect = true,
                };
                using HttpClient client1 = new HttpClient(clientHandler);
                HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url1);
                var response = await client1.SendAsync(request);
                string html_content = await response.Content.ReadAsStringAsync();
            }
    
            static async Task Http1_Get_With_Compression(string url1)
            {
                ServicePointManager.Expect100Continue = false;
                ServicePointManager.SecurityProtocol =
                    SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
                ServicePointManager.ServerCertificateValidationCallback += (sender, cert, chain, sslPolicyErrors) => true;
                HttpClientHandler clientHandler = new HttpClientHandler()
                {
                    AllowAutoRedirect = true,
                    AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip,
                };
                using HttpClient client1 = new HttpClient(clientHandler);
                HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url1);
                var response = await client1.SendAsync(request);
                string html_content = await response.Content.ReadAsStringAsync();
            }
    
            static async Task Http2_Get_No_Compression(string url1)
            {
                using var client = new HttpClient(new WinHttpHandler(), true);
                var request = new HttpRequestMessage(HttpMethod.Get, url1)
                {
                    Version = new Version(2, 0)
                };
                var response = await client.SendAsync(request);
                string html_content = await response.Content.ReadAsStringAsync();
            }
    
            static async Task Main()
            {
                await Http1_Get_No_Compression(@"http://31.172.83.181:8080/free/markets/{""Cat"":1, ""OnlyActive"": true}");
                await Http1_Get_With_Compression(@"http://31.172.83.181:8080/free/markets/{""Cat"":1, ""OnlyActive"": true}");
                await Http2_Get_No_Compression(@"http://31.172.83.181:8080/free/markets/{""Cat"":1, ""OnlyActive"": true}");
            }
    

    When I run my above code, for HTTP1 and HTTP2 without compression, the Length of html_content is the same: 8903, with (de-)compression for Function: HTTP1_Get_With_Compression(url1), the Length of html_content is nearly 10 times bigger than without compression.

    You can take a look!

    Wednesday, October 2, 2019 6:48 PM
  • User475983607 posted

    As I understand HTTP/2 the results are expected.  The HTTP/2 binary frames are handled at lower level.  You need to use utility like Wireshark or Chrome's dev tools to see the protocol. 

    https://developers.google.com/web/fundamentals/performance/http2/

    Wednesday, October 2, 2019 7:15 PM
  • User-474980206 posted

    you are looking at the content length after its been decompressed. the compression is at the protocol level. 

    Wednesday, October 2, 2019 7:38 PM
  • User-760472622 posted

    Hello:

    I think you misunderstand my meaning.  I show you the code, what I want is like this: using the HTTP1_Get_With_Compression(url1), I can get the real meaningful HTML content in ASCII code, so I can use Json to parse the result.  But with HTTP1_Get_No_Compression(url1) or HTTP2_Get_No_Compression(url1), what I get is the binary content, not in ASCII code.  Therefore, I can't use Json to parse the result.

    What I want to do is: if I can write another function: HTTP2_Get_With_Compression(url1) to get the meaningful HTML content in ASCII code, therefore, I can use Json to parse the text result.  But on the second thought, I think .NET core 3 has the built-in Josn support: System.Text.Json, if I can use System.Text.Json to parse the binary result returned by HTTP2 client GET request, it is also OK.

    Hope you understand my meaning, I want to be able to parse the HTTP2 client returned result be it in binary or not, compressed or not.

    Wednesday, October 2, 2019 8:06 PM
  • User753101303 posted

    Hi,

    You still have to enable "gzip" and "deflate" as you are doing for http/1. For a start the only change should be to opt for this protocol version and your code is basically unchanged (if I remember it compresses the header but it is part of the standard and done for you behind the scene a bit like encryption/decryption with https).

    Thursday, October 3, 2019 8:13 AM
  • User-760472622 posted

    Hello:

    If you can show me your code on my real world example, for example, if you write a function HTTP2_Get_With_Compression(url1) using the url in my previous example, I can test to see if it works.

    Thanks,

    Thursday, October 3, 2019 12:07 PM
  • User-474980206 posted

    you seem to have absolutely no understanding of the http protocol and its use of compression. 

    in http 1, payload compression is optional feature added to the protocol. if the payload is compressed , the encoding header is set to let the client know the compression type. The client first sends an accept header with the compressions algorithms supported. if the server supported one of the requested algorithms, it responds with a compression payload. 

      https://en.wikipedia.org/wiki/HTTP_compression

    with http1, the web server needs to implement payload compression support. as compression is not part of the protocol, but an agreement between services using the protocol, the client library also needs to know about compression. older libraries have no support (you need to add the code yourself), new  libraries have it builtin. in your sample code for http 1, you are setting the accept headers, but there is no guarantee the server supports them or responds with compressed content.

    in http 2 payload compression is no longer required. the protocol has compression built in. so any library that supports http 2, will support http 2 compression (which is wire compression, not payload). your web server must still support http 1, because many clients do not yet support http 2, and will make http 1 requests.  as you are using client code, there is no guarantee that the server supports http 2, and it will fall back to http 1.

    Thursday, October 3, 2019 3:30 PM
  • User-760472622 posted

    Hello:

    I don't really care about the theory, but I care about how to use HTTP 2 to get the data I want.  If you can have to code to download the data and parse it, then it is OK.

    Actually, I figured out how to use HTTP 2 to download and decompress the data, the server did send the data using HTTP 2 as I can see the time comparing to use HTTP 1.1 is different after many times of testing, but I want to see if you have a better solution.

    Thursday, October 3, 2019 5:53 PM
  • User475983607 posted

    zydjohn2019

    Actually, I figured out how to use HTTP 2 to download and decompress the data, the server did send the data using HTTP 2 as I can see the time comparing to use HTTP 1.1 is different after many times of testing, but I want to see if you have a better solution.

    You thought the response was HTTP/2 but in reality it was HTTP/1.1?  Use the Version property if you need to check what protocol is in use.

    The sample code below is based on the ASP.NET reference docs and the ASP.NET Core 3.0 Web API template.  It returns version 2.

    Client

            static void Main(string[] args)
            {
                RunAsync1().GetAwaiter().GetResult();
            }
    
            static async Task RunAsync()
            {
                var client = new HttpClient() { BaseAddress = new Uri("https://localhost:5001") };
    
                using (var request = new HttpRequestMessage(HttpMethod.Get, "weatherforecast") { Version = new Version(2, 0) })
                using (var response = await client.SendAsync(request))
                {
                    Console.WriteLine(response.Version);
                    Console.WriteLine("------------------------------");
                    string content = await response.Content.ReadAsStringAsync();
                    Console.WriteLine(content);
                }
            }

    If I change the URL to a 1.1 web app the version is shows 1.1.  Use this to figure out if you need to deflate the response.

    Thursday, October 3, 2019 6:17 PM
  • User-474980206 posted

    HttpClient handles the deflate and gzip automatically. Depending on version it may or may not default to sending the accept headers which are required for deflate or gzip transmissions.

    you can check the response encoding header to know if gzip or deflate was used.

    Thursday, October 3, 2019 7:48 PM