none
system.net.http.httpclient does not respect dns update in a timely manner

    Question

  • We have a single instance of httpclient that handles all outgoing requests from our service to another service.

    When DNS is updated (can repro using hosts file change) for the remote endpoint, httpclient connections continue to use the old connection, in some cases for several hours.

    I understand that the underlying connections have keep-alive on by default, and that they are pooled (2?) connections.  We're handling several thousand requests per second, so for performance reasons we don't want to create a new instance for each request, or specify ConnectionClose = true to force each connection to close once complete.

    Is there a nice way to periodically have an httpclient instance check if the dns resolution for the persistent connections are still valid, and re-connect if they are no longer valid?


    Taj Thoresen


    • Edited by Taj MSFT Thursday, April 2, 2015 3:19 PM
    Thursday, April 2, 2015 3:17 PM

Answers

  • Hi Taj MSFT,

    It seems like this is a default behavior by HttpClient. You could see there is only once to call DNS::TryInternalResolve method to resolve the request uri when HttpClient fire the first request from diagnostics trace log.

    Enable diagnostics trace by following configuration in application config file:

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <system.net>
        <settings>
          <servicePointManager dnsRefreshTimeout="0"/>
        </settings>
      </system.net>
    
      <system.diagnostics>
        <trace autoflush="true" />
        <sources>
          <source name="System.Net.Http">
            <listeners>
              <add name="System.Http.Trace"/>
            </listeners>
          </source>
          <source name="System.Net.Sockets">
            <listeners>
              <add name="System.Sockets.Trace"/>
            </listeners>
          </source>
        </sources>
        <sharedListeners>
          <add
            name="System.Http.Trace"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="System.Net.Http.trace.log"/>
    
          <add
            name="System.Sockets.Trace"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="System.Net.Sockets.trace.log"/>
        </sharedListeners>
        <switches>
          <add name="System.Net.Http" value="Verbose" />
          <add name="System.Net.Sockets" value="Verbose"/>
        </switches>
      </system.diagnostics>
      <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
      </startup>
    </configuration>

    You may get the System.Net.Sockets.trace log like this, but there would be only one record for DNS::TryInternalResolve:

    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::Socket(AddressFamily#2)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::Socket() 
    System.Net.Sockets Verbose: 0 : [19080] Socket#37489757::Socket(AddressFamily#23)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#37489757::Socket() 
    System.Net.Sockets Verbose: 0 : [19080] DNS::TryInternalResolve(www.bing.com)
    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::BeginConnectEx()
    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::InternalBind(0.0.0.0:0#0)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::InternalBind() 
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::BeginConnectEx() 	-> ConnectOverlappedAsyncResult#64828693
    System.Net.Sockets Verbose: 0 : [13732] Socket#33711845::EndConnect(ConnectOverlappedAsyncResult#64828693)
    System.Net.Sockets Information: 0 : [13732] Socket#33711845 - Created connection from 10.172.18.234:4010 to 204.79.197.200:80.
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#33711845::EndConnect() 
    System.Net.Sockets Verbose: 0 : [13732] Socket#37489757::Close()
    System.Net.Sockets Verbose: 0 : [13732] Socket#37489757::Dispose()
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#37489757::Close() 
    System.Net.Sockets Verbose: 0 : [13732] Socket#33711845::UnsafeBeginSend()
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#33711845::UnsafeBeginSend() 	-> OverlappedAsyncResult#11429296
    System.Net.Sockets Verbose: 0 : [20440] Data from Socket#33711845::PostCompletion
    System.Net.Sockets Verbose: 0 : [20440] 00000000 : 47 45 54 20 2F 20 48 54-54 50 2F 31 2E 31 0D 0A : GET / HTTP/1.1..
    System.Net.Sockets Verbose: 0 : [20440] 00000010 : 48 6F 73 74 3A 20 77 77-77 2E 62 69 6E 67 2E 63 : Host: www.bing.c
    System.Net.Sockets Verbose: 0 : [20440] 00000020 : 6F 6D 0D 0A 43 6F 6E 6E-65 63 74 69 6F 6E 3A 20 : om..Connection: 
    System.Net.Sockets Verbose: 0 : [20440] 00000030 : 4B 65 65 70 2D 41 6C 69-76 65 0D 0A 0D 0A       : Keep-Alive....
    System.Net.Sockets Verbose: 0 : [20440] Socket#33711845::EndSend(OverlappedAsyncResult#11429296)
    System.Net.Sockets Verbose: 0 : [20440] Exiting Socket#33711845::EndSend() 	-> Int32#62
    System.Net.Sockets Verbose: 0 : [20440] Socket#33711845::UnsafeBeginReceive()
    System.Net.Sockets Verbose: 0 : [20440] Exiting Socket#33711845::UnsafeBeginReceive() 	-> OverlappedAsyncResult#12036987
    System.Net.Sockets Verbose: 0 : [20440] Data from Socket#33711845::PostCompletion
    System.Net.Sockets Verbose: 0 : [20440] (printing 1024 out of 3339)
    System.Net.Sockets Verbose: 0 : [20440] 00000000 : 48 54 54 50 2F 31 2E 31-20 32 30 30 20 4F 4B 0D : HTTP/1.1 200 OK.
    System.Net.Sockets Verbose: 0 : [20440] 00000010 : 0A 43 61 63 68 65 2D 43-6F 6E 74 72 6F 6C 3A 20 : .Cache-Control: 
    System.Net.Sockets Verbose: 0 : [20440] 00000020 : 70 72 69 76 61 74 65 2C-20 6D 61 78 2D 61 67 65 : private, max-age
    System.Net.Sockets Verbose: 0 : [20440] 00000030 : 3D 30 0D 0A 43 6F 6E 74-65 6E 74 2D 4C 65 6E 67 : =0..Content-Leng
    System.Net.Sockets Verbose: 0 : [20440] 00000040 : 74 68 3A 20 36 30 34 39-31 0D 0A 43 6F 6E 74 65 : th: 60491..Conte

    But I would suggest you could try following idea to make your application sensitive dns change:

    1.  Replace the host name with ip address to splice the external service url before each request sent.

    static void Main(string[] args)
            {
                var client = new HttpClient();
    
                for (int i = 0; i < 200; i++)
                {
                    
                    {
                        Console.WriteLine("===================================");
                        Thread.Sleep(2000);
    
                        var host = Dns.GetHostEntry("www.bing.com");
                        
                        var response = client.GetAsync(string.Format("http://{0}", host.AddressList.FirstOrDefault()));
                        
                        Console.WriteLine("[RESPONSE] Service Uri: {0}\n[REQUEST] Header: {1}\n[RESPONSE] Header: {2}", response.Result.RequestMessage.RequestUri, response.Result.RequestMessage.Headers, response.Result.Headers);
    
                        Console.WriteLine("===================================");
                    }
                }
            }

    2.  For performance reason, you may hold a dictionary variable to cache the external service ip-host mapping, and detect this service host mapping with ip by timer thread. If there is any change for this external service host then update the cache variable. HttpClient must use external service uri from this cache variable.

    Best regards,

    Tuesday, April 14, 2015 7:13 AM

All replies

  • Hi Taj MSFT,

    I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

    Best Regards,
    Amy Peng

    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, April 3, 2015 5:52 AM
    Moderator
  • Hi Taj MSFT,

    It seems like this is a default behavior by HttpClient. You could see there is only once to call DNS::TryInternalResolve method to resolve the request uri when HttpClient fire the first request from diagnostics trace log.

    Enable diagnostics trace by following configuration in application config file:

    <?xml version="1.0" encoding="utf-8" ?>
    <configuration>
      <system.net>
        <settings>
          <servicePointManager dnsRefreshTimeout="0"/>
        </settings>
      </system.net>
    
      <system.diagnostics>
        <trace autoflush="true" />
        <sources>
          <source name="System.Net.Http">
            <listeners>
              <add name="System.Http.Trace"/>
            </listeners>
          </source>
          <source name="System.Net.Sockets">
            <listeners>
              <add name="System.Sockets.Trace"/>
            </listeners>
          </source>
        </sources>
        <sharedListeners>
          <add
            name="System.Http.Trace"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="System.Net.Http.trace.log"/>
    
          <add
            name="System.Sockets.Trace"
            type="System.Diagnostics.TextWriterTraceListener"
            initializeData="System.Net.Sockets.trace.log"/>
        </sharedListeners>
        <switches>
          <add name="System.Net.Http" value="Verbose" />
          <add name="System.Net.Sockets" value="Verbose"/>
        </switches>
      </system.diagnostics>
      <startup>
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
      </startup>
    </configuration>

    You may get the System.Net.Sockets.trace log like this, but there would be only one record for DNS::TryInternalResolve:

    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::Socket(AddressFamily#2)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::Socket() 
    System.Net.Sockets Verbose: 0 : [19080] Socket#37489757::Socket(AddressFamily#23)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#37489757::Socket() 
    System.Net.Sockets Verbose: 0 : [19080] DNS::TryInternalResolve(www.bing.com)
    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::BeginConnectEx()
    System.Net.Sockets Verbose: 0 : [19080] Socket#33711845::InternalBind(0.0.0.0:0#0)
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::InternalBind() 
    System.Net.Sockets Verbose: 0 : [19080] Exiting Socket#33711845::BeginConnectEx() 	-> ConnectOverlappedAsyncResult#64828693
    System.Net.Sockets Verbose: 0 : [13732] Socket#33711845::EndConnect(ConnectOverlappedAsyncResult#64828693)
    System.Net.Sockets Information: 0 : [13732] Socket#33711845 - Created connection from 10.172.18.234:4010 to 204.79.197.200:80.
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#33711845::EndConnect() 
    System.Net.Sockets Verbose: 0 : [13732] Socket#37489757::Close()
    System.Net.Sockets Verbose: 0 : [13732] Socket#37489757::Dispose()
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#37489757::Close() 
    System.Net.Sockets Verbose: 0 : [13732] Socket#33711845::UnsafeBeginSend()
    System.Net.Sockets Verbose: 0 : [13732] Exiting Socket#33711845::UnsafeBeginSend() 	-> OverlappedAsyncResult#11429296
    System.Net.Sockets Verbose: 0 : [20440] Data from Socket#33711845::PostCompletion
    System.Net.Sockets Verbose: 0 : [20440] 00000000 : 47 45 54 20 2F 20 48 54-54 50 2F 31 2E 31 0D 0A : GET / HTTP/1.1..
    System.Net.Sockets Verbose: 0 : [20440] 00000010 : 48 6F 73 74 3A 20 77 77-77 2E 62 69 6E 67 2E 63 : Host: www.bing.c
    System.Net.Sockets Verbose: 0 : [20440] 00000020 : 6F 6D 0D 0A 43 6F 6E 6E-65 63 74 69 6F 6E 3A 20 : om..Connection: 
    System.Net.Sockets Verbose: 0 : [20440] 00000030 : 4B 65 65 70 2D 41 6C 69-76 65 0D 0A 0D 0A       : Keep-Alive....
    System.Net.Sockets Verbose: 0 : [20440] Socket#33711845::EndSend(OverlappedAsyncResult#11429296)
    System.Net.Sockets Verbose: 0 : [20440] Exiting Socket#33711845::EndSend() 	-> Int32#62
    System.Net.Sockets Verbose: 0 : [20440] Socket#33711845::UnsafeBeginReceive()
    System.Net.Sockets Verbose: 0 : [20440] Exiting Socket#33711845::UnsafeBeginReceive() 	-> OverlappedAsyncResult#12036987
    System.Net.Sockets Verbose: 0 : [20440] Data from Socket#33711845::PostCompletion
    System.Net.Sockets Verbose: 0 : [20440] (printing 1024 out of 3339)
    System.Net.Sockets Verbose: 0 : [20440] 00000000 : 48 54 54 50 2F 31 2E 31-20 32 30 30 20 4F 4B 0D : HTTP/1.1 200 OK.
    System.Net.Sockets Verbose: 0 : [20440] 00000010 : 0A 43 61 63 68 65 2D 43-6F 6E 74 72 6F 6C 3A 20 : .Cache-Control: 
    System.Net.Sockets Verbose: 0 : [20440] 00000020 : 70 72 69 76 61 74 65 2C-20 6D 61 78 2D 61 67 65 : private, max-age
    System.Net.Sockets Verbose: 0 : [20440] 00000030 : 3D 30 0D 0A 43 6F 6E 74-65 6E 74 2D 4C 65 6E 67 : =0..Content-Leng
    System.Net.Sockets Verbose: 0 : [20440] 00000040 : 74 68 3A 20 36 30 34 39-31 0D 0A 43 6F 6E 74 65 : th: 60491..Conte

    But I would suggest you could try following idea to make your application sensitive dns change:

    1.  Replace the host name with ip address to splice the external service url before each request sent.

    static void Main(string[] args)
            {
                var client = new HttpClient();
    
                for (int i = 0; i < 200; i++)
                {
                    
                    {
                        Console.WriteLine("===================================");
                        Thread.Sleep(2000);
    
                        var host = Dns.GetHostEntry("www.bing.com");
                        
                        var response = client.GetAsync(string.Format("http://{0}", host.AddressList.FirstOrDefault()));
                        
                        Console.WriteLine("[RESPONSE] Service Uri: {0}\n[REQUEST] Header: {1}\n[RESPONSE] Header: {2}", response.Result.RequestMessage.RequestUri, response.Result.RequestMessage.Headers, response.Result.Headers);
    
                        Console.WriteLine("===================================");
                    }
                }
            }

    2.  For performance reason, you may hold a dictionary variable to cache the external service ip-host mapping, and detect this service host mapping with ip by timer thread. If there is any change for this external service host then update the cache variable. HttpClient must use external service uri from this cache variable.

    Best regards,

    Tuesday, April 14, 2015 7:13 AM
  • You can discard the HttpClient and create a new one every several minutes. Should be more light-weight than dealing with IP addresses.

    It works because a new HttpClient creates a new HttpClientHandler, and thus creates a new ConnectionGroupName for the HttpWebRequest. The connections in the old connection group are thus discarded.

    Sunday, February 5, 2017 10:35 AM
    Administrator