locked
Keep-alive packets continue being sent after client closes (server socket is CLOSE_WAIT, client socket is FIN_WAIT_2) RRS feed

  • Question

  • I was testing out creating a connection to a server and specifying that the socket should use keep-alives. My end goal in using keep-alives is to prevent disconnection due to network inactivity. That isn't the problem though. 

    I created a simple server which just does this:

    1. Listen for connections
    2. Accept the connection
    It does not read/send data or close the connections, just accumulates them.

    I created a client which:
    1. connects to the server (and the Socket that it creates is using keep-alives)
    2. waits a while (so that I can use a packet sniffing tool like WireShark to verify that the keep-alive packets are being sent)
    3. quits 
    The last step (quits) is where I'm running into problems. Depending upon how I "quit" (how I close/disconnect/shutdown/dispose the Socket), it can result in the sockets not being fully closed. If I run the "netstat -aon" command, I can see that the client->server connection is in a FIN_WAIT_2 state, and the server->client connection is in a CLOSE_WAIT state. The keep-alive packets continue being sent by the client (and ack-ed by the server).

    Alternatively, I've coded the client in such a way that when it "quits", the connections completely go away (don't show up on client or server when checking with the netstat command), and the keep-alive packets are no longer exchanged (all is well). My problem is that it doesn't make sense to me why the connection remains when I code it in a certain way. 

    Here is the server code:
    using System;
    using System.Collections.Generic;
    using System.Net;
    using System.Net.Sockets;
    using System.Threading;
    
    public class Test
    {
    	public static void Main()
    	{
    		const int PORT = 19999;
    		try
    		{
    			List<Socket> connections = new List<Socket>();
    			TcpListener listener = new TcpListener(IPAddress.Any, PORT);
    			listener.Start();
    			Console.WriteLine("Listening for connections on port " + PORT + "...");
    
    			while (true)
    			{
    				Thread.Sleep(100);
    				if (listener.Pending())
    				{
    					Socket oSocket = listener.AcceptSocket();
    					connections.Add(oSocket);
    					Console.WriteLine("Server added a connection at UTC \t" + DateTime.UtcNow);
    				}
    			}//while
    		}
    		catch (Exception e)
    		{
    			Console.WriteLine("Exception at UTC \t" + DateTime.UtcNow + "\n" + e);
    		}
    	}
    }
    

    and here is the client code:

    using System;
    using System.Net;
    using System.Net.Sockets;
    using System.Runtime.InteropServices;
    using System.Threading;
    
    public class P
    {
    	public static void Main()
    	{
    		Console.WriteLine("Starting.");
    
    		string ip = "172.17.1.133";
    		int port = 19999;
    		int waitMillis = 15 * 1000;
    
    		Socket s = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
    
    		s.SetSocketKeepAliveValues(3000, 1); //send keep-alive message every 3 seconds
    
    		//s.LingerState = new LingerOption(false, 0);
    
    		Console.WriteLine("Connecting to " + ip + ":" + port);
    
    		s.Connect(IPAddress.Parse(ip), port);
    
    		Thread.Sleep(waitMillis); //sleep for a while (use WireShark to see if keep alives are sent during this time)
    
    		s.Disconnect(false);
    		//s.Shutdown(SocketShutdown.Both);
    		//s.Close();
    		//s.Dispose();
    
    		Console.WriteLine("Done.");
    	}
    }
    
    public static class SocketExtensions
    {
    	//found at http://www.extensionmethod.net/Details.aspx?ID=271, slightly modified
    	/// <summary>
    	/// Using IOControl code to configue socket KeepAliveValues for line disconnection detection(because default is toooo slow) 
    	/// </summary>
    	/// <param name="instance">A socket instance</param>
    	/// <param name="KeepAliveTime">The keep alive time. (ms)</param>
    	/// <param name="KeepAliveInterval">The keep alive interval. (ms)</param>
    	public static void SetSocketKeepAliveValues(this Socket instance, int KeepAliveTime, int KeepAliveInterval)
    	{
    		//KeepAliveTime: default value is 2hr
    		//KeepAliveInterval: default value is 1s and Detect 5 times
    
    		//the native structure
    		//struct tcp_keepalive {
    		//ULONG onoff;
    		//ULONG keepalivetime;
    		//ULONG keepaliveinterval;
    		//};
    
    		uint dummy = 0; //lenth = 4
    		byte[] inOptionValues = new byte[System.Runtime.InteropServices.Marshal.SizeOf(dummy) * 3]; //size = lenth * 3 = 12
    		bool OnOff = true;
    
    		BitConverter.GetBytes((uint)(OnOff ? 1 : 0)).CopyTo(inOptionValues, 0);
    		BitConverter.GetBytes((uint)KeepAliveTime).CopyTo(inOptionValues, Marshal.SizeOf(dummy));
    		BitConverter.GetBytes((uint)KeepAliveInterval).CopyTo(inOptionValues, Marshal.SizeOf(dummy) * 2);
    		// of course there are other ways to marshal up this byte array, this is just one way
    		// call WSAIoctl via IOControl
    
    		// .net 1.1 type
    		//int SIO_KEEPALIVE_VALS = -1744830460; //(or 0x98000004)
    		//socket.IOControl(SIO_KEEPALIVE_VALS, inOptionValues, null); 
    
    		// .net 3.5 type
    		instance.IOControl(IOControlCode.KeepAliveValues, inOptionValues, null);
    	}
    }
    

    Here is the network traffic in the "Good Case" (note that the ">" means traffic from the client to the server, and "<" means from the server to the client):

    Time
    0	> SYN
    0	< SYN, ACK
    0	> ACK
    
    3	> TCP Keep-Alive
    3	< TCP Window update
    
    6	> TCP Keep-Alive
    6	< TCP Keep-Alive ACK
    
    9	> TCP Keep-Alive
    9	< TCP Keep-Alive ACK
    
    12	> TCP Keep-Alive
    12	< TCP Keep-Alive ACK
    
    15	> TCP Keep-Alive
    15	< TCP Keep-Alive ACK
    
    15	> FIN, ACK
    15	< ACK
    15	> RST, ACK
    on client side socket is gone (doesn't show up with "netstat -aon")

    Here is the network traffic in the "Bad Case":

    Time
    0	> SYN
    0	< SYN, ACK
    0	> ACK
    
    3	> TCP Keep-Alive
    3	< TCP Window update
    
    6	> TCP Keep-Alive
    6	< TCP Keep-Alive ACK
    
    9	> TCP Keep-Alive
    9	< TCP Keep-Alive ACK
    
    12	> TCP Keep-Alive
    12	< TCP Keep-Alive ACK
    
    15	> TCP Keep-Alive
    15	< TCP Keep-Alive ACK
    
    15	> FIN, ACK
    15	< ACK
    
    18	> TCP Keep-Alive
    18	< TCP Keep-Alive ACK	
    ...above two packets repeat every three seconds
    on client side socket is in FIN_WAIT_2 state
    on server side socket is in CLOSE_WAIT state

    Now, many different ways of quitting in the client can result in the "Bad Case", here are a few of the things I've tried doing to the client socket that result in the bad case:

    Dispose()
    
    Disconnect(true)
    
    Disconnect(false)
    Dispose()
    
    Shutdown(SocketShutdown.Both);
    
    LingerState = new LingerOption(false, 0);
    Disconnect(false)
    Dispose()
    
    Close()
    
    Shutdown(SocketShutdown.Both)
    Close()
    
    Shutdown(SocketShutdown.Both)
    Close()
    Dispose()
    
    Disconnect(false)
    Shutdown(SocketShutdown.Both)
    Close()
    
    LingerState = new LingerOption(false, 0); --note that this is commented out in the above client code
    Disconnect(false)
    Shutdown(SocketShutdown.Both)
    Close()
    

    and the only way I've been able to produce the "Good case" is by doing this:

    Disconnect(false)

    Can anyone explain why this behavior occurs, suggest a change, or point me to a link that would explain this?

    Thanks!

     

    Monday, January 3, 2011 7:32 PM

All replies

  • bump
    Wednesday, January 19, 2011 11:02 PM
  • Excellent detective work... and as you discovered the RST has to fly to close those connections out.  In the bad case the client never RSTs the session... The only problem of course is discovering if this is a stack issue or application issue.

    In some cases the TCP/IP stack will RST converstation based on protocol violations or other stack like things is has to do like Keep alive.  It's not clear on different systems who does what.  For example I would ask can the application invoke both the FIN AND RST on a close?  Or does the application send the FIN with the stack RST after all rules are exhausted.  The bottom line is we call these half-sessions, where theres a FIN Wait or Close Wait, the stack should eventually clean these up...

    Now because you've indicated this is occasional....and that Disconnect(False) works, let's look at that first.

    Disconnect False will tell the stack don't reuse that socket.  So in my mind that means the reset is sent by the stack because everything is going away including stack resources.  In the case of Disconnect(True)... Then Fin Wait 2 would seem to indicate another Open is needed or a RST from other side.  In other words the Stack is waiting around for something else before it hits a time-out and closes up shop...

    That's my best guess.


    Javaman Cowboy Coders Unite!
    Wednesday, January 19, 2011 11:36 PM
  • I'll get a more thorough reading of your message later, but a note on TCP/IP bidirectionality for now...

    A TCP connection is open until BOTH sides close the connection.  It is defined as a bidrectional stream, and one end can say: "I have finished SENDING" independently of the other end, that's exactly what FIN means and no more.  The connection is not closed until both end have sent FINs (and both have received ACKs for them??), or until either side says 'abort, something wrong with the connection' and sends a RST.

    For instance in the olden days with Unix "rsh" command e.g. "rsh pc2 slow_process.exe < the_input.txt > the_output.txt".  The client would send FIN at EoF of stdin/the_input.txt and would listen for ages for all the response followed by the FIN representing the EoF of stdout/the_output.txt. :-)


    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Thursday, January 20, 2011 3:51 PM
  • The weird thing is that if I do anything extra after "Disconnect(false)", then I get the "bad case" (keepalive packets keep being sent).

    	Disconnect(false)
    	Dispose()

    and 

    	Disconnect(false)
    	Shutdown(SocketShutdown.Both)
    	Close()

    both resulted in the "bad case".

    Monday, January 24, 2011 11:04 PM
  • bump
    Tuesday, February 1, 2011 7:03 PM
  • Ok Disconnect is a really wierd function and is called DisconnectEx in winsock so that it is more obvious that it's only for special case usage in advanced cases.  If you don't use Disconnect then everything should be ok.

    I wrote a comment on this at MSDN but unfortunately the comment stayed on the contemporary version of the framework and isn't visible in the current versions. :-( See the comment at http://msdn.microsoft.com/en-us/library/system.net.sockets.socket.disconnect(VS.85).aspx

    • Presumably this method calls the Winsock function DisconnectEx (http://msdn2.microsoft.com/en-us/library/ms737757.aspx). Its summary says:

      The DisconnectEx function closes a connection on a socket, and allows the socket handle to be reused. [my emphasis]

      So it affects the socket handle not the TCP endpoint.

      Thus this is for very advanced uses only, do not use unless you are creating an server that has very high connection volumes, and you really know what you are doing. In the majority of cases, calling Close, possibly preceded by Shutdown, is the correct action.

      It does not help with the TIME_WAIT issue that (by design) prevents a new TCP socket from being opened with the same address and port number(s) as one closed recently.

     


    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Wednesday, February 2, 2011 10:04 AM
  • Thanks for the information on Disconnect. Unfortunately, I have tried Close and Close-preceded-by-Shutdown (see above list of function calls which led to the "bad case"). Some variants below:

     

    Close()
    
    Shutdown(SocketShutdown.Both)
    Close()
    
    Shutdown(SocketShutdown.Both)
    Close()
    Dispose()
    
    Disconnect(false)
    Shutdown(SocketShutdown.Both)
    Close()

    If I had not seen one example of the "good case" actually happen I would be starting to think that it is not possible to get this situation to happen if the server is not closing these connections (recall it just accepts connections and holds them in a collection). Is there some fundamental thing about socket closing that I'm not getting? 


     

    Wednesday, February 2, 2011 3:26 PM
  • So is it still the: a TCP/IP connection is a duplex conection problem??  If its still like this:

    • The last step (quits) is where I'm running into problems. Depending upon how I "quit" (how I close/disconnect/shutdown/dispose the Socket), it can result in the sockets not being fully closed. If I run the "netstat -aon" command, I can see that the client->server connection is in a FIN_WAIT_2 state, and the server->client connection is in a CLOSE_WAIT state. The keep-alive packets continue being sent by the client (and ack-ed by the server).

    Both ends _must_ close a connection before the connection is closed -- it can happily sit in half-closed (or is it half-open, I can never remember[1])[2].  The server is not closing the connection (*CLOSE_WAIT*).

    So you must get the server program to call Close or Shutdown(Send).

    Or...... One can brutally close the connection by sending a RST, one does that by setting socket option SO_LINGER with a *zero* timeout will do a hard close at closesocket, see http://msdn.microsoft.com/en-us/library/aa917514.aspx.  That's in winsock terms, so that's socket.LingerState = new LingerOption(true, 0);  (see http://social.msdn.microsoft.com/Forums/en-US/netfxnetcom/thread/529d54fd-594c-4420-b581-3670defd76b6)

    Alan

     

     

    [1] Half-closed and half-open are different states...

    [2] Well until a security update a year or two back where MSFT added timeouts to lots of connection states that previously could be used to cause a resources-exhaustion DoS attack.


    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Wednesday, February 2, 2011 10:03 PM