none
TFS check-in problem with connections being closed

    Question

  • We have a frustrating problem in that we can only check in single files at a time without getting an error that the underlying connection was closed. Every once in a while we get lucky and can check in multiple files but, for the most part cannot. I have enabled verbose logging and have captured this

     

    4/26/2007 4:26:56 PM (pid 7888, tid 1184, 469593 ms) Recorded m_exception in UpDownAsyncResult.Completed(): Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: C:\Code\Hydra\Hydra\HydraConfiguration\Debug\CUCorp.Hydra.BusinessEntity.DAC.config: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       --- End of inner exception stack trace ---
       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
       --- End of inner exception stack trace ---
       at System.Net.HttpWebRequest.GetResponse()
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadChunk(FileStream fileContentStream, Int64 end)
       --- End of inner exception stack trace ---
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadChunk(FileStream fileContentStream, Int64 end)
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadContent()

     

    The TFS server is not showing any errors. The network admins don't believe it's a network issue. I enabled Windows firewall logging and see connection drops (?) over 8080 to the server

    2007-04-26 16:26:56 OPEN TCP 10.2.2.10 10.2.6.20 2597 8080 - - - - - - - - -
    2007-04-26 16:26:56 CLOSE TCP 10.2.2.10 10.2.6.20 2597 8080 - - - - - - - - -
    2007-04-26 16:26:56 DROP TCP 10.2.6.20 10.2.2.10 8080 2597 40 A 3276693809 1328312781 65535 - - - RECEIVE
    2007-04-26 16:26:56 DROP TCP 10.2.6.20 10.2.2.10 8080 2597 40 A 3276693809 1328314257 65535 - - - RECEIVE
    2007-04-26 16:26:56 DROP TCP 10.2.6.20 10.2.2.10 8080 2597 40 R 3276693809 0 62395 - - - RECEIVE
    2007-04-26 16:26:56 DROP TCP 10.2.6.20 10.2.2.10 8080 2597 40 R 3276693809 0 62395 - - - RECEIVE
    2007-04-26 16:26:56 DROP TCP 10.2.6.20 10.2.2.10 8080 2597 40 R 3276693809 0 62395 - - - RECEIVE
    2007-04-26 16:26:56 OPEN TCP 10.2.2.10 10.2.6.20 2598 8080 - - - - - - - - -
    2007-04-26 16:26:56 OPEN TCP 10.2.2.10 10.2.6.20 2599 8080 - - - - - - - - -
    2007-04-26 16:26:56 DROP ICMP 10.2.0.1 10.2.2.10 - - 56 - - - - 5 1 - RECEIVE
    2007-04-26 16:26:56 DROP UDP 10.2.0.124 10.2.3.255 137 137 78 - - - - - - - RECEIVE
    2007-04-26 16:26:56 OPEN TCP 10.2.2.10 10.2.6.20 2600 8080 - - - - - - - - -

     

    Anyone run into a problem like this before?

    Thursday, April 26, 2007 9:46 PM

Answers

  • In the past we have had some issues regarding multiple connections but I dont know if it is really this.

     

    I think the best thing for EVERYONE who is hitting this issue at this point is to call product support and see if we can get involved and debug the issue for those customers. We apologize for this incident and hopefully we can get to the bottom of it as quickly as possible but we need you to submitted to support so we can properly debug it.

     

    Thanks mario

    Monday, August 06, 2007 12:02 PM
  •  

    I had the same problem on a very regular basis. When I disable Norton Internet Security it goes away completely.
    Saturday, October 13, 2007 7:50 PM

All replies

  • Does the TFS Server have plenty of resources?  I've never seen this, but I've seen something similar when the ASP_NET worker process is receycling very frequently due to memory etc being low on the TFS AT (usually when running the server in an underpowered virtualised environment)

    Also, are you sure that your connection is not being proxied by some network device, firewall or NAT?


    Thursday, April 26, 2007 11:19 PM
  • The server is a dual Xeon processor (3GHz) and has 1GB of RAM. It's running Windows 2003 Server Standard Edition and is at SP1. It has McAfee VirusScan Enterprise 7.1.0. Our workstations are running McAfee VirusScan Enterprise version 8.5.0i. I disabled  the On-Access Scan before and was able to check in several files but, a co-worker tried the same steps and got the same connection closed error.

     

    The worker processes are set to recycle every 1740 minutes.

     

    TFS is the full RTM version (8.0.50727.147) and wasn't migrated from a pre-release version.

     

    I've verified that I have the latest drivers for the network card in my computer. The only protocols enabled for the network connection are Client for Microsoft Networks and Internet Protocol.

    Friday, April 27, 2007 12:10 PM
  • Hello Jim:

     

      Some other users said  that the disable of the real time protection of McAfee does not work.

     

    They said that they uninstall it. then it works ok. But I am not sure. 

     

     Good luck.

     

    Thursday, May 03, 2007 7:28 AM
  • We thought we made some progress because we were error free for a couple of days after performing the following:

    1) Updated the driver on our NICs to the latest version.

    2) Created exceptions on port 8080 for TCP and UDP in the Microsoft firewall (we allow exceptions)

    3) Took the items of the connection property down to Client for Microsoft Networks and Internet Protocol.

     

    Went to check in two items to TFS and got the connection error again.

     

    5/2/2007 10:52:37 AM (pid 5588, tid 828, 8473187 ms) Recorded m_exception in UpDownAsyncResult.Completed(): Microsoft.TeamFoundation.VersionControl.Client.VersionControlException: C:\Code\Hydra\Hydra\HydraStoredProcedures\Tables\Create Tables\CreateCUCorpFilesTables.sql: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       --- End of inner exception stack trace ---
       at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
       --- End of inner exception stack trace ---
       at System.Net.HttpWebRequest.GetResponse()
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadChunk(FileStream fileContentStream, Int64 end)
       --- End of inner exception stack trace ---
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadChunk(FileStream fileContentStream, Int64 end)
       at Microsoft.TeamFoundation.VersionControl.Client.FileUploader.UploadContent()

    Thursday, May 03, 2007 5:18 PM
  • Hi Jim:

     

    We are trying to take a look at this and see if we can assist you without having product support involved.  We will let you know within a day.

     

    Thanks,

    mario

    Tuesday, May 08, 2007 1:03 PM
  • Mario,

     

    Thank you. I can provide netmon and tfs logging.

     

    Jim

    Tuesday, May 08, 2007 9:14 PM
  • We are also having this issue.  We have about 4 developers using TFS and they get the dropped connection problems also.  So any help you can provide would be appreciated.  We are running 8.0.50727.142
    Tuesday, May 29, 2007 4:52 PM
  • Any ideas Mario?
    Thursday, May 31, 2007 5:06 PM
  • I'm having this "underlying connection being closed" problem too. I am accessing TFS externally using https through a reverse web proxy.

    First I tested from inside our network with the reverse web proxy configured as the TFS server. It didn't seem to have this  trouble. Now I am at home with the same client machine (a laptop) and I get this problem. It isn't possible to interact well with TFS due to the error. My connection is on a FIOS network (15 mbps downlink, 2 mbps uplink), so I don't think bandwidth is the issue.

    This is a really big problem because our company uses offshore teams, and we want them to user our TFS server. This is probably a use case scenario that gets more frequent all the time...
    Thursday, June 21, 2007 4:53 AM
  • We are also seeing this issue.  "The underlying connection was closed: A connection that was expected to be kept alive was closed by the server".  It's an intermittent issue and re-trying the operation will occasionally let us connect.  It seems to be more common when trying file operations on multiple files rather than individual ones.  Has anyone arrived at a solution for this? 

     

     

    Thursday, August 02, 2007 7:38 PM
  • In the past we have had some issues regarding multiple connections but I dont know if it is really this.

     

    I think the best thing for EVERYONE who is hitting this issue at this point is to call product support and see if we can get involved and debug the issue for those customers. We apologize for this incident and hopefully we can get to the bottom of it as quickly as possible but we need you to submitted to support so we can properly debug it.

     

    Thanks mario

    Monday, August 06, 2007 12:02 PM
  •  

    I had the same problem on a very regular basis. When I disable Norton Internet Security it goes away completely.
    Saturday, October 13, 2007 7:50 PM
  • Hi,

    our developers are reporting what appear to be similar problems - did you get to the bottom of this?

    Thanks.

     

    Monday, January 14, 2008 9:36 AM
  • Does anyone know what files we should make Symantec skip so this is not a problem?  We are having the same issues but do not have the option to disable Symantec.

     

    Thanks,

     

    Chad

     

    Monday, August 25, 2008 7:00 PM
  • For us it ended seeming to be a network issue. We moved to a different room (about 10 feet away from where we were) and the problems "magically" went away.
    Friday, November 14, 2008 6:06 PM
  • I know it's been a couple years now, but can you provide any more detail on the nature of the network problem? I've started getting this error more and more over the last few months. I'm working over a VPN. It wouldn't be such a big deal if Visual Studio could recover from it.
    Wednesday, August 11, 2010 6:46 PM