how do i mitigate " The number of HTTP requests per minute exceeded the configured limit " problem in TFS 2010

Question

With a user base of 50 users, My TFS 2010 installation gives the following errors when there is a get latest or check-in on Multiple files.

Error 1:

"Team foundation Services are not available from server ....... Technical information (for administrator): the underlying connection was closed: The connection was closed unexpectedly.'

Error 2:

"...............Please contact your administrator. There was an error contacting the server. Technical information (for administrator): HTTP code 502: Proxy Error ( The number of HTTP requests per minute exceeded the configured limit. Contact your ISA Server administrator. )"

This isn't a show-stopper, but its quite a nuisance. I've tried finding answers for the same, The following link talks about the issue

http://social.msdn.microsoft.com/Forums/en/tfsversioncontrol/thread/71d70be7-ac43-4a35-ba47-2f0663433fec

The answer for this question ( http://support.microsoft.com/kb/937551 ) talks about making changes in the registry

KEY_CURRENT_USER\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\SourceControl

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\8.0\TeamFoundation\SourceControl

but my registry does not have the above mentioned nodes.

I really would like to mitigate if not fix this problem.

your help would be really appreciatd.

-Ben Salins

Answer 1

Have you checked Key 10.0 instead of 8.0?

---

Ahmed is an independent consultant in Ottawa, Canada specialized in .NET, Biztalk, WCF, WPF, TFS and other Microsoft technologies. Blog: http://lajak.wordpress.com Twitter: ahmedalasaad

Answer 2

Hi Ben Salins,

 

Thanks for your post.

 

In order to resolve this issue, I want to confirm some information with you:

1        What’s the version of your VS?

2        Try to reproduce this scenario on other client, still receive this error? If so, could you please provide the detailed reproduce steps here, I will reproduce this scenario on my machine follow your steps.

3        If your VS’ version is 2010, the node should be …\Microsoft\VisualStudio\10.0\TeamFoundation\....

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 3

Hi,

Lajak,
Yes The Key is 10.0.   But the keyword "MaxBackgroundThreads" still remains elusive in the key structure. (http://support.microsoft.com/kb/937551 )
infact, I've searched the entire registry for this keyword without any success.

John,
1. I'am using Visual Studio 2010 Ultimate version.
2. A brief background abt the installation
     a. Windows 2008 R2 Server installed on a Virtual Server.
     b. TFS 2010.
     c. Sql Server 2008
     d. currently used by around 20 users.
     e. The TFS database has been migrated from TFS 2008 to TFS 2010.
3. I had never faced this problem on my previous TFS 2008 installation. This problem has been specific to TFS 2010.
4. The Volume of the GET LATEST is around 2000 Files (10 MB of Data).  This get latest fails for the first time with errors mentioned in my first post. This problem gets mitigated by repeatedly doing a get latest, eventually getting all the latest files. however,  This is not a very pleasant thing to do.

I hope this info is sufficient. Please let me know if i could help you with more information.

-Ben Salins

 

 

Answer 4

Ahmed,

 
Yes The Key is 10.0.   But the keyword "MaxBackgroundThreads" still remains elusive in the key structure. (http://support.microsoft.com/kb/937551 )
infact, I've searched the entire registry for this keyword without any success.

Answer 5

Hi Ben,

 

Thanks for your reply.

 

I reproduce this scenario as your steps base on 100+ MB data files on my machine, and it work normally. So please confirm the following steps:

1        Perform this scenario base on a few files(1MB data or less) on your machine, still receive this error?

2        Reproduce this scenario on other clients.

3        Reproduce this scenario on other the server machine.

4        When this error happened, please check the Event View log, have any error in there?

5        Install the TFS Power Tools 2010, then click Start>> All Programs>> Microsoft TFS Power Tools 2010>> Best Practices Analyzer to scan the TFS installation. To download TFS Power Tools 2010 at: http://visualstudiogallery.msdn.microsoft.com/c255a1e4-04ba-4f68-8f4e-cd473d6b971f/  or go to Tools menu in our VS2010, and open Extension Manager… to download  and install.

 

 

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 6

Hi Ben,

 

What about the progress for this issue?

 

If you have any further research of this issue or resolved it, please share your experience here.

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 7

hi John, Thanks for the ping. i was leave. Joined back work today, will update my findings soon. ta, ben

Answer 8

Hi Ben,

 

Thanks for your reply.

 

We will wait your result.

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 9

hi John,

 I've tried the Get-Latest for less than an MB from multiple users. the error mentioned earlier could not be reproduced. Infact, i never had problems when less amount of data was fetched, this error only occured during large GETS.

 I have executed the best practices analyser as suggested by you. There was no problem (12 Green Check marks).

On going through the event log, i found TWO EVENTS which would interest you. I haven't been able to surmise the problem after looking at the logs, Maybe you can.

 thanks,

 ben

 

Log Name: Application Source: TFS Services Date: 3/2/2011 2:56:05 PM Event ID: 6003 Task Category: None Level: Warning Keywords: Classic User: N/A Computer: IND-MHP1V40W809.mastek.com Description: TF53010: The following error has occurred in a Team Foundation component or extension: Date (UTC): 3/2/2011 9:26:04 AM Machine: IND-MHP1V40W809 Application Domain: /LM/W3SVC/8080/ROOT/tfs-1-129435097340858233 Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727 Service Host: 715c88fe-7334-44bd-bd33-53163d3e63fe (TEAM FOUNDATION) Process Details: Process Name: w3wp Process Id: 4920 Thread Id: 1664 Account name: MASTEK\Gaurav10508 Detailed Message: TF30065: An unhandled exception occurred. Web Request Details Url: http://172.16.209.159:8080/tfs/_tfs_resources/VersionControl/v1.0/repository.asmx [method: POST] User Agent: Team Foundation (devenv.exe, 10.0.30319.1) Headers: Connection=Keep-Alive&Via=1.1+IND-MHP1IFP001&Content-Length=565&Content-Type=application%2fsoap%2bxml%3b+charset%3dutf-8&Accept-Language=en-US&Authorization=NTLM+TlRMTVNTUAADAAAAGAAYAIgAAAAYABgAoAAAAAwADABIAAAAFgAWAFQAAAAeAB4AagAAAAAAAAC4AAAABYKIogUBKAoAAAAPTQBBAFMAVABFAEsARwBhAHUAcgBhAHYAMQAwADUAMAA4AEkATgBEAC0ATQBIAFAAMwBEAFgAUAAwADAAMAAyAM5IGv5v267lAAAAAAAAAAAAAAAAAAAAAPaTl8zBMqkje7fPsBfGqYYLzwz05alOVg%3d%3d&Host=172.16.209.159%3a8080&User-Agent=Team+Foundation+(devenv.exe%2c+10.0.30319.1)&SOAPAction=http%3a%2f%2fschemas.microsoft.com%2fTeamFoundation%2f2005%2f06%2fVersionControl%2fClientServices%2f03%2fGet&X-TFS-Version=1.0.0.0&X-TFS-Session=ef274d92-9d54-41f6-b447-e8c748196428&TF-Instance=ef274d92-9d54-41f6-b447-e8c748196428 Path: /tfs/_tfs_resources/VersionControl/v1.0/repository.asmx Local Request: False Host Address: 172.16.202.4 User: MASTEK\Gaurav10508 [authentication type: NTLM] Exception Message: Server cannot clear headers after HTTP headers have been sent. (type HttpException) Exception Stack Trace: at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="TFS Services" /> <EventID Qualifiers="0">6003</EventID> <Level>3</Level> <Task>0</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2011-03-02T09:26:05.000000000Z" /> <EventRecordID>9187</EventRecordID> <Channel>Application</Channel> <Computer>IND-MHP1V40W809.mastek.com</Computer> <Security /> </System> <EventData> <Data>TF53010: The following error has occurred in a Team Foundation component or extension: Date (UTC): 3/2/2011 9:26:04 AM Machine: IND-MHP1V40W809 Application Domain: /LM/W3SVC/8080/ROOT/tfs-1-129435097340858233 Assembly: Microsoft.TeamFoundation.Framework.Server, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727 Service Host: 715c88fe-7334-44bd-bd33-53163d3e63fe (TEAM FOUNDATION) Process Details: Process Name: w3wp Process Id: 4920 Thread Id: 1664 Account name: MASTEK\Gaurav10508 Detailed Message: TF30065: An unhandled exception occurred. Web Request Details Url: http://172.16.209.159:8080/tfs/_tfs_resources/VersionControl/v1.0/repository.asmx [method: POST] User Agent: Team Foundation (devenv.exe, 10.0.30319.1) Headers: Connection=Keep-Alive&Via=1.1+IND-MHP1IFP001&Content-Length=565&Content-Type=application%2fsoap%2bxml%3b+charset%3dutf-8&Accept-Language=en-US&Authorization=NTLM+TlRMTVNTUAADAAAAGAAYAIgAAAAYABgAoAAAAAwADABIAAAAFgAWAFQAAAAeAB4AagAAAAAAAAC4AAAABYKIogUBKAoAAAAPTQBBAFMAVABFAEsARwBhAHUAcgBhAHYAMQAwADUAMAA4AEkATgBEAC0ATQBIAFAAMwBEAFgAUAAwADAAMAAyAM5IGv5v267lAAAAAAAAAAAAAAAAAAAAAPaTl8zBMqkje7fPsBfGqYYLzwz05alOVg%3d%3d&Host=172.16.209.159%3a8080&User-Agent=Team+Foundation+(devenv.exe%2c+10.0.30319.1)&SOAPAction=http%3a%2f%2fschemas.microsoft.com%2fTeamFoundation%2f2005%2f06%2fVersionControl%2fClientServices%2f03%2fGet&X-TFS-Version=1.0.0.0&X-TFS-Session=ef274d92-9d54-41f6-b447-e8c748196428&TF-Instance=ef274d92-9d54-41f6-b447-e8c748196428 Path: /tfs/_tfs_resources/VersionControl/v1.0/repository.asmx Local Request: False Host Address: 172.16.202.4 User: MASTEK\Gaurav10508 [authentication type: NTLM] Exception Message: Server cannot clear headers after HTTP headers have been sent. (type HttpException) Exception Stack Trace: at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) </Data> </EventData> </Event> ------------------------------------------------------------------------------------------------------------------------------------- Log Name: Application Source: ASP.NET 4.0.30319.0 Date: 3/2/2011 2:56:06 PM Event ID: 1309 Task Category: Web Event Level: Warning Keywords: Classic User: N/A Computer: IND-MHP1V40W809.mastek.com Description: Event code: 3005 Event message: An unhandled exception has occurred. Event time: 3/2/2011 2:56:06 PM Event time (UTC): 3/2/2011 9:26:06 AM Event ID: 78e440e54b9f4efbb1ee9d2f98e5d2f1 Event sequence: 2 Event occurrence: 1 Event detail code: 0 Application information: Application domain: /LM/W3SVC/8080/ROOT/tfs-1-129435097340858233 Trust level: Full Application Virtual Path: /tfs Application Path: C:\Program Files\Microsoft Team Foundation Server 2010\Application Tier\Web Services\ Machine name: IND-MHP1V40W809 Process information: Process ID: 4920 Process name: w3wp.exe Account name: MASTEK\tfsservice Exception information: Exception type: HttpException Exception message: Server cannot clear headers after HTTP headers have been sent. at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) Request information: Request URL: http://172.16.209.159:8080/tfs/_tfs_resources/VersionControl/v1.0/repository.asmx Request path: /tfs/_tfs_resources/VersionControl/v1.0/repository.asmx User host address: 172.16.202.4 User: MASTEK\Gaurav10508 Is authenticated: True Authentication Type: NTLM Thread account name: MASTEK\tfsservice Thread information: Thread ID: 30 Thread account name: MASTEK\tfsservice Is impersonating: False Stack trace: at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) Custom event details: Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="ASP.NET 4.0.30319.0" /> <EventID Qualifiers="32768">1309</EventID> <Level>3</Level> <Task>3</Task> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2011-03-02T09:26:06.000000000Z" /> <EventRecordID>9188</EventRecordID> <Channel>Application</Channel> <Computer>IND-MHP1V40W809.mastek.com</Computer> <Security /> </System> <EventData> <Data>3005</Data> <Data>An unhandled exception has occurred.</Data> <Data>3/2/2011 2:56:06 PM</Data> <Data>3/2/2011 9:26:06 AM</Data> <Data>78e440e54b9f4efbb1ee9d2f98e5d2f1</Data> <Data>2</Data> <Data>1</Data> <Data>0</Data> <Data>/LM/W3SVC/8080/ROOT/tfs-1-129435097340858233</Data> <Data>Full</Data> <Data>/tfs</Data> <Data>C:\Program Files\Microsoft Team Foundation Server 2010\Application Tier\Web Services\</Data> <Data>IND-MHP1V40W809</Data> <Data> </Data> <Data>4920</Data> <Data>w3wp.exe</Data> <Data>MASTEK\tfsservice</Data> <Data>HttpException</Data> <Data>Server cannot clear headers after HTTP headers have been sent. at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) </Data> <Data>http://172.16.209.159:8080/tfs/_tfs_resources/VersionControl/v1.0/repository.asmx</Data> <Data>/tfs/_tfs_resources/VersionControl/v1.0/repository.asmx</Data> <Data>172.16.202.4</Data> <Data>MASTEK\Gaurav10508</Data> <Data>True</Data> <Data>NTLM</Data> <Data>MASTEK\tfsservice</Data> <Data>30</Data> <Data>MASTEK\tfsservice</Data> <Data>False</Data> <Data> at System.Web.HttpResponse.ClearHeaders() at System.Web.Services.Protocols.SoapServerProtocol.WriteException(Exception e, Stream outputStream) at System.Web.Services.Protocols.WebServiceHandler.WriteException(Exception e) at System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest() at System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(HttpContext context) at System.Web.Script.Services.ScriptHandlerFactory.HandlerWrapper.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) </Data> </EventData> </Event>

Answer 10

Hi Ben,

 

Thanks for your reply.

 

According your description and the practices result, it seems that there have no issue with TFS, the error log may be caused by the network connection.

 

So we need to confirm, please try the Get scenario on TFS server machine, check the result.

 

If that can work normally on TFS server machine, please check the network/firewall and IIS configure, may be have any limit in somewhere.

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 11

Hi Ben,

 

Have you tried my previous suggestion?

---

John Qiao [MSFT]
MSDN Community Support | Feedback to us
Get or Request Code Sample from Microsoft
Please remember to mark the replies as answers if they help and unmark them if they provide no help.

Answer 12

I'm having a similar problem on my TFS.
What was done to solve the problem?

Thank you

Answer 13

The same issue here.

Did you solve the problem?

Answer 14

Similar issue here -

Details:

Users connected locally or via VPN to the corporate domain have no problem.

Remote users i.e. subcontractors are reciving the "500 error exceeding the limit..." when performing a "Get Latest"

I have tried the 'MaxBackgroundThreads' entries in the registry (both hives-Local Machine and Current User), the TF.exe.config, and the devenv.exe.config files to no avail.

I have tried with all (4) entries set to "2" and again with "1" but still receive the error. However, with the MaxBackgoundThreads set to "1" I did receive more of the files and almost completed the get latest successfully.

With the setting at "2" we would receive about 317MB before the error.

With the setting at "1" we would receive about 377MB before the error.

Apparently the settings on the TMG for the corporate domain and external (non-VPN) connections sees this as a DOS attack and drops the connection.

I'm not a member of the IT department, just a CM guy trying to make sure the development team can access and use TFS. We did not have this issue with Rational ClearCase Remote Client and do not want to go back to that environment.

Is there a way to limit the number of HTTP requests between the VS 2010 client and TFS server during a get latest?

The corporate IT department is hesitant on changing the settings on their side because the only change they can make on the proxy server is the global threshold used to police all clients.  Unfortunately, increasing the global threshold would make the entire company more vulnerable to denial of service attacks.

This is a real critical issue for us so if someone has a tried and true solution I would love to see it :)

Cheers,

Jeff