TFS11 upgrade - builds not working - stuck @ Running for 0 seconds

Question

We are running TFS11 as a pilot for the rest of our company.  We upgraded a project from TFS10 to TFS11. We are using the CMMI 5 template . 

TFS 2011 is installed on Server 8 Beta.  It is an advanced configuration, with SQL, SharePoint, Reporting services installed on different servers. The build components are installed on a Win 2008 R2 machine, with VS11 Team explorer installed (based on another post which indicated that those bits would be needed to work with TFS11. The Build service is configured with a service account which has local admin privileges. Build configuration is successful, controller and agent show as being available in TFS and the service account is registered in the project collection as both build admin and build service account.

Build templates were updated to DefaulTemplate11.1.xaml.  As a test, I manually kicked off a build.  The build never completes, and instead remains in the queue.  When I would double-click on the build, there was no status/errors, and instead only said "Running for 0 seconds".  Queuing more builds didn't help, as the first one would never be started or completed.  If I try to cancel one of these builds, I get this error:  "Failed to stop build 8 because the build machine did not respond to a stop request within 00:01:00."  The build machine is available and all services are running as far as I can tell. I had to resort to manually deleting records from the SQL server instance.

After reading some post around the interwebs, I proceeded to install Build Services on a Win 7 box, provisioned the same was as a Dev machine (so VS2010, all the packages in he build installed et)c.  I created a new controller and agents, and a new "Build Definition"  but I still see the exact same behavior when queuing a build from VS2011.  If I try to cancel one of these builds, I get this error:  "Failed to stop build 8 because the build machine did not respond to a stop request within 00:01:00."  The build machine is available and all services are running as far as I can tell.

I've also tried enabling the logging for the build service, by changing the TFSBuildServiceHost.exe.config file to log diagnostics (based on http://geekswithblogs.net/TarunArora/archive/2011/11/15/how-to-troubleshoot-tfs-build-server-failure.aspx )  However, even after giving the service account permission to write to that location, no log files are produced on any machine.  The only event log messages are "Service has successfully started".  

The only way to remove the builds is to edit them directly in the DB. (I have just been setting the status to failed)

So to summarize...no builds are being executed, builds cannot be cancelled, logging doesn't seem to work, and there are no event log errors to help figure this out.  

Any ideas? We need to get builds working, so we can move on to automated testing and integration with VMM 2012.

Thanks,

Martin Green

Answer 1

Hello Martin,

As far as I know that when you queue a build on the TFS build server, the new queued builds are put to the tail of the TFS builds queue. So the new queued builds will not run until the builds before it run completely.

And in my opinion, for your scenario to deal with it you need to create one more extra build agents. The new queued build will run on the free build agent.

If you still get the same issue, please post back.

Thanks.

---

Vicky Song [MSFT]
MSDN Community Support | Feedback to us

Answer 2

Vicky,

Thanks for the reply.  I think I did not make myself clear.  The build enters the queue and begins running.  It then stays in "Running for 0 seconds". It never progresses further than that. It doesn't complete, fail, partially fail. You also can not stop or cancel it. It just hangs.  The only way to remove it from the build instance is to delete it from tbl_builds and tbl_BuildQueue. 

This only occurs on projects that were migrated from TFS2010 (our custom app and the Tailspin toys collection from the VSALM 2010 vhd).  I have not had this problem with a sample application that we created on TFS11 from scratch (registered with the same build controllers - Builds work)

Martin

Answer 3

Hi Martin,

Welcome to the forum. 

Builds being stuck for TFS10 upgrade projects is a known issue with VS TFS 11 Beta. 

To my knowledge this issue has been fixed and will be rolled out with RC. 

Check out another thread where this has been reported before => http://social.msdn.microsoft.com/Forums/en-US/tfsbuild/thread/92b6aa39-63e4-45fd-91cc-ffbee683d486

HTH
Cheers, Tarun

---

Please remember to mark the replies as answers if they help.

Blog: http://geekswithblogs.net/TarunArora  Subscribe in a reader

Answer 4

Tarun,

I haven't heard or read anything about this being a known issue.  The thread you linked to does not seem similar, and the scripts do not have any effect.  Do you have a link to documentation or blogs that talk about this being a known issue with upgrade projects?  Is there a hotfix available?  I am currently working with a support engineer, and if I can request an internal hotfix, that would be fantastic.

Regards,

Martin

Answer 5

Martin,

I apologize for the issues you have encountered with the beta product. Regarding your troubles with logging/tracing, the build machine has gone through an entire redesign between TFS 2010 and TFS Dev11. Unfortunately the tracing was one of the first pieces to get cut, and was one of the last pieces to get added back in. However, I believe the changes to the tracing infrastructure will be a nice improvement as it is now entirely based on ETW (Event Tracing for Windows). This enables a couple of things:

1. There are two trace channels, Operational and Analytic. The Operational channel will be turned on by default and log things such as drop deletions, process errors, service start/stop, etc. This should provide a much better experience when attempting to diagnose issues with the machine.

2. The Analytic channel is off by default, but provides a pretty in-depth trace of the inner-workings of the machine service. This should allow us to gather information much more effectively than in beta. Ultimately this will also be better than TFS 2010 since you can enable/disable the tracing while the service is running, removing the necessity of stopping and starting the service which typically fixes an issue.

The improvements to the tracing infrastructure will be present in the RC build of TFS Dev11, so hopefully this will enable you to more easily diagnose issues.

Now going back to your issue, due to the lack of tracing in beta we will have to gather a dump of your build process to debug locally. You should be able to work with the support engineer you mentioned to get this sent to my team so I can investigate. As far as I know, this is not something specific to upgrading a team project but I can tell you more after I analyze your build service dump.

Thanks for the feedback and I hope you enjoy the improvements in RC!

Patrick

Answer 6

Thanks Patrick - working with you and Anjana we determined the problem was the message queue web config file was set to ntlm -

The answer to this specific problem was as simple as - 

• Please open the following config file:

C:\Program Files\Microsoft Team Foundation Server Dev11\Application Tier\Message Queue\web.config

In this file, there will be a custom binding configuration which looks like the following:

    <!-- Configure a custom binding for this endpoint -->

    <bindings>

      <customBinding>

        <binding name="TfsSoapBinding">

          <textMessageEncoding messageVersion="Soap12WSAddressing10" />

          <httpTransport authenticationScheme="Ntlm" manualAddressing="true" />

        </binding>

      </customBinding>

    </bindings>

Change the 'authenticationScheme' value from 'Ntlm' to 'IntegratedWindowsAuthentication'. If that doesn't work, the next thing to try would be to use 'Negotiate'. Once you make this change the application pool will automatically recycle and you can see if your build machine can now communicate with the message queue