locked
VSO build.preview / build.vnext builds will not start with on prem build agent RRS feed

  • Question

  • I'm having a problem with VSO Build.Preview builds getting put on the build queue but never actually sent to the on prem build agent.

    We have a single build agent and the agent is sitting idle.  However, none of the queued builds are sent to the agent.

    The status page for the builds just say:

      The build has been queued and is waiting to start.

    This is currently happening today (may 19th 2015) as of 2:00 pm CST.  It also happened yesterday between the hours of about 1:00pm and 5:00 pm CST.

    With no success I have tried rebooting the server the agent lives on and deleting all queued builds and re-queueing them.

    Any help would be greatly appreciated.  Even if it's just to point me in the right direction to where I can report this issue.



    • Edited by aaron.carlson Wednesday, May 27, 2015 1:19 PM fixed grammer
    Tuesday, May 19, 2015 7:35 PM

Answers

  • Ok, it sounds like you did hit the issue we had during the deployment rollout, you should definitely not see this again. Let me know if you do and I'll dig in.

    Thanks!

    Dennis

    • Marked as answer by aaron.carlson Thursday, May 21, 2015 2:35 PM
    Thursday, May 21, 2015 2:19 PM

All replies

  • Hi, are you saying your builds were working most of the time (you've had successful builds during other time periods) and hitting issues only during the given time periods? I just want to confirm that...

    We did have an issue during the most recent deployment that caused some Build.Preview builds to be delayed in the queue but I believe we resolved that before opening up the Build Preview publicly, were you on an early adopter account? Have you seen it since?

    Thursday, May 21, 2015 11:47 AM
  • Hi Dennis,  I am an early adopter.   I've been successfully using the new build system for about 2 months before it went public preview.  It's great BTW.

    On Monday May 18, 2015 at about 1:00 pm I noticed that our agent was out of date and VSO would not allow me to queue new builds.  So I manually updated it and VSO would allow me to queue new builds.  However, at that point was when I noticed queued builds were not being processed.  I thought I updated it wrong and ended deleting the agent completely and registering/installing the agent from scratch which didn't fix anything.  At about 5:00 pm builds started being processed again on their own.  Then Tuesday May 19, 2015 the same thing happened and builds stopped being processed around 1:00 pm and came back around 5:00 pm.


    Thursday, May 21, 2015 2:13 PM
  • Ok, it sounds like you did hit the issue we had during the deployment rollout, you should definitely not see this again. Let me know if you do and I'll dig in.

    Thanks!

    Dennis

    • Marked as answer by aaron.carlson Thursday, May 21, 2015 2:35 PM
    Thursday, May 21, 2015 2:19 PM
  • Another interesting thing to note.

    In the past the queued builds that I canceled have immediately shown up on the Completed tab with a canceled status.

    During the time the queued builds were not being processed I canceled some of those builds .  Those builds were removed from the build queue instantaneously.  However, They did not show up on the Completed builds tab until the queued builds started being processed again.

    I'm not sure but this makes me think there was something wrong with the service that schedules jobs with the agent pool and not the agent itself.


    • Edited by aaron.carlson Thursday, May 21, 2015 2:37 PM spelling
    Thursday, May 21, 2015 2:22 PM
  • Yeah, it was all service side. At a high level, we have a job queue that we're pushing a lot of different types of jobs through and while deploying updates we push a lot of servicing jobs through this queue. Build itself uses this queue to push builds down to the correct build agents and handle various other parts of the build flow. Due to a bad setting during the last rollout we had some periods of time where the servicing jobs were piling up ahead of the build jobs in the queue causing them to get delayed (significantly in some cases, as you saw).

    We fixed the setting and job processing corrected itself, allowing all of that built up work to start flowing again. Several work items came out of the event so we can avoid this happening again in the future. Please do let me know if you see these types of issues again. 

    Thanks,

    Dennis

    Thursday, May 21, 2015 2:33 PM
  • I'm actually experiencing similar behavior now.

    A builds been sitting on the queue for 20 minutes now.

    Thursday, May 21, 2015 6:57 PM
  • Ok, I'll take a look. What is your visualstudio.com account name? You can email it to me at densto at microsoft.com if you'd prefer.


    Thursday, May 21, 2015 7:06 PM
  • Thanks Dennis.  I sent my information in an email.
    Thursday, May 21, 2015 7:23 PM
  • Strange, I still haven't seen it come through. I'll keep an eye out for it.
    Thursday, May 21, 2015 7:39 PM
  • Resent with correct email address this time.  :)
    Thursday, May 21, 2015 8:14 PM
  • I'm seeing very similar behavior today 6/10. Our source code is hosted in visual studio online, and we have an on prem build server. TFS 2015 RTM. I can clear the build queue, stop and start the agent service on the build server, and the pool looks ready for work in VSO. When I queue the build, I get the same error message "queued and waiting to start". I looked at the agent log files, and there are no error messages that stand out. Any suggestions? Jeff
    Wednesday, June 10, 2015 9:35 PM
  • Hi Jeff, if you leave the build queued for a while is it eventually starting? If you're still having the issue please send me a mail at densto at microsoft.com with your account name (ex: yourAccountName.visualstudio.com) and I will investigate further.

    Thursday, June 11, 2015 11:56 AM
  • Hi Dennis, 

    I'm experiencing similar issues. Builds have been starting very quickly, within a minute of them being queued, up until about 30 minutes ago. Now any build I queue just sort of sits there waiting.

    I'm using Visual Studio Online, with the hosted queues. 

    Aside from this small delay - it's a very impressive product. It's very functional and I'm really enjoying using it. Thank you!

    Wednesday, July 22, 2015 3:43 PM
  • Hi Phil, we had some performance issues in one of our data centers earlier today, they should be all resolved by now though. If you're still seeing long delays please send me your visualstudio.com account name and I'll take a look at what's going on (densto@microsoft.com)

    Thanks,

    Dennis

    Wednesday, July 22, 2015 4:13 PM
  • Hi Dennis, I am experiencing exactly the same issue. I have just migrated my entire solution to VS 2015 and have created a brand new build. The build loads and then hangs perpetually on the 

    The build has been queued and is waiting to start.

    Were there any fixes I should know about?

    Many thanks

    Laurence


    Laurence Lemmon-Warde

    Tuesday, August 4, 2015 3:05 PM
  • Hi Laurence, the hosted build queues were backed up this morning in several data centers due to some Azure Virtual Machine & Storage issues. I believe the issue was mitigated a little while ago and our backed up queue's are catching up now.

    You can track status here:

    http://blogs.msdn.com/b/vsoservice/archive/2015/08/04/issues-with-visual-studio-online-hosted-build-service-8-4-investigating.aspx

    Tuesday, August 4, 2015 3:23 PM
  • Thanks for the quick reply Dennis. When looking at the Hosted Build service for VS Online we notice these values pointing to older versions of VS. Are these correct?

    

    


    Laurence Lemmon-Warde

    Tuesday, August 4, 2015 3:59 PM
  • Actually that list looks a bit out of date, we just deployed some updates over the last week. I'll work on getting that updated, thanks for pointing it out.

    For the VS related software, we have 2015 RTM on the box, and AzurePS is actually 0.9.4 (June 2015).

    Although confusing this shouldn't actually impact you unless you're writing custom build tasks that demand specific capability versions, I believe all of our built-in tasks just verify the existence of a capability.

    Tuesday, August 4, 2015 4:17 PM
  • Thanks for the fast reply Dennis. Apologies for not replying sooner - I missed the notification for it.

    It is all working just fine now, and yes it looks like it was just a temporary spike.

    Kind Regards,

    Phil

    Tuesday, September 1, 2015 1:00 AM