locked
LabDefaultTemplate and Gated Check-ins RRS feed

  • Question

  • I understand that this functionality is not supported at this point. 

    My question to the forum is, is there technical reason why this is not enable-able?

    I am aware that as a Gated Check-In, the time required to build, deploy, test will make the cycle long for some.  In our shop, they have requested that ALL of our check-ins go through Unit and Integration tests prior to being committed to our trunk.  We've told them that this will definitely increase the time between builds, but they are willing to see just how bad it is, and decide from there.

    The LabDefaultTemplate workflow would work perfectly (on the surface at least) as you can queue up a specific build definition, which would run our UnitTests, and if that passes, proceeds to go to the Integration portion. 

    Has any other groups out there done something similar?  If so, how did you approach either the task, or management :)? 

    I would appreciate any anecdotal experiences,

    Cheers!

    Eric.

    Wednesday, July 11, 2012 2:09 PM

Answers

  • Just an update on how we've been going.  Initially, we left this as a 'research' type of project.  We had modified the stock build definition to launch addition build workflows at the end of the gating decision.

    So basically, once Unit tests were done, and gated, we then kicked off via an activity, another build workflow which was our lab environment which had integration/system tests.  These were not part of the gating decision.

    We've been running this way for a few months now while we worked on other engineering stories.  We're lucky enough that the build+unit tests take ~25minutes, and with 6 agents, there's not too much waiting for a build to complete (they're ok with the sub-30 minute build time + gating).

    Our current step now is to compartmentalize the testing so that it isn't executed as part of the compile stage.  This will allow us to compile everything, and if it compiles, then run unit, then if those pass, integration, etc.  We're adopting a 'dirty build agent' which will be executing the gating checkin's -- it'll have all of the necessary components to run the end product (IIS, SQL).  With the separation of test and compile, we're able to turn off IIS completely at the start of the test cycle to ensure that Unit tests aren't relying on some extra component to be there.  Keeps our tests honest and on some defined schedule, we'll be building on a 'clean' agent which will be our release builds.

    After the compile test phase, we turn IIS back on and execute the integration.  Once those are done, we then do the gating decision, etc.  So far, it's working beautifully.   

    The step after is to modify the workflow again to launch a Hyper-V set of machines at the start of compile, to which we'll deploy our compiled/unit tested build onto, and then execute our integration/system tests on that, effectively merging the two workflows (stock + labtemplate) into one.

    That alone has been challenging (working with workflows).  It's not the fastest, or easiest to debug (I'd love to hear if there are any tool/tips/tricks to do this efficiently).

    Thanks for the added discussion :)

    Cheers,


    Eric.

    • Proposed as answer by Ed Blankenship Wednesday, October 3, 2012 2:05 PM
    • Marked as answer by eric_pelletier Wednesday, October 3, 2012 2:28 PM
    Wednesday, October 3, 2012 1:45 PM
  • This is certainly a scenario that I'm interested in diving in some more details.  I have definitely heard of other customers that want to enable the BDT workflow as part of their verification process with check-ins and also want to reduce the amount of time for those test runs.  Feel free to ping me over e-mail and we can chat some more about this scenario:  ed.blankenship@microsoft.com

    In the meantime, I don't see any particular reasons why this may not work.  The best part of Gated Check-In in TFS Build is that you can define what pass/fail means.  If that means going through a full BDT run then you would be able to do that with the existing tools.  Reply back if you do happen to run into any issues.


    Ed Blankenship
    Professional TFS 2012 Book [New!]|  Professional TFS 2010 Book

    Ed Squared Blog


    • Edited by Ed Blankenship Thursday, July 12, 2012 7:31 AM Update Signature
    • Proposed as answer by Ed Blankenship Wednesday, October 3, 2012 2:05 PM
    • Marked as answer by eric_pelletier Wednesday, October 3, 2012 2:28 PM
    Thursday, July 12, 2012 7:28 AM

All replies

  • This is certainly a scenario that I'm interested in diving in some more details.  I have definitely heard of other customers that want to enable the BDT workflow as part of their verification process with check-ins and also want to reduce the amount of time for those test runs.  Feel free to ping me over e-mail and we can chat some more about this scenario:  ed.blankenship@microsoft.com

    In the meantime, I don't see any particular reasons why this may not work.  The best part of Gated Check-In in TFS Build is that you can define what pass/fail means.  If that means going through a full BDT run then you would be able to do that with the existing tools.  Reply back if you do happen to run into any issues.


    Ed Blankenship
    Professional TFS 2012 Book [New!]|  Professional TFS 2010 Book

    Ed Squared Blog


    • Edited by Ed Blankenship Thursday, July 12, 2012 7:31 AM Update Signature
    • Proposed as answer by Ed Blankenship Wednesday, October 3, 2012 2:05 PM
    • Marked as answer by eric_pelletier Wednesday, October 3, 2012 2:28 PM
    Thursday, July 12, 2012 7:28 AM
  • Since this is not supported, say it :)

    Ignoring the reality, measure build-deploy-test cycle duration and number of check-ins. On a decent project you'll be backlogged because gated changes must be accepted or rejected sequentially. 4 hours CI cycle allows you as little as 6 check-ins on an average day.

    Approaching the task: write your own workflow. It's not that hard, but learnign and testing workflows takes certain time. Use this as an opportunity to learn team build. Ask 4 weeks for POC project. Ask what is their plan B when daily changes won't squeeze through non-scalable gated check-in hole.

    However, I would approach management with 2 weeks notice. Seriously, there is a real challenging work out there.


    Regards, Dmitry


    • Edited by Demchuk Wednesday, October 3, 2012 8:34 AM
    Wednesday, October 3, 2012 8:33 AM
  • Hi Dmitry,

    It is a supported scenario.  Customizing build process templates is supported.  A full BDT gated check-in feature is not available out of the box in Team Foundation Server 2012 but if you were to implement it with a customized build process template, then it would be supported by the Microsoft Product Support Services team if you run into issues.

    Team Foundation Server 2012 also introduces the concept of Batched Gated Check-Ins which will group several changes together and attempt to build & test them together.  That new feature would be helpful in this scenario.

    It would take some work but could be done.  If you are not familiar with how to customize the build process, there are some good resources available for learning.


    Ed Blankenship
    Professional TFS 2012 Book [New!] |  Professional TFS 2010 Book

    Ed Squared Blog

    Wednesday, October 3, 2012 1:14 PM
  • Just an update on how we've been going.  Initially, we left this as a 'research' type of project.  We had modified the stock build definition to launch addition build workflows at the end of the gating decision.

    So basically, once Unit tests were done, and gated, we then kicked off via an activity, another build workflow which was our lab environment which had integration/system tests.  These were not part of the gating decision.

    We've been running this way for a few months now while we worked on other engineering stories.  We're lucky enough that the build+unit tests take ~25minutes, and with 6 agents, there's not too much waiting for a build to complete (they're ok with the sub-30 minute build time + gating).

    Our current step now is to compartmentalize the testing so that it isn't executed as part of the compile stage.  This will allow us to compile everything, and if it compiles, then run unit, then if those pass, integration, etc.  We're adopting a 'dirty build agent' which will be executing the gating checkin's -- it'll have all of the necessary components to run the end product (IIS, SQL).  With the separation of test and compile, we're able to turn off IIS completely at the start of the test cycle to ensure that Unit tests aren't relying on some extra component to be there.  Keeps our tests honest and on some defined schedule, we'll be building on a 'clean' agent which will be our release builds.

    After the compile test phase, we turn IIS back on and execute the integration.  Once those are done, we then do the gating decision, etc.  So far, it's working beautifully.   

    The step after is to modify the workflow again to launch a Hyper-V set of machines at the start of compile, to which we'll deploy our compiled/unit tested build onto, and then execute our integration/system tests on that, effectively merging the two workflows (stock + labtemplate) into one.

    That alone has been challenging (working with workflows).  It's not the fastest, or easiest to debug (I'd love to hear if there are any tool/tips/tricks to do this efficiently).

    Thanks for the added discussion :)

    Cheers,


    Eric.

    • Proposed as answer by Ed Blankenship Wednesday, October 3, 2012 2:05 PM
    • Marked as answer by eric_pelletier Wednesday, October 3, 2012 2:28 PM
    Wednesday, October 3, 2012 1:45 PM