Bug? Occasional scheduling problem with the build server and TFS

Question

I am having a strange problem which occurs occasionally (I can't seem to see a pattern) with our nightly build.
We currently have 2 builds scheduled:
-"Build1" scheduled at 05:00
-"Build2" scheduled at 06:00
Occasionally "Build1" fails to start at the correct time and runs exactly two hours later, and then "build2" fails when attempting to check in some modified files with the error:
"Conflict: A newer version of $/xxxxx/xxx.dll exists on the server"

The build server reports the build was scheduled around the correct time (tbl_buildDefinition.LastSystemBuildStartTime = 05:10:15) but according to the build summary
it started at 07:10:15.

My problem is not the 2 hour delay (which is annoying); but that "build2" fails affecting the reliability of our nightly builds.
The check in seems to think that the files on the server is newer than the files that was just compiled, this error occurs only when this strange scheduling problem occurs.

We are using TFS 2008 SP 1, and Team Build SP 1.
The TFS and Team Build are on separate machines on the same domain.

Answer 1

Hi,

Are the TFS server and the build machine on the same time zone? Also - do these builds have the same work directory? If the work directories are different, your builds should not impact each other.

Thanks,
Alin,
PRAKTIK Consulting
TFS Hosting and TFS Consulting Services.

Answer 2

Hi Fred,

For the check-in conflict error, can you check the dll file's history and find the last one who checked in.

If it is checked in by Build1, then you may have dependency issues between Build1 and Build2.

As Alin said, it may have time zone issues between log file and database. You should check both build's log file to determin which build is run first.

A workaround is to invoke tf resolve command before you check-in. You can choose to AcceptYours and discard the changes in server. For detail, please check: http://msdn.microsoft.com/en-us/library/6yw3tcdy.aspx

 

Hongye Sun [MSFT]
MSDN Subscriber Support in Forum
If you have any feedback on our support, please contact msdnmg @ microsoft.com

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.

Answer 3

Thanks for the reply.
Yes the build server and TFS are in the same timezone, and this error occurs only occasionally, not every scheduled build.
They have different working directories and they don't overlap in any way.

Answer 4

Thanks for the reply.
The version history indicates that "Build1" checked files in before "Build2" started which is correct and it is supposed to do - it does so without problem when the 2 hour build delay doesn't occur.

Both builds are full builds (NOT incremental) and "Build2" starts after "Build1" as they are being build on the same Build server with different workspace mappings.

Im not fan of hacks and would like to resolve this by finding the root cause of the problem.

1. Sometimes the build is scheduled at the correct time but doesn't start before much later (both servers in the same timezone)
2. When this delay occurs "build2" fails, claiming that some of the files it attempts to check in have been modified on the server in the meantime.

Thanks!

Answer 5

Hi Fred,

Sorry for my delayed response.

For the build schedule issue, I suggest you to turn on tracing in TFS build. It is configurable in <Visual Studio Installation Folder>\Common7\IDE\PrivateAssemblies\TFSBuildService.exe.config

Uncomment the system.diagnostics section and ensure that C:\logs directory exists and is writable by the build service account.

One possibility for the build2 fails is that build2 builds too long time and build1 starts and completes before build2 completes. Here is the build order:

1. Build2 starts building
2. Build2 checkout latest source code
3. Build1 starts building
4. Build1 checkout latest source code
5. Build1 checkin code
6. Build1 completes build
7. Build2 checkin code (Here the error occurs because an newer version was checked in by Build1).

It's not a good practice that two builds will check in same file. That will make dependencies between multiple team build. TFS currently doesn't run different build type's build in sequential. So you'd better to fix the build schedule issue first or elimates the dependencies between two builds by avoiding check-in same file.

Hope it helps. 

Hongye Sun [MSFT]
MSDN Subscriber Support in Forum
If you have any feedback on our support, please contact msdnmg @ microsoft.com

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.

Answer 6

Hi,

Please correct me if I'm wrong, but:
1. An build-agent builds scheduled builds sequentially.
2. We have have only one build agent.
3. How can we have an race-condition if the builds are sequential?
   - "Build2" cannot start building before "Build1" is finished
   - According to the log, "Build2" started after "Build1"

In regards to the "tf Resolve" workaround, you cannot resolve unless you have attempted an checkin.
This means that you have to implement "tf resolve" as part of the errorhandler.
It's not an very elegant solution.

 

 

 

<!-- attempt to resolve conflict and checkin  -->
<

 

Target Name="_BGHandleNETCheckinErrors">
<Exec Command="$(TeamBuildRefPath)\..\tf.exe resolve &quot;$(NETFrameworkBinariesPath)&quot; /auto:AcceptYours" />

<MSBuild.ExtensionPack.VisualStudio.TfsSource TaskAction="Checkin" ItemPath="$(SCNETCommonBinaries)/Framework/*.*" OverrideText="Build" Comments="***NO_CI***" WorkingDirectory="$(NETFrameworkBinariesPath)" />
<OnError ExecuteTargets="_BGHandleCheckinErrors"/>

</

 

 

Target>


I *know* that it's not best practice that both builds have the same dependencies, but unfortunately that is how it currently is.
:)

<!-- Checkin .NET binaries  -->
<Target Name="CheckinNET" Condition=" $(BuildFailed)!='true' ">
<MSBuild.ExtensionPack.VisualStudio.TfsSource TaskAction="Checkin" ItemPath="$(SCNETCommonBinaries)/Framework/*.*" OverrideText="Build" Comments="***NO_CI***" WorkingDirectory="$(NETFrameworkBinariesPath)" />

<OnError ExecuteTargets="_BGHandleNETCheckinErrors"/>
</Target>

Answer 7

Hi Fred,

Thanks for your reply.

I forget to ask: are Build1 and Build2 in same team project? Build agent executes builds in serial only when the builds are in same team project.

If the two builds did run sequentially, please check the Build2's log file when it fails:
What the version number is when it gets the files in $(NETFrameworkBinariesPath)? Is it the latest version number?

Check the files in $(NETFrameworkBinariesPath) in Team Explorer:
What's its latest check-in time? Is it before the Build2 starts?


Hongye Sun [MSFT]
MSDN Subscriber Support in Forum
If you have any feedback on our support, please contact msdnmg @ microsoft.com

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.

Answer 8

Hi fred,

Is this issue resolved? Thanks.

Hongye Sun [MSFT]
MSDN Subscriber Support in Forum
If you have any feedback on our support, please contact msdnmg @ microsoft.com

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.

Answer 9

Sorry for my delayed response & thank you for comments!

"Build1" and "Build2" are in the same team project, so yes they are built sequentially.

The check-in time for the NETFrameworkBinaries are before "build2" started.

Answer 10

Hi Fred,

It seems that the in Build2 doesn't get the latest source. I don't know why. Can you have a try to set following property in your TFS project to get the latest version?

<GetVersion>T</GetVersion>

Thanks,
 

Hongye Sun [MSFT]
MSDN Subscriber Support in Forum
If you have any feedback on our support, please contact msdnmg @ microsoft.com

---

Please remember to mark the replies as answers if they help and unmark them if they provide no help.
Welcome to the All-In-One Code Framework! If you have any feedback, please tell us.