VS2010 dependencies problems

Question

We have recently  switched our solution from vs2008 to vs2010 and we are sometimes having dependencies problems when compiling our win32 targets. Basically, sometimes, our pch file do not get recompiled when a .h file included in it has changed. Then some cpp files that changed get recompiled but they issue errors because they are still using an old pch. This prevents our build system from working properly.

 

I was able to isolate the cause. Our pch cpp files are sometimes listed in the dependency file cl.read.1.tlog in a wrong way. Three things can happen whenever we rebuild a project. The first 2 are OK, the third is our problem.

 

1)      Sometimes, the pch.cpp file does not have an entry in the file cl.read.1.tlog. So when we compile again later, the compiler tells us that the pch is not listed in the cache and it checks the dependencies in some other way. That will be ok and the pch will be recompiled only if some of its dependencies have changed.

2)      Sometimes, the pch.cpp file is listed in cl.read.1.tlog and all its dependencies are listed with it. This is, I guess, the normal behaviour and works fine when we compile again later and some .h files have changed.

3)      Sometimes, the pch.cpp will get an entry in cl.read.1.tlog, but its list of dependencies is incomplete! It only lists the first 4 files (dll files from the windows directory) and no .h files are listed. Then when we compile again, after a sync from perforce for example, the compiler finds the pch.cpp in the cl.read.1.tlog, and uses its dependencies listed to decide if it needs recompiling. Since this dependency list is incomplete, the compiler does not see that some .h files have changed and so does not recompile the pch.cpp. Then other cpp files try to compile with an old pch file and can have build errors.

 

The third behaviour is of course the one causing us problems.

 

Here is an sample of cl.read.1.tlog for behaviour 2:

 

^F:\FC3\MAIN\CODE\CORE\CORE_PCH.CPP

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\BIN\1033\CLUI.DLL

C:\WINDOWS\GLOBALIZATION\SORTING\SORTDEFAULT.NLS

C:\WINDOWS\SYSTEM32\RSAENH.DLL

C:\WINDOWS\SYSTEM32\TZRES.DLL

F:\FC3\MAIN\CODE\CORE\COREBASIC.H

F:\FC3\MAIN\CODE\CORE\COREBASICCOMMON.H

F:\FC3\MAIN\CODE\CORE\BASIC\BASICTYPES.H

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\INCLUDE\EXCEPTION

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\INCLUDE\XSTDDEF

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\INCLUDE\YVALS.H

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\INCLUDE\CRTDEFS.H

Etc...

 

Here is an sample of cl.read.1.tlog for problematic behaviour 3:

 

^F:\FC3\MAIN\CODE\CORE\CORE_PCH.CPP

C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 10.0\VC\BIN\1033\CLUI.DLL

C:\WINDOWS\GLOBALIZATION\SORTING\SORTDEFAULT.NLS

C:\WINDOWS\SYSTEM32\RSAENH.DLL

C:\WINDOWS\SYSTEM32\TZRES.DLL

^F:\FC3\MAIN\CODE\CORE\CONTAINERS\VECTOR.CPP

F:\FC3\MAIN\TMP\WIN32\RELEASE_DLL\CORE\CORE_PCH.H.PCH

F:\FC3\MAIN\CODE\PLATFORMS\WIN32\CORE\MEM\MEMMNGBUCKETSPLATFORM.H

F:\FC3\MAIN\CODE\CORE\MEM\MEMMNGPRIVATE.H

F:\FC3\MAIN\CODE\CORE\THREAD\CRITSECTION.H

F:\FC3\MAIN\CODE\CORE\MEM\CONTEXTLOG.H

F:\FC3\MAIN\CODE\PLATFORMS\WIN32\CORE\MEM\MEMMNGCONFIG.H

Etc…

 

Notice that CORE_PCH.CPP is present in the file, but its list is incomplete.

 

Any fix for this kind of problems?

 

To temporarily bypass this problem, we have added a pre-build step on our buildmachiens that deletes all the files cl.read.1.tlog. This as fixed the problem.

 

Finally, note that we also have similar problems where we recompile some projects when they are actually already fully build. This does not break the builds, but makes our work longer because we recompile more often then needed. These problems are not happening all the time and not on all PCs. We have had this same problem by making a new simple win32 projet and chaging some .h files outside of Visual Studio. Same thing happens, only more rarely.

 

Thanks,

 

David Chabot

Ubisoft

Senior Game Engine Programmer

 

P.S. Our PCs are running Windows 7 64 bits with Visual Studio 2010 with SP1

Answer 1

Hello ,

Thank you for your question.

 

Starting from VS2010, VC++ uses MSBuild as the build engine. You may take a look the following blog: http://blogs.msdn.com/b/vcblog/archive/2010/03/02/visual-studio-2010-c-project-upgrade-guide.aspx

 

I am trying to involve someone familiar with this topic to further look at this issue. There might be some time delay. Appreciate your patience.

Thank you for your understanding and support.

 

Yi

 

---

Yi Feng Li [MSFT]
MSDN Community Support | Feedback to us

Answer 2

We have found the problem.

It is a PlayStation3 VS AddIn for vs2008 that was causing the problem. Uninstalling it fixes all.

Sorry for bothering you guys with this.

Cheers,

 

David Chabot

Answer 3

We are also having this problem, but without the PS3 vs2008 addin.

In our case, what is causing the incomplete tlog is cancelling an in-progress build. The tlog files will have an incomplete dependency list for the .cpp that was compiling at the moment the build was cancelled.

The problem is more easily apparent when we "blob" .cpp file - that is, we don't compile the individual .cpp files separately, but rather include many of them in a master .cpp file (having 30-ish big files compiles faster than hundreds of smaller files).
When doing this, we have a pattern where the compiler processes a bunch of include headers, then C++ code, then more headers, more code, and so on.

This master cpp file is only a collection of entries like the following:
#pragma message("ProjectName - path/file.cpp")
#include "../../../path/file.cpp"

So, with this pattern, the problem becomes easy to reproduce when the master cpp file is large enough:

1. Prepare a master .cpp file including several .cpp files
2. Compile normally, make sure the code compiles completely without errors.
3. Modify one of the individual .cpp files included near the end of the master cpp file, write junk that should not compile
4. Launch a build
5. Cancel the build before reaching the modified .cpp file
6. The tlog file for our master cpp is now incomplete and lists only dependencies up to the point where we cancelled the build.
7. Launch a build again
8. Since the modified file that contains the junk code is not listed int he dependencies anymore, the build system thinks the master cpp file is up to date
9. The build completes without error

When a build is cancelled in progress the dependency data is overwritten with incomplete one, which is an incorrect behavior. The expected correct behavior would be to either ensure to maintain the correctness of the dependency check or force to consider that .cpp file as outdated on the next build.
In the case we have, maintaining the correctness of the dependency list simply means to update the list only when the compilation successfully terminates.
If the compilation is aborted, the dependency list should NOT be updated. This would ensure the file gets recompiled on the next build.

This appears to be a major flaw in the way the dependencies are processed, and makes an incremental build quite unreliable. When the result is code that doesn't link, the problem is readily apparent and it is easy to locate the file(s) that didn't rebuild. A much more problematic situation arises when the code successfully links - in that case, some compilation units are still using obsolete code, leading to very tricky and time consuming bugs.

We have wasted a lot of valuable time because of this issue. And not using the cancel command but wait until the build completes would be a waste of time as well, when we have compile errors in header files.

Thanks for any help with this issue,

Marc Allaire
Ubisoft
Programmer